You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/10/13 17:07:01 UTC
[GitHub] [airflow] marcusianlevine opened a new pull request #11509: Refactor Elasticsearch provider to support 1.10.x
marcusianlevine opened a new pull request #11509:
URL: https://github.com/apache/airflow/pull/11509
Resolves #11479
In order to properly read logs from Elasticsearch in 1.10.x with the latest Elasticsearch backport provider, we need to use the newer `FileTaskHandler` base class
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on pull request #11509: Refactor Elasticsearch provider to support 1.10.x
Posted by GitBox <gi...@apache.org>.
kaxil commented on pull request #11509:
URL: https://github.com/apache/airflow/pull/11509#issuecomment-730535748
> Will do, though honestly I'm not sure what the problem is, and I'm not sure how to test this refactor logic locally
>
> Do you think it could be that I'm only specifying the top-level `./airflow/providers/elasticsearch/` directory instead of the specific `log/` directory where `es_task_handler.py` lives? https://github.com/apache/airflow/pull/11509/files#diff-11dd8f34d8ae58d0c8cc576f298e4e83d40b512384d1e1707b6313f406b5f256R461
@potiuk @turbaszek Might be able to help with some pointers around that and using Bowler
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on pull request #11509: Refactor Elasticsearch provider to support 1.10.x
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on pull request #11509:
URL: https://github.com/apache/airflow/pull/11509#issuecomment-730619931
@marcusianlevine. I believe is that you have not modified your logging configuration. The thing is that in airflow 1.10 the ElasticsearchTaskHandler is part of the airflow 1.10 itself in `airflow/utils/log/es_task_handler.py` - and almost by definition, those files are not touched by the new provider. What you need to do, is to modify your logging configuration to use the new "provider" task handler: airflow.providers.elasticsearch.log.es_task_handler
In 1.10 the configuration looked like this (this is a default, so you might have modified it):
```
ELASTIC_REMOTE_HANDLERS = {
'task': {
'class': 'airflow.utils.log.es_task_handler.ElasticsearchTaskHandler',
'formatter': 'airflow',
'base_log_folder': os.path.expanduser(BASE_LOG_FOLDER),
'log_id_template': ELASTICSEARCH_LOG_ID_TEMPLATE,
'filename_template': FILENAME_TEMPLATE,
'end_of_log_mark': ELASTICSEARCH_END_OF_LOG_MARK,
'host': ELASTICSEARCH_HOST,
'write_stdout': ELASTICSEARCH_WRITE_STDOUT,
'json_format': ELASTICSEARCH_JSON_FORMAT,
'json_fields': ELASTICSEARCH_JSON_FIELDS
},
}
```
And when you switch it to the new provider you have to change it to this (note changed 'class' attribute):
```
ELASTIC_REMOTE_HANDLERS: Dict[str, Dict[str, Union[str, bool]]] = {
'task': {
'class': 'airflow.providers.elasticsearch.log.es_task_handler.ElasticsearchTaskHandler',
'formatter': 'airflow',
'base_log_folder': str(os.path.expanduser(BASE_LOG_FOLDER)),
'log_id_template': ELASTICSEARCH_LOG_ID_TEMPLATE,
'filename_template': FILENAME_TEMPLATE,
'end_of_log_mark': ELASTICSEARCH_END_OF_LOG_MARK,
'host': ELASTICSEARCH_HOST,
'frontend': ELASTICSEARCH_FRONTEND,
'write_stdout': ELASTICSEARCH_WRITE_STDOUT,
'json_format': ELASTICSEARCH_JSON_FORMAT,
'json_fields': ELASTICSEARCH_JSON_FIELDS,
},
}
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] marcusianlevine commented on pull request #11509: Refactor Elasticsearch provider to support 1.10.x
Posted by GitBox <gi...@apache.org>.
marcusianlevine commented on pull request #11509:
URL: https://github.com/apache/airflow/pull/11509#issuecomment-733078360
Thanks for looking into this @potiuk - I just confirmed that I do indeed have the correct version installed (these commands were run inside my webserver's Docker container):
```
$ pip list | grep elasticsearch
apache-airflow-backport-providers-elasticsearch 2020.11.13
```
I confirmed as well that, as before, the import in `/usr/local/lib/python3.7/site-packages/airflow/providers/elasticsearch` has not been modified:
```
$ cat /usr/local/lib/python3.7/site-packages/airflow/providers/elasticsearch/log/es_task_handler.py | grep FileTask
from airflow.utils.log.file_task_handler import FileTaskHandler
class ElasticsearchTaskHandler(FileTaskHandler, LoggingMixin):
```
What's peculiar is that the copied version of the `file_task_handler` module is present inside the provider at the correct location:
```
$ ls /usr/local/lib/python3.7/site-packages/airflow/providers/elasticsearch/common/utils/log/
__init__.py __pycache__ file_task_handler.py
```
Maybe I'm not understanding the backport provider structure correctly... am I not looking in the right place for the refactored code?
As you observed, I did specify the custom logging class based on where I thought the backport provider class should be found (`airflow.providers.elasticsearch.log.es_task_handler.ElasticsearchTaskHandler`) but that corresponds to the file above which does not have the proper refactored import... 🤯
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on pull request #11509: Refactor Elasticsearch provider to support 1.10.x
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on pull request #11509:
URL: https://github.com/apache/airflow/pull/11509#issuecomment-722613481
For now @marcusianlevine -> I made an innocent change in "scripts/ci/provider_packages" folder that should trigger all tests (just in case)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] marcusianlevine commented on pull request #11509: Refactor Elasticsearch provider to support 1.10.x
Posted by GitBox <gi...@apache.org>.
marcusianlevine commented on pull request #11509:
URL: https://github.com/apache/airflow/pull/11509#issuecomment-730502958
Will do, though honestly I'm not sure what the problem is, and I'm not sure how to test this refactor logic locally
Do you think it could be that I'm only specifying the top-level `./airflow/providers/elasticsearch/` directory instead of the specific `log/` directory where `es_task_handler.py` lives? https://github.com/apache/airflow/pull/11509/files#diff-11dd8f34d8ae58d0c8cc576f298e4e83d40b512384d1e1707b6313f406b5f256R461
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] marcusianlevine commented on pull request #11509: Refactor Elasticsearch provider to support 1.10.x
Posted by GitBox <gi...@apache.org>.
marcusianlevine commented on pull request #11509:
URL: https://github.com/apache/airflow/pull/11509#issuecomment-722487907
I rebased this PR and resolved all the Black issues, let me know if there's anything else required to get this merged
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #11509: Refactor Elasticsearch provider to support 1.10.x
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #11509:
URL: https://github.com/apache/airflow/pull/11509#issuecomment-730623586
Ah sorry - my bad I see that you actually used the right one. Let me take a closer look
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] marcusianlevine commented on pull request #11509: Refactor Elasticsearch provider to support 1.10.x
Posted by GitBox <gi...@apache.org>.
marcusianlevine commented on pull request #11509:
URL: https://github.com/apache/airflow/pull/11509#issuecomment-730441486
Thanks for your help getting this in everyone, but it seems that the programmatic refactor didn't work as expected.
I installed the latest backport provider release, but while the copy of the Airflow 2.0 `file_task_handler.py` file is copied into the proper location inside the installed backport provider, the provider's `log/es_task_handler.py` file was not properly refactored to import the `FileTaskHandler` from the correct module:
```
$ cat /usr/local/lib/python3.7/site-packages/airflow/providers/elasticsearch/log/es_task_handler.py | grep file_task_handler
from airflow.utils.log.file_task_handler import FileTaskHandler
```
This block is supposed to select the appropriate files inside the provider to target for the refactor, but it doesn't seem to work: https://github.com/apache/airflow/pull/11509/files#diff-11dd8f34d8ae58d0c8cc576f298e4e83d40b512384d1e1707b6313f406b5f256R494
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] marcusianlevine commented on pull request #11509: Refactor Elasticsearch provider to support 1.10.x
Posted by GitBox <gi...@apache.org>.
marcusianlevine commented on pull request #11509:
URL: https://github.com/apache/airflow/pull/11509#issuecomment-733094652
This Docker image is built from the official `python:3.7-stretch` base image, installing all packages into the system Python installation - I'm not using virtualenv at all, and I only ever install official `1.10.x` releases in this image's Dockerfile
Here is the exact `pip` invocation from my Dockerfile:
```
RUN pip3 install \
...
apache-airflow[kubernetes,elasticsearch]==1.10.12 \
apache-airflow-backport-providers-elasticsearch \
apache-airflow-backport-providers-hashicorp \
--constraint "https://raw.githubusercontent.com/apache/airflow/constraints-1.10.12/constraints-3.7.txt"
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] marcusianlevine commented on a change in pull request #11509: Refactor Elasticsearch provider to support 1.10.x
Posted by GitBox <gi...@apache.org>.
marcusianlevine commented on a change in pull request #11509:
URL: https://github.com/apache/airflow/pull/11509#discussion_r504191490
##########
File path: provider_packages/refactor_provider_packages.py
##########
@@ -435,6 +435,62 @@ def amazon_package_filter(node: LN, capture: Capture, filename: Filename) -> boo
rename("airflow.providers.amazon.common.utils.email")
)
+ def refactor_elasticsearch_package(self):
Review comment:
Doh! Forgot to invoke it down below 🤦♂️
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on pull request #11509: Refactor Elasticsearch provider to support 1.10.x
Posted by GitBox <gi...@apache.org>.
kaxil commented on pull request #11509:
URL: https://github.com/apache/airflow/pull/11509#issuecomment-721445877
Can you please rebase your PR on latest Master since we have applied [Black](https://github.com/apache/airflow/commit/4e8f9cc8d02b29c325b8a5a76b4837671bdf5f68) and [PyUpgrade](https://github.com/apache/airflow/commit/8c42cf1b00c90f0d7f11b8a3a455381de8e003c5) on Master.
It will help if your squash your commits into single commit first so that there are less conflicts.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #11509: Refactor Elasticsearch provider to support 1.10.x
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #11509:
URL: https://github.com/apache/airflow/pull/11509#issuecomment-733092048
And are you sure you did not have an earlier version of airflow 2.0 installed there before (before alphas ? ) We split airflow before releasing it but if you installed it from sources before, it could be that some remnant files are there . Can you repeat the exercise and install Airflow + provider from the scratch ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on pull request #11509: Refactor Elasticsearch provider to support 1.10.x
Posted by GitBox <gi...@apache.org>.
kaxil commented on pull request #11509:
URL: https://github.com/apache/airflow/pull/11509#issuecomment-730485369
> Thanks for your help getting this in everyone, but it seems that the programmatic refactor didn't work as expected.
>
> I installed the latest backport provider release, but while the copy of the Airflow 2.0 `file_task_handler.py` file is copied into the proper location inside the installed backport provider, the provider's `log/es_task_handler.py` file was not properly refactored to import the `FileTaskHandler` from the correct module:
>
> ```
> $ cat /usr/local/lib/python3.7/site-packages/airflow/providers/elasticsearch/log/es_task_handler.py | grep file_task_handler
> from airflow.utils.log.file_task_handler import FileTaskHandler
> ```
>
> This block is supposed to select the appropriate files inside the provider to target for the refactor, but it doesn't seem to work: https://github.com/apache/airflow/pull/11509/files#diff-11dd8f34d8ae58d0c8cc576f298e4e83d40b512384d1e1707b6313f406b5f256R494
Can you please create a PR to fix that @marcusianlevine ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk merged pull request #11509: Refactor Elasticsearch provider to support 1.10.x
Posted by GitBox <gi...@apache.org>.
potiuk merged pull request #11509:
URL: https://github.com/apache/airflow/pull/11509
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on pull request #11509: Refactor Elasticsearch provider to support 1.10.x
Posted by GitBox <gi...@apache.org>.
mik-laj commented on pull request #11509:
URL: https://github.com/apache/airflow/pull/11509#issuecomment-722556598
No tests were run on this PR. There seems to be some bug in the test selection logic.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #11509: Refactor Elasticsearch provider to support 1.10.x
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #11509:
URL: https://github.com/apache/airflow/pull/11509#issuecomment-707964218
[The Workflow run](https://github.com/apache/airflow/actions/runs/305061965) is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks$,^Build docs$,^Spell check docs$,^Backport packages$,^Checks: Helm tests$,^Test OpenAPI*.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on pull request #11509: Refactor Elasticsearch provider to support 1.10.x
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on pull request #11509:
URL: https://github.com/apache/airflow/pull/11509#issuecomment-730619931
@marcusianlevine. I believe is that you have not modified your logging configuration. The thing is that in airflow 1.10 the ElasticsearchTaskHandler is part of the airflow 1.10 itself in `airflow/utils/log/es_task_handler.py` - and almost by definition, those files are not touched by the new provider. What you need to do, is to modify your logging configuration to use the new "provider" task handler: `airflow.providers.elasticsearch.log.es_task_handler`
In 1.10 the configuration looked like this (this is a default, so you might have modified it):
```
ELASTIC_REMOTE_HANDLERS = {
'task': {
'class': 'airflow.utils.log.es_task_handler.ElasticsearchTaskHandler',
'formatter': 'airflow',
'base_log_folder': os.path.expanduser(BASE_LOG_FOLDER),
'log_id_template': ELASTICSEARCH_LOG_ID_TEMPLATE,
'filename_template': FILENAME_TEMPLATE,
'end_of_log_mark': ELASTICSEARCH_END_OF_LOG_MARK,
'host': ELASTICSEARCH_HOST,
'write_stdout': ELASTICSEARCH_WRITE_STDOUT,
'json_format': ELASTICSEARCH_JSON_FORMAT,
'json_fields': ELASTICSEARCH_JSON_FIELDS
},
}
```
And when you switch it to the new provider you have to change it to this (note changed 'class' attribute):
```
ELASTIC_REMOTE_HANDLERS: Dict[str, Dict[str, Union[str, bool]]] = {
'task': {
'class': 'airflow.providers.elasticsearch.log.es_task_handler.ElasticsearchTaskHandler',
'formatter': 'airflow',
'base_log_folder': str(os.path.expanduser(BASE_LOG_FOLDER)),
'log_id_template': ELASTICSEARCH_LOG_ID_TEMPLATE,
'filename_template': FILENAME_TEMPLATE,
'end_of_log_mark': ELASTICSEARCH_END_OF_LOG_MARK,
'host': ELASTICSEARCH_HOST,
'frontend': ELASTICSEARCH_FRONTEND,
'write_stdout': ELASTICSEARCH_WRITE_STDOUT,
'json_format': ELASTICSEARCH_JSON_FORMAT,
'json_fields': ELASTICSEARCH_JSON_FIELDS,
},
}
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #11509: Refactor Elasticsearch provider to support 1.10.x
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #11509:
URL: https://github.com/apache/airflow/pull/11509#issuecomment-730619931
@marcusianlevine. I believe is that you have not modified your logging configuration. The thing is that in airflow 1.10 the ElasticsearchTaskHandler is part of the airflow 1.10 itself in `airflow/utils/log/es_task_handler.py` - and almost by definition, those files are not touched by the new provider. What you need to do, is to modify your logging configuration to use thee new "provider" task handler: airflow.providers.elasticsearch.log.es_task_handler
In 1.10 the configuration looked like this (this is a default, so you might have modified it):
```
ELASTIC_REMOTE_HANDLERS = {
'task': {
'class': 'airflow.utils.log.es_task_handler.ElasticsearchTaskHandler',
'formatter': 'airflow',
'base_log_folder': os.path.expanduser(BASE_LOG_FOLDER),
'log_id_template': ELASTICSEARCH_LOG_ID_TEMPLATE,
'filename_template': FILENAME_TEMPLATE,
'end_of_log_mark': ELASTICSEARCH_END_OF_LOG_MARK,
'host': ELASTICSEARCH_HOST,
'write_stdout': ELASTICSEARCH_WRITE_STDOUT,
'json_format': ELASTICSEARCH_JSON_FORMAT,
'json_fields': ELASTICSEARCH_JSON_FIELDS
},
}
```
And when you switch it to the new provider you have to change it to this (note changed 'class' attribute):
```
ELASTIC_REMOTE_HANDLERS: Dict[str, Dict[str, Union[str, bool]]] = {
'task': {
'class': 'airflow.providers.elasticsearch.log.es_task_handler.ElasticsearchTaskHandler',
'formatter': 'airflow',
'base_log_folder': str(os.path.expanduser(BASE_LOG_FOLDER)),
'log_id_template': ELASTICSEARCH_LOG_ID_TEMPLATE,
'filename_template': FILENAME_TEMPLATE,
'end_of_log_mark': ELASTICSEARCH_END_OF_LOG_MARK,
'host': ELASTICSEARCH_HOST,
'frontend': ELASTICSEARCH_FRONTEND,
'write_stdout': ELASTICSEARCH_WRITE_STDOUT,
'json_format': ELASTICSEARCH_JSON_FORMAT,
'json_fields': ELASTICSEARCH_JSON_FIELDS,
},
}
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on pull request #11509: Refactor Elasticsearch provider to support 1.10.x
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on pull request #11509:
URL: https://github.com/apache/airflow/pull/11509#issuecomment-730630169
When I look at he latest 2020.11.13 release of the elastic search i see this in es_log_task_handler.py (seems perfectly OK):
```
from airflow.configuration import conf
from airflow.models import TaskInstance
from airflow.utils import timezone
from airflow.utils.helpers import parse_template_string
from airflow.providers.elasticsearch.common.utils.log.file_task_handler import FileTaskHandler
from airflow.utils.log.json_formatter import JSONFormatter
from airflow.utils.log.logging_mixin import LoggingMixin
```
You can see it yourself. I checked it here: https://dist.apache.org/repos/dist/release/airflow/backport-providers/apache-airflow-backport-providers-elasticsearch-2020.11.13-bin.tar.gz
as well as in the files downloaded from PyPI: https://pypi.org/project/apache-airflow-backport-providers-elasticsearch/#files
So I'd say you have some problem with your installation - most likely the 2020.11.13 version of elasticsearch has not been installed properly - can you check with the PyPI and which version is really installed (send us the output of `pip freeze`? We have a change in progress to be able to dump the installation information of Airflow (will be there in Airflow 2.0) and I hope to get the providers info as part of the 2.0 release (cc: @ashb) so that you will be able to dump information of which provider versions are installed and accessible in your installation, but for now we have to rely on PyPI information and hope it is correct.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #11509: Refactor Elasticsearch provider to support 1.10.x
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #11509:
URL: https://github.com/apache/airflow/pull/11509#issuecomment-730630169
When I look at he latest 2020.11.13 release of the elastic search i see this in es_log_task_handler.py (seems perfectly OK):
```
from airflow.configuration import conf
from airflow.models import TaskInstance
from airflow.utils import timezone
from airflow.utils.helpers import parse_template_string
from airflow.providers.elasticsearch.common.utils.log.file_task_handler import FileTaskHandler
from airflow.utils.log.json_formatter import JSONFormatter
from airflow.utils.log.logging_mixin import LoggingMixin
```
You can see it yourself. I checked it here: https://dist.apache.org/repos/dist/release/airflow/backport-providers/apache-airflow-backport-providers-elasticsearch-2020.11.13-bin.tar.gz
as well as in the files downloaded from PyPI: https://pypi.org/project/apache-airflow-backport-providers-elasticsearch/#files
So I'd say you have some problem with your installation - most likely the 2020.11.13 version of elasticsearch has not been installed properly - can you check with the PyPI and which version is really installed (send us the output of `pip freeze` ? We have a change in progress to be able to dump the installation information of Airflow (will be there in Airflow 2.0) and I hope to get the providers info as part of the 2.0 release (cc: @ashb) so that you will be able to dump information of which provider versions are installed and accessible in your installation, but for now we have to rely on PyPI information and hope it is corret.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #11509: Refactor Elasticsearch provider to support 1.10.x
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #11509:
URL: https://github.com/apache/airflow/pull/11509#issuecomment-722609269
> No tests were run on this PR. There seems to be some bug in the test selection logic.
The refactor is not typical "airflow" part, but changes to it should indeed trigger the provider packages checks. I will add a separate issue for that.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #11509: Refactor Elasticsearch provider to support 1.10.x
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #11509:
URL: https://github.com/apache/airflow/pull/11509#issuecomment-722613481
For now @marcusianlevine -> I made an innocent change in "scripts/ci/provider_packages" folder that should trigger all tests.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #11509: Refactor Elasticsearch provider to support 1.10.x
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #11509:
URL: https://github.com/apache/airflow/pull/11509#issuecomment-707945953
Hey @marcusianlevine Can you please rebase this one to latest master. We fixed (hopefully) a problem with queues of jobs for GitHub actions and I think when you rebase, it shoudl run much faster (more info on devlist shortly).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #11509: Refactor Elasticsearch provider to support 1.10.x
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #11509:
URL: https://github.com/apache/airflow/pull/11509#issuecomment-722610428
https://github.com/apache/airflow/issues/12115
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #11509: Refactor Elasticsearch provider to support 1.10.x
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #11509:
URL: https://github.com/apache/airflow/pull/11509#issuecomment-733090211
Are you sure the backport is installed in this python3.7 folder and not in another Python version/virtualenv ? What does Python -m site tell you ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #11509: Refactor Elasticsearch provider to support 1.10.x
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #11509:
URL: https://github.com/apache/airflow/pull/11509#issuecomment-707943621
[The Workflow run](https://github.com/apache/airflow/actions/runs/304853869) is cancelling this PR. Building image for the PR has been cancelled
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on a change in pull request #11509: Refactor Elasticsearch provider to support 1.10.x
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #11509:
URL: https://github.com/apache/airflow/pull/11509#discussion_r504182722
##########
File path: provider_packages/refactor_provider_packages.py
##########
@@ -435,6 +435,62 @@ def amazon_package_filter(node: LN, capture: Capture, filename: Filename) -> boo
rename("airflow.providers.amazon.common.utils.email")
)
+ def refactor_elasticsearch_package(self):
Review comment:
Is this function used somewhere?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #11509: Refactor Elasticsearch provider to support 1.10.x
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #11509:
URL: https://github.com/apache/airflow/pull/11509#issuecomment-707964543
[The Workflow run](https://github.com/apache/airflow/actions/runs/305063315) is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks$,^Build docs$,^Spell check docs$,^Backport packages$,^Checks: Helm tests$,^Test OpenAPI*.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on pull request #11509: Refactor Elasticsearch provider to support 1.10.x
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on pull request #11509:
URL: https://github.com/apache/airflow/pull/11509#issuecomment-722610428
https://github.com/apache/airflow/issues/12115 created
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org