You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@superset.apache.org by GitBox <gi...@apache.org> on 2021/09/17 01:28:07 UTC
[GitHub] [superset] nelson-lark opened a new issue #16723: Support AWS OpenSearch / Elasticsearch
nelson-lark opened a new issue #16723:
URL: https://github.com/apache/superset/issues/16723
**Is your feature request related to a problem? Please describe.**
As of a couple months ago, the Elasticsearch organization has made the official python elasticsearch plugin incompatible with Amazon supported OpenSearch. If you fire up Superset using the current helm chart and attempt to connect to a recently deployed AWS "Elasticsearch" - which is now an Apache 2.0 licensed OpenSearch - you will receive the ambiguous error from Superset
````
superset 2021-09-16 17:34:09,348:WARNING:superset.views.base:[SupersetError(message='(builtins.NoneType) None\n(Background on this error at: http://sqlalche.me/e/13/dbapi)', error_type=<SupersetErrorType.GENERIC_DB_ENGINE_ERROR: 'GENERIC_DB_ENGINE_ERROR'>, level=<ErrorLevel.ERROR: 'error'>, extra={'engine_name': 'ElasticSearch (SQL API)', 'issue_codes': [{'code': 1002, '
message': 'Issue 1002 - The database returned an unexpected error.'}]})]
```
If I run python directly on the Superset pod and connect to AWS "OpenSearch/Elasticsearch" cluster, i get the more detailed error from the elasticsearch python module directly:
```elasticsearch.exceptions.UnsupportedProductError: The client noticed that the server is not a supported distribution of Elasticsearch
```
It would be nice if Superset could support OpenSearch natively.
**Describe the solution you'd like**
- At the least, highlight in the docs that the elasticsearch plugin needs downgrading to be compatible with AWS Elasticsearch/OpenSearch clusters for the time being.
- Find a way to support OpenSearch natively. Quite annoyingly, I think AWS only published a java library for OpenSearch. An official python library that could be used by Sqlalchemy is needed I believe.
**Describe alternatives you've considered**
I am currently using the following `boostrapScript` which downgrades the elasticsearch python plugin to the last known compatible version (before Elasticsearch became hostile to AWS, rightfully or not).
```
bootstrapScript: |
#!/bin/bash
rm -rf /var/lib/apt/lists/* && \\
pip uninstall elasticsearch && \\
pip install \\
elasticsearch==7.13.4 \\
elasticsearch-dbapi==0.2.5 \\
elasticsearch-dbapi[opendistro] \\
psycopg2==2.8.5 \\
redis==3.2.1 &&
if [ ! -f ~/bootstrap ]; then echo "Running Superset with uid {{ .Values.runAsUser }}" > ~/bootstrap; fi
```
**Additional context**
Ultimately, the fix may need to be made in SQLALCHEMY since that is the middle man between Superset and the databases, but I thought it was worth requesting here in case anyone else is trying to connect to an Amazon "Elasticsearch"/Opensearch cluster and was having difficulty.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] dpgaspar commented on issue #16723: Support AWS OpenSearch / Elasticsearch
Posted by GitBox <gi...@apache.org>.
dpgaspar commented on issue #16723:
URL: https://github.com/apache/superset/issues/16723#issuecomment-925839183
Hi @stockholmux, I'm a maintainer/creator of elasticsearch-dbapi. I'll look into including `opensearch-py` on elasticsearch-dbapi
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] harshgadhia edited a comment on issue #16723: Support AWS OpenSearch / Elasticsearch
Posted by GitBox <gi...@apache.org>.
harshgadhia edited a comment on issue #16723:
URL: https://github.com/apache/superset/issues/16723#issuecomment-961532592
@stockholmux I have some trouble using the elasticsearch-dbapi==0.2.6, Here is the detailed description of the issue: https://github.com/apache/superset/issues/17347
Any help is greatly appreciated. Thanks in advance!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] stockholmux commented on issue #16723: Support AWS OpenSearch / Elasticsearch
Posted by GitBox <gi...@apache.org>.
stockholmux commented on issue #16723:
URL: https://github.com/apache/superset/issues/16723#issuecomment-925805468
:wave: I'm on the OpenSearch team.
Here is [some background on the client issue](https://opensearch.org/blog/community/2021/08/community-clients/) that appears to be causing this. I'm only vaguely familiar with SuperSet but I believe the issue originates here:
https://github.com/preset-io/elasticsearch-dbapi/blob/master/setup.py#L31
Effectively, Elasticsearch clients past will only work with non-OSS Elasticsearch moving forward (this break also affects Elasticsearch versions prior to 7.12 IIRC). The break occurred at ES Client v7.14, so the way by including `elasticsearch>7` new installs are pulling the latest which includes the blocking code.
Opensearch clients ([python client here](https://pypi.org/project/opensearch-py/)) will work with both OpenSearch and ES and are syntax compatible. Not sure how SuperSet as a project wants to handle this, but I'm glad to help personally and get some other folks from our side to contribute.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] dpgaspar edited a comment on issue #16723: Support AWS OpenSearch / Elasticsearch
Posted by GitBox <gi...@apache.org>.
dpgaspar edited a comment on issue #16723:
URL: https://github.com/apache/superset/issues/16723#issuecomment-934800721
it's not 100% solved, latest elasticsearch-dbapi release will not throw any errors because I've pinned down elastic elasticsearch API, so that it didn't include the AWS OpenSearch lost of compatibility. This is a quick fix, the final solution would be to include the new `opensearch-py` on elasticsearch-dbapi.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] harshgadhia edited a comment on issue #16723: Support AWS OpenSearch / Elasticsearch
Posted by GitBox <gi...@apache.org>.
harshgadhia edited a comment on issue #16723:
URL: https://github.com/apache/superset/issues/16723#issuecomment-961532592
@stockholmux I have some trouble using the elasticsearch-dbapi==0.2.6, Here is the detailed description of the issue: https://github.com/apache/superset/issues/17347
Any help is greatly appreciated. Thanks in advance!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] dpgaspar commented on issue #16723: Support AWS OpenSearch / Elasticsearch
Posted by GitBox <gi...@apache.org>.
dpgaspar commented on issue #16723:
URL: https://github.com/apache/superset/issues/16723#issuecomment-934800721
it's not 100% solved, latest elasticsearch-dbapi release will not throw any errors because I've pinned down elastic elasticsearch API, so that it didn't included the AWS OpenSearch lost of compatibility. This is a quick fix, the final solution would be to include the new `opensearch-py` on elasticsearch-dbapi.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] dpgaspar commented on issue #16723: Support AWS OpenSearch / Elasticsearch
Posted by GitBox <gi...@apache.org>.
dpgaspar commented on issue #16723:
URL: https://github.com/apache/superset/issues/16723#issuecomment-927672810
Related issue on the driver repo: https://github.com/preset-io/elasticsearch-dbapi/issues/70
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] junlincc commented on issue #16723: Support AWS OpenSearch / Elasticsearch
Posted by GitBox <gi...@apache.org>.
junlincc commented on issue #16723:
URL: https://github.com/apache/superset/issues/16723#issuecomment-922006167
> * At the least, highlight in the docs that the elasticsearch plugin needs downgrading to be compatible with AWS Elasticsearch/OpenSearch clusters for the time being.
@nelson-lark thanks for the note! i'm not familiar with this area, looping @craig-rueda in. 🙏 please do feel free to open a PR to update the docs accordingly! 🙏
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] harshgadhia commented on issue #16723: Support AWS OpenSearch / Elasticsearch
Posted by GitBox <gi...@apache.org>.
harshgadhia commented on issue #16723:
URL: https://github.com/apache/superset/issues/16723#issuecomment-961532592
@stockholmux I have some trouble using the elasticsearch-dbapi==0.2.6, Here is the detailed description of the issue: https://github.com/apache/superset/issues/17347
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] srinify commented on issue #16723: Support AWS OpenSearch / Elasticsearch
Posted by GitBox <gi...@apache.org>.
srinify commented on issue #16723:
URL: https://github.com/apache/superset/issues/16723#issuecomment-934716295
Can we close this @dpgaspar ? 🤔
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] dpgaspar edited a comment on issue #16723: Support AWS OpenSearch / Elasticsearch
Posted by GitBox <gi...@apache.org>.
dpgaspar edited a comment on issue #16723:
URL: https://github.com/apache/superset/issues/16723#issuecomment-925839183
Hi @stockholmux, I'm a maintainer/creator of elasticsearch-dbapi. I'll look into including `opensearch-py` on elasticsearch-dbapi
Thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] harshgadhia commented on issue #16723: Support AWS OpenSearch / Elasticsearch
Posted by GitBox <gi...@apache.org>.
harshgadhia commented on issue #16723:
URL: https://github.com/apache/superset/issues/16723#issuecomment-961532592
@stockholmux I have some trouble using the elasticsearch-dbapi==0.2.6, Here is the detailed description of the issue: https://github.com/apache/superset/issues/17347
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] harshgadhia edited a comment on issue #16723: Support AWS OpenSearch / Elasticsearch
Posted by GitBox <gi...@apache.org>.
harshgadhia edited a comment on issue #16723:
URL: https://github.com/apache/superset/issues/16723#issuecomment-961532592
@stockholmux I have some trouble using the elasticsearch-dbapi==0.2.6, Here is the detailed description of the issue: https://github.com/apache/superset/issues/17347
Any help is greatly appreciated. Thanks in advance!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] harshgadhia commented on issue #16723: Support AWS OpenSearch / Elasticsearch
Posted by GitBox <gi...@apache.org>.
harshgadhia commented on issue #16723:
URL: https://github.com/apache/superset/issues/16723#issuecomment-961532592
@stockholmux I have some trouble using the elasticsearch-dbapi==0.2.6, Here is the detailed description of the issue: https://github.com/apache/superset/issues/17347
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org