You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@superset.apache.org by "tullis (via GitHub)" <gi...@apache.org> on 2023/09/25 16:13:10 UTC
[GitHub] [superset] tullis opened a new issue, #25397: KeyError: 'druid' after upgrade to version 2.0.1
tullis opened a new issue, #25397:
URL: https://github.com/apache/superset/issues/25397
We have a Superset instance with Presto, Druid, and MySQL databases included.
The production instance is currently at version 1.5.3
Upgrading to version 2.0.1 resulted in the following error in the server's log.
```
Sep 25 13:48:48 an-tool1005 superset[929767]: 2023-09-25 13:48:48,951:ERROR:superset.views.base:'druid'
Sep 25 13:48:48 an-tool1005 superset[929767]: Traceback (most recent call last):
Sep 25 13:48:48 an-tool1005 superset[929767]: File "/srv/deployment/analytics/superset/venv/lib/python3.9/site-packages/superset/views/base.py", line 210, in wraps
Sep 25 13:48:48 an-tool1005 superset[929767]: return f(self, *args, **kwargs)
Sep 25 13:48:48 an-tool1005 superset[929767]: File "/srv/deployment/analytics/superset/venv/lib/python3.9/site-packages/superset/views/base_api.py", line 423, in get_list_headless
Sep 25 13:48:48 an-tool1005 superset[929767]: duration, response = time_function(super().get_list_headless, **kwargs)
Sep 25 13:48:48 an-tool1005 superset[929767]: File "/srv/deployment/analytics/superset/venv/lib/python3.9/site-packages/superset/utils/core.py", line 1524, in time_function
Sep 25 13:48:48 an-tool1005 superset[929767]: response = func(*args, **kwargs)
Sep 25 13:48:48 an-tool1005 superset[929767]: File "/srv/deployment/analytics/superset/venv/lib/python3.9/site-packages/flask_appbuilder/api/__init__.py", line 1526, in get_list_headless
Sep 25 13:48:48 an-tool1005 superset[929767]: _response[API_RESULT_RES_KEY] = _list_model_schema.dump(lst, many=True)
Sep 25 13:48:48 an-tool1005 superset[929767]: File "/srv/deployment/analytics/superset/venv/lib/python3.9/site-packages/marshmallow/schema.py", line 552, in dump
Sep 25 13:48:48 an-tool1005 superset[929767]: result = self._serialize(processed_obj, many=many)
Sep 25 13:48:48 an-tool1005 superset[929767]: File "/srv/deployment/analytics/superset/venv/lib/python3.9/site-packages/marshmallow/schema.py", line 514, in _serialize
Sep 25 13:48:48 an-tool1005 superset[929767]: return [
Sep 25 13:48:48 an-tool1005 superset[929767]: File "/srv/deployment/analytics/superset/venv/lib/python3.9/site-packages/marshmallow/schema.py", line 515, in <listcomp>
Sep 25 13:48:48 an-tool1005 superset[929767]: self._serialize(d, many=False)
Sep 25 13:48:48 an-tool1005 superset[929767]: File "/srv/deployment/analytics/superset/venv/lib/python3.9/site-packages/marshmallow/schema.py", line 520, in _serialize
Sep 25 13:48:48 an-tool1005 superset[929767]: value = field_obj.serialize(attr_name, obj, accessor=self.get_attribute)
Sep 25 13:48:48 an-tool1005 superset[929767]: File "/srv/deployment/analytics/superset/venv/lib/python3.9/site-packages/marshmallow/fields.py", line 338, in serialize
Sep 25 13:48:48 an-tool1005 superset[929767]: return self._serialize(value, attr, obj, **kwargs)
Sep 25 13:48:48 an-tool1005 superset[929767]: File "/srv/deployment/analytics/superset/venv/lib/python3.9/site-packages/marshmallow/fields.py", line 1922, in _serialize
Sep 25 13:48:48 an-tool1005 superset[929767]: return self._call_or_raise(self.serialize_func, obj, attr)
Sep 25 13:48:48 an-tool1005 superset[929767]: File "/srv/deployment/analytics/superset/venv/lib/python3.9/site-packages/marshmallow/fields.py", line 1936, in _call_or_raise
Sep 25 13:48:48 an-tool1005 superset[929767]: return func(value)
Sep 25 13:48:48 an-tool1005 superset[929767]: File "/srv/deployment/analytics/superset/venv/lib/python3.9/site-packages/superset/models/slice.py", line 164, in datasource_url
Sep 25 13:48:48 an-tool1005 superset[929767]: datasource = self.datasource
Sep 25 13:48:48 an-tool1005 superset[929767]: File "/srv/deployment/analytics/superset/venv/lib/python3.9/site-packages/superset/models/slice.py", line 133, in datasource
Sep 25 13:48:48 an-tool1005 superset[929767]: return self.get_datasource
Sep 25 13:48:48 an-tool1005 superset[929767]: File "/srv/deployment/analytics/superset/venv/lib/python3.9/site-packages/superset/utils/memoized.py", line 50, in __call__
Sep 25 13:48:48 an-tool1005 superset[929767]: value = self.func(*args, **kwargs)
Sep 25 13:48:48 an-tool1005 superset[929767]: File "/srv/deployment/analytics/superset/venv/lib/python3.9/site-packages/superset/models/slice.py", line 151, in get_datasource
Sep 25 13:48:48 an-tool1005 superset[929767]: return db.session.query(self.cls_model).filter_by(id=self.datasource_id).first()
Sep 25 13:48:48 an-tool1005 superset[929767]: File "/srv/deployment/analytics/superset/venv/lib/python3.9/site-packages/superset/models/slice.py", line 129, in cls_model
Sep 25 13:48:48 an-tool1005 superset[929767]: return ConnectorRegistry.sources[self.datasource_type]
Sep 25 13:48:48 an-tool1005 superset[929767]: KeyError: 'druid'
```
- browser type and version: n/a
- superset version: 2.0.1
- python version: 3.9.2
- node.js version: 16
- any feature flags active:
```
"ENABLE_TEMPLATE_PROCESSING": True,
"DASHBOARD_NATIVE_FILTERS": True,
"ENABLE_FILTER_BOX_MIGRATION": True,
```
### Screenshots
Also it will not show the list of Charts available.

### Checklist
Make sure to follow these steps before submitting your issue - thank you!
- [x] I have checked the superset logs for python stacktraces and included it here as text if there are any.
- [ ] I have reproduced the issue with at least the latest released version of superset. n/a I cannot upgrade to 2.1.1 because of [#23483](https://github.com/apache/superset/issues/23483)
- [x] I have checked the issue tracker for the same issue and I haven't found one similar.
### Additional context
Here is the output from `pip list`
```
(venv) superset@an-tool1005:~$ pip list
Package Version
---------------------- ---------
aiohttp 3.8.1
aiosignal 1.2.0
alembic 1.6.5
amqp 5.1.0
apache-superset 2.0.1
apispec 3.3.2
async-timeout 4.0.2
attrs 21.2.0
Babel 2.9.1
backoff 1.11.1
billiard 3.6.4.0
bleach 3.3.1
boto3 1.28.53
botocore 1.31.53
Brotli 1.0.9
cached-property 1.5.2
cachelib 0.4.1
celery 5.2.2
certifi 2023.7.22
cffi 1.14.6
chardet 5.2.0
charset-normalizer 2.0.4
click 8.0.4
click-didyoumean 0.3.0
click-plugins 1.1.1
click-repl 0.2.0
colorama 0.4.4
convertdate 2.4.0
cron-descriptor 1.2.24
croniter 1.0.15
cryptography 3.4.7
decorator 5.1.1
deprecation 2.1.0
dnspython 2.1.0
email-validator 1.1.3
et-xmlfile 1.1.0
Flask 2.0.3
Flask-AppBuilder 4.1.3
Flask-Babel 1.0.0
Flask-Caching 1.10.1
Flask-Compress 1.10.1
Flask-JWT-Extended 4.3.1
Flask-Login 0.4.1
Flask-Migrate 3.1.0
Flask-SQLAlchemy 2.5.1
flask-talisman 0.8.1
Flask-WTF 0.14.3
frozenlist 1.3.0
func-timeout 4.3.5
future 0.18.3
geographiclib 1.52
geopy 2.2.0
gevent 23.9.1
graphlib-backport 1.0.3
greenlet 2.0.2
gssapi 1.8.3
gunicorn 20.1.0
hashids 1.3.1
hijri-converter 2.3.1
holidays 0.14.2
humanize 3.11.0
idna 3.2
ijson 3.2.3
isodate 0.6.0
itsdangerous 2.1.1
Jinja2 3.0.3
jmespath 1.0.1
jsonlines 4.0.0
jsonschema 3.2.0
kombu 5.2.4
korean-lunar-calendar 0.2.1
krb5 0.5.1
linear-tsv 1.1.0
Mako 1.1.4
Markdown 3.3.4
MarkupSafe 2.0.1
marshmallow 3.13.0
marshmallow-enum 1.5.1
marshmallow-sqlalchemy 0.23.1
msgpack 1.0.2
multidict 5.1.0
mysqlclient 2.2.0
numpy 1.22.1
openpyxl 3.1.2
packaging 21.3
pandas 1.3.4
parsedatetime 2.6
pgsanity 0.2.9
pip 23.2.1
pkg_resources 0.0.0
polyline 1.4.0
prison 0.2.1
prompt-toolkit 3.0.28
psycopg2-binary 2.9.1
pure-sasl 0.6.2
pyarrow 5.0.0
pycparser 2.20
pydruid 0.6.5
PyHive 0.7.0
PyJWT 2.4.0
pylibmc 1.6.1
PyMeeus 0.5.11
pyparsing 3.0.6
pyrsistent 0.16.1
pyspnego 0.9.2
python-dateutil 2.8.2
python-dotenv 0.19.0
python-editor 1.0.4
python-geohash 0.8.5
pytz 2021.3
PyYAML 5.4.1
redis 3.5.3
requests 2.31.0
requests-kerberos 0.14.0
rfc3986 2.0.0
s3transfer 0.6.2
sasl 0.3.1
selenium 3.141.0
setuptools 68.2.2
simplejson 3.17.3
six 1.16.0
slackclient 2.5.0
SQLAlchemy 1.3.24
SQLAlchemy-Utils 0.37.8
sqlparse 0.3.0
tableschema 1.20.2
tabulate 0.8.9
tabulator 1.53.5
thrift 0.16.0
thrift-sasl 0.4.3
trino 0.326.0
typing-extensions 3.10.0.0
tzlocal 5.0.1
unicodecsv 0.14.1
urllib3 1.26.6
vine 5.0.0
wcwidth 0.2.5
webencodings 0.5.1
Werkzeug 2.0.3
wheel 0.34.2
WTForms 2.3.3
WTForms-JSON 0.3.3
xlrd 2.0.1
yarl 1.6.3
zope.event 5.0
zope.interface 6.0
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [I] KeyError: 'druid' after upgrade to version 2.0.1 [superset]
Posted by "eschutho (via GitHub)" <gi...@apache.org>.
eschutho commented on issue #25397:
URL: https://github.com/apache/superset/issues/25397#issuecomment-1749622944
Hi @tullis. It looks like you have a chart that is using a deprecated/unsupported data type of 'druid' which was from the nosql druid implementation, now removed. Do you have the ability to run an adhoc query on your metadata? If so, a quick answer would be to change any `slice.datasource_type` values from 'druid' to 'table' in your metadata db. The other option is for us to write a migration to update this for you, but it's going to require updating to the next minor release when it comes out, and I'd rather see if I can unblock you sooner rather than later if possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [I] KeyError: 'druid' after upgrade to version 2.0.1 [superset]
Posted by "tullis (via GitHub)" <gi...@apache.org>.
tullis commented on issue #25397:
URL: https://github.com/apache/superset/issues/25397#issuecomment-1751019592
@eschutho - Many thanks for that. Yes, I think you are right. I ran the following ad-hoc command and the error has now gone.
```
Welcome to the MariaDB monitor. Commands end with ; or \g.
Your MariaDB connection id is 330001
Server version: 10.4.18-MariaDB-log MariaDB Server
Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
MariaDB [(none)]> use superset_staging;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A
Database changed
MariaDB [superset_staging]> select count(*) from slices where datasource_type='druid';
+----------+
| count(*) |
+----------+
| 513 |
+----------+
1 row in set (0.000 sec)
MariaDB [superset_staging]> update slices set datasource_type='table' where datasource_type='druid';
Query OK, 513 rows affected (0.050 sec)
Rows matched: 513 Changed: 513 Warnings: 0
MariaDB [superset_staging]>
```
You're right that we used to use the nosql version of Druid for this installation and I believe that the metadata db has been migrated in-place since version 0.20.6.
I'm checking to make sure that all of the charts we expect to see are still there after this ad-hoc command, but I'd say that you have likely unblocked us for now and demonstrated that this migration is worth adding.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [I] KeyError: 'druid' after upgrade to version 2.0.1 [superset]
Posted by "eschutho (via GitHub)" <gi...@apache.org>.
eschutho closed issue #25397: KeyError: 'druid' after upgrade to version 2.0.1
URL: https://github.com/apache/superset/issues/25397
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [I] KeyError: 'druid' after upgrade to version 2.0.1 [superset]
Posted by "tullis (via GitHub)" <gi...@apache.org>.
tullis commented on issue #25397:
URL: https://github.com/apache/superset/issues/25397#issuecomment-1757367856
> The other option is for us to write a migration to update this for you, but it's going to require updating to the next minor release when it comes out, and I'd rather see if I can unblock you sooner rather than later if possible.
Would you consider writing this migration anyway, as this legacy type information might still make this issue affect other users?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [I] KeyError: 'druid' after upgrade to version 2.0.1 [superset]
Posted by "eschutho (via GitHub)" <gi...@apache.org>.
eschutho commented on issue #25397:
URL: https://github.com/apache/superset/issues/25397#issuecomment-1756353417
Ok, perfect. I'll close this issue for now, but feel free to comment or reopen if you need anything else.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org