You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@superset.apache.org by "tullis (via GitHub)" <gi...@apache.org> on 2023/09/25 16:13:10 UTC

[GitHub] [superset] tullis opened a new issue, #25397: KeyError: 'druid' after upgrade to version 2.0.1

tullis opened a new issue, #25397:
URL: https://github.com/apache/superset/issues/25397

   We have a Superset instance with Presto, Druid, and MySQL databases included.
   
   The production instance is currently at version 1.5.3
   
   Upgrading to version 2.0.1 resulted in the following error in the server's log.
   ```
   Sep 25 13:48:48 an-tool1005 superset[929767]: 2023-09-25 13:48:48,951:ERROR:superset.views.base:'druid'
   Sep 25 13:48:48 an-tool1005 superset[929767]: Traceback (most recent call last):
   Sep 25 13:48:48 an-tool1005 superset[929767]:   File "/srv/deployment/analytics/superset/venv/lib/python3.9/site-packages/superset/views/base.py", line 210, in wraps
   Sep 25 13:48:48 an-tool1005 superset[929767]:     return f(self, *args, **kwargs)
   Sep 25 13:48:48 an-tool1005 superset[929767]:   File "/srv/deployment/analytics/superset/venv/lib/python3.9/site-packages/superset/views/base_api.py", line 423, in get_list_headless
   Sep 25 13:48:48 an-tool1005 superset[929767]:     duration, response = time_function(super().get_list_headless, **kwargs)
   Sep 25 13:48:48 an-tool1005 superset[929767]:   File "/srv/deployment/analytics/superset/venv/lib/python3.9/site-packages/superset/utils/core.py", line 1524, in time_function
   Sep 25 13:48:48 an-tool1005 superset[929767]:     response = func(*args, **kwargs)
   Sep 25 13:48:48 an-tool1005 superset[929767]:   File "/srv/deployment/analytics/superset/venv/lib/python3.9/site-packages/flask_appbuilder/api/__init__.py", line 1526, in get_list_headless
   Sep 25 13:48:48 an-tool1005 superset[929767]:     _response[API_RESULT_RES_KEY] = _list_model_schema.dump(lst, many=True)
   Sep 25 13:48:48 an-tool1005 superset[929767]:   File "/srv/deployment/analytics/superset/venv/lib/python3.9/site-packages/marshmallow/schema.py", line 552, in dump
   Sep 25 13:48:48 an-tool1005 superset[929767]:     result = self._serialize(processed_obj, many=many)
   Sep 25 13:48:48 an-tool1005 superset[929767]:   File "/srv/deployment/analytics/superset/venv/lib/python3.9/site-packages/marshmallow/schema.py", line 514, in _serialize
   Sep 25 13:48:48 an-tool1005 superset[929767]:     return [
   Sep 25 13:48:48 an-tool1005 superset[929767]:   File "/srv/deployment/analytics/superset/venv/lib/python3.9/site-packages/marshmallow/schema.py", line 515, in <listcomp>
   Sep 25 13:48:48 an-tool1005 superset[929767]:     self._serialize(d, many=False)
   Sep 25 13:48:48 an-tool1005 superset[929767]:   File "/srv/deployment/analytics/superset/venv/lib/python3.9/site-packages/marshmallow/schema.py", line 520, in _serialize
   Sep 25 13:48:48 an-tool1005 superset[929767]:     value = field_obj.serialize(attr_name, obj, accessor=self.get_attribute)
   Sep 25 13:48:48 an-tool1005 superset[929767]:   File "/srv/deployment/analytics/superset/venv/lib/python3.9/site-packages/marshmallow/fields.py", line 338, in serialize
   Sep 25 13:48:48 an-tool1005 superset[929767]:     return self._serialize(value, attr, obj, **kwargs)
   Sep 25 13:48:48 an-tool1005 superset[929767]:   File "/srv/deployment/analytics/superset/venv/lib/python3.9/site-packages/marshmallow/fields.py", line 1922, in _serialize
   Sep 25 13:48:48 an-tool1005 superset[929767]:     return self._call_or_raise(self.serialize_func, obj, attr)
   Sep 25 13:48:48 an-tool1005 superset[929767]:   File "/srv/deployment/analytics/superset/venv/lib/python3.9/site-packages/marshmallow/fields.py", line 1936, in _call_or_raise
   Sep 25 13:48:48 an-tool1005 superset[929767]:     return func(value)
   Sep 25 13:48:48 an-tool1005 superset[929767]:   File "/srv/deployment/analytics/superset/venv/lib/python3.9/site-packages/superset/models/slice.py", line 164, in datasource_url
   Sep 25 13:48:48 an-tool1005 superset[929767]:     datasource = self.datasource
   Sep 25 13:48:48 an-tool1005 superset[929767]:   File "/srv/deployment/analytics/superset/venv/lib/python3.9/site-packages/superset/models/slice.py", line 133, in datasource
   Sep 25 13:48:48 an-tool1005 superset[929767]:     return self.get_datasource
   Sep 25 13:48:48 an-tool1005 superset[929767]:   File "/srv/deployment/analytics/superset/venv/lib/python3.9/site-packages/superset/utils/memoized.py", line 50, in __call__
   Sep 25 13:48:48 an-tool1005 superset[929767]:     value = self.func(*args, **kwargs)
   Sep 25 13:48:48 an-tool1005 superset[929767]:   File "/srv/deployment/analytics/superset/venv/lib/python3.9/site-packages/superset/models/slice.py", line 151, in get_datasource
   Sep 25 13:48:48 an-tool1005 superset[929767]:     return db.session.query(self.cls_model).filter_by(id=self.datasource_id).first()
   Sep 25 13:48:48 an-tool1005 superset[929767]:   File "/srv/deployment/analytics/superset/venv/lib/python3.9/site-packages/superset/models/slice.py", line 129, in cls_model
   Sep 25 13:48:48 an-tool1005 superset[929767]:     return ConnectorRegistry.sources[self.datasource_type]
   Sep 25 13:48:48 an-tool1005 superset[929767]: KeyError: 'druid'
   ```
   
   - browser type and version: n/a
   - superset version: 2.0.1
   - python version: 3.9.2
   - node.js version: 16
   - any feature flags active:
   ```
   "ENABLE_TEMPLATE_PROCESSING": True,
   "DASHBOARD_NATIVE_FILTERS": True,
   "ENABLE_FILTER_BOX_MIGRATION": True,
   ```
   ### Screenshots
   Also it will not show the list of Charts available.
   
   ![image](https://github.com/apache/superset/assets/2117946/30a9a1be-eb04-4edf-9e90-d4cd23f685f7)
   
   ### Checklist
   
   Make sure to follow these steps before submitting your issue - thank you!
   
   - [x] I have checked the superset logs for python stacktraces and included it here as text if there are any.
   - [ ] I have reproduced the issue with at least the latest released version of superset. n/a I cannot upgrade to 2.1.1 because of [#23483](https://github.com/apache/superset/issues/23483)
   - [x] I have checked the issue tracker for the same issue and I haven't found one similar.
   
   ### Additional context
   
   Here is the output from `pip list`
   ```
   (venv) superset@an-tool1005:~$ pip list
   Package                Version
   ---------------------- ---------
   aiohttp                3.8.1
   aiosignal              1.2.0
   alembic                1.6.5
   amqp                   5.1.0
   apache-superset        2.0.1
   apispec                3.3.2
   async-timeout          4.0.2
   attrs                  21.2.0
   Babel                  2.9.1
   backoff                1.11.1
   billiard               3.6.4.0
   bleach                 3.3.1
   boto3                  1.28.53
   botocore               1.31.53
   Brotli                 1.0.9
   cached-property        1.5.2
   cachelib               0.4.1
   celery                 5.2.2
   certifi                2023.7.22
   cffi                   1.14.6
   chardet                5.2.0
   charset-normalizer     2.0.4
   click                  8.0.4
   click-didyoumean       0.3.0
   click-plugins          1.1.1
   click-repl             0.2.0
   colorama               0.4.4
   convertdate            2.4.0
   cron-descriptor        1.2.24
   croniter               1.0.15
   cryptography           3.4.7
   decorator              5.1.1
   deprecation            2.1.0
   dnspython              2.1.0
   email-validator        1.1.3
   et-xmlfile             1.1.0
   Flask                  2.0.3
   Flask-AppBuilder       4.1.3
   Flask-Babel            1.0.0
   Flask-Caching          1.10.1
   Flask-Compress         1.10.1
   Flask-JWT-Extended     4.3.1
   Flask-Login            0.4.1
   Flask-Migrate          3.1.0
   Flask-SQLAlchemy       2.5.1
   flask-talisman         0.8.1
   Flask-WTF              0.14.3
   frozenlist             1.3.0
   func-timeout           4.3.5
   future                 0.18.3
   geographiclib          1.52
   geopy                  2.2.0
   gevent                 23.9.1
   graphlib-backport      1.0.3
   greenlet               2.0.2
   gssapi                 1.8.3
   gunicorn               20.1.0
   hashids                1.3.1
   hijri-converter        2.3.1
   holidays               0.14.2
   humanize               3.11.0
   idna                   3.2
   ijson                  3.2.3
   isodate                0.6.0
   itsdangerous           2.1.1
   Jinja2                 3.0.3
   jmespath               1.0.1
   jsonlines              4.0.0
   jsonschema             3.2.0
   kombu                  5.2.4
   korean-lunar-calendar  0.2.1
   krb5                   0.5.1
   linear-tsv             1.1.0
   Mako                   1.1.4
   Markdown               3.3.4
   MarkupSafe             2.0.1
   marshmallow            3.13.0
   marshmallow-enum       1.5.1
   marshmallow-sqlalchemy 0.23.1
   msgpack                1.0.2
   multidict              5.1.0
   mysqlclient            2.2.0
   numpy                  1.22.1
   openpyxl               3.1.2
   packaging              21.3
   pandas                 1.3.4
   parsedatetime          2.6
   pgsanity               0.2.9
   pip                    23.2.1
   pkg_resources          0.0.0
   polyline               1.4.0
   prison                 0.2.1
   prompt-toolkit         3.0.28
   psycopg2-binary        2.9.1
   pure-sasl              0.6.2
   pyarrow                5.0.0
   pycparser              2.20
   pydruid                0.6.5
   PyHive                 0.7.0
   PyJWT                  2.4.0
   pylibmc                1.6.1
   PyMeeus                0.5.11
   pyparsing              3.0.6
   pyrsistent             0.16.1
   pyspnego               0.9.2
   python-dateutil        2.8.2
   python-dotenv          0.19.0
   python-editor          1.0.4
   python-geohash         0.8.5
   pytz                   2021.3
   PyYAML                 5.4.1
   redis                  3.5.3
   requests               2.31.0
   requests-kerberos      0.14.0
   rfc3986                2.0.0
   s3transfer             0.6.2
   sasl                   0.3.1
   selenium               3.141.0
   setuptools             68.2.2
   simplejson             3.17.3
   six                    1.16.0
   slackclient            2.5.0
   SQLAlchemy             1.3.24
   SQLAlchemy-Utils       0.37.8
   sqlparse               0.3.0
   tableschema            1.20.2
   tabulate               0.8.9
   tabulator              1.53.5
   thrift                 0.16.0
   thrift-sasl            0.4.3
   trino                  0.326.0
   typing-extensions      3.10.0.0
   tzlocal                5.0.1
   unicodecsv             0.14.1
   urllib3                1.26.6
   vine                   5.0.0
   wcwidth                0.2.5
   webencodings           0.5.1
   Werkzeug               2.0.3
   wheel                  0.34.2
   WTForms                2.3.3
   WTForms-JSON           0.3.3
   xlrd                   2.0.1
   yarl                   1.6.3
   zope.event             5.0
   zope.interface         6.0
   ```
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [I] KeyError: 'druid' after upgrade to version 2.0.1 [superset]

Posted by "eschutho (via GitHub)" <gi...@apache.org>.
eschutho commented on issue #25397:
URL: https://github.com/apache/superset/issues/25397#issuecomment-1749622944

   Hi @tullis. It looks like you have a chart that is using a deprecated/unsupported data type of 'druid' which was from the nosql druid implementation, now removed. Do you have the ability to run an adhoc query on your metadata? If so, a quick answer would be to change any `slice.datasource_type` values from 'druid' to 'table' in your metadata db. The other option is for us to write a migration to update this for you, but it's going to require updating to the next minor release when it comes out, and I'd rather see if I can unblock you sooner rather than later if possible.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [I] KeyError: 'druid' after upgrade to version 2.0.1 [superset]

Posted by "tullis (via GitHub)" <gi...@apache.org>.
tullis commented on issue #25397:
URL: https://github.com/apache/superset/issues/25397#issuecomment-1751019592

   @eschutho - Many thanks for that. Yes, I think you are right. I ran the following ad-hoc command and the error has now gone.
   ```
   Welcome to the MariaDB monitor.  Commands end with ; or \g.
   Your MariaDB connection id is 330001
   Server version: 10.4.18-MariaDB-log MariaDB Server
   
   Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.
   
   Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
   
   MariaDB [(none)]> use superset_staging;
   Reading table information for completion of table and column names
   You can turn off this feature to get a quicker startup with -A
   
   Database changed
   MariaDB [superset_staging]> select count(*) from slices where datasource_type='druid';
   +----------+
   | count(*) |
   +----------+
   |      513 |
   +----------+
   1 row in set (0.000 sec)
   
   MariaDB [superset_staging]> update slices set datasource_type='table' where datasource_type='druid';
   Query OK, 513 rows affected (0.050 sec)
   Rows matched: 513  Changed: 513  Warnings: 0
   
   MariaDB [superset_staging]> 
   ```
   You're right that we used to use the nosql version of Druid for this installation and I believe that the metadata db has been migrated in-place since version 0.20.6.
   
   I'm checking to make sure that all of the charts we expect to see are still there after this ad-hoc command, but I'd say that you have likely unblocked us for now and demonstrated that this migration is worth adding.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [I] KeyError: 'druid' after upgrade to version 2.0.1 [superset]

Posted by "eschutho (via GitHub)" <gi...@apache.org>.
eschutho closed issue #25397: KeyError: 'druid' after upgrade to version 2.0.1
URL: https://github.com/apache/superset/issues/25397


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [I] KeyError: 'druid' after upgrade to version 2.0.1 [superset]

Posted by "tullis (via GitHub)" <gi...@apache.org>.
tullis commented on issue #25397:
URL: https://github.com/apache/superset/issues/25397#issuecomment-1757367856

   > The other option is for us to write a migration to update this for you, but it's going to require updating to the next minor release when it comes out, and I'd rather see if I can unblock you sooner rather than later if possible.
   
   Would you consider writing this migration anyway, as this legacy type information might still make this issue affect other users?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [I] KeyError: 'druid' after upgrade to version 2.0.1 [superset]

Posted by "eschutho (via GitHub)" <gi...@apache.org>.
eschutho commented on issue #25397:
URL: https://github.com/apache/superset/issues/25397#issuecomment-1756353417

   Ok, perfect. I'll close this issue for now, but feel free to comment or reopen if you need anything else. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org