You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@superset.apache.org by GitBox <gi...@apache.org> on 2021/07/27 12:00:53 UTC
[GitHub] [superset] erickg-k opened a new issue #15906: Unable to explore dataset from Opendistro Elasticsearch because of timestamp format
erickg-k opened a new issue #15906:
URL: https://github.com/apache/superset/issues/15906
I try to explore an index in Superset dataset (/superset/explore/). The database connector reports an error. I tried to fix the datetime format in the setting but it didn't work.
### Expected results
To be able to see the data as configured.
### Actual results
Superset reports an error ElasticsearchParseException[failed to parse date field [2021-07-20T00:00:00] with format [yyyy-MM-dd'T'HH:mm:ss.SSSSSS].
```Error: (Error occurred in Elasticsearch engine: all shards failed): Shard[0]: ElasticsearchParseException[failed to parse date field [2021-07-20T00:00:00] with format [yyyy-MM-dd'T'HH:mm:ss.SSSSSS]: [failed to parse date field [2021-07-20T00:00:00] with format [yyyy-MM-dd'T'HH:mm:ss.SSSSSS]]]; nested: IllegalArgumentException[failed to parse date field [2021-07-20T00:00:00] with format [yyyy-MM-dd'T'HH:mm:ss.SSSSSS]]; nested: DateTimeParseException[Text '2021-07-20T00:00:00' could not be parsed at index 19]; Shard[1]: ElasticsearchParseException[failed to parse date field [2021-07-20T00:00:00] with format [yyyy-MM-dd'T'HH:mm:ss.SSSSSS]: [failed to parse date field [2021-07-20T00:00:00] with format [yyyy-MM-dd'T'HH:mm:ss.SSSSSS]]]; nested: IllegalArgumentException[failed to parse date field [2021-07-20T00:00:00] with format [yyyy-MM-dd'T'HH:mm:ss.SSSSSS]]; nested: NotSerializableExceptionWrapper[date_time_parse_exception: Text '2021-07-20T00:00:00' could not be parsed at index 1
9]; Shard[2]: ElasticsearchParseException[failed to parse date field [2021-07-20T00:00:00] with format [yyyy-MM-dd'T'HH:mm:ss.SSSSSS]: [failed to parse date field [2021-07-20T00:00:00] with format [yyyy-MM-dd'T'HH:mm:ss.SSSSSS]]]; nested: IllegalArgumentException[failed to parse date field [2021-07-20T00:00:00] with format [yyyy-MM-dd'T'HH:mm:ss.SSSSSS]]; nested: NotSerializableExceptionWrapper[date_time_parse_exception: Text '2021-07-20T00:00:00' could not be parsed at index 19]; Shard[3]: ElasticsearchParseException[failed to parse date field [2021-07-20T00:00:00] with format [yyyy-MM-dd'T'HH:mm:ss.SSSSSS]: [failed to parse date field [2021-07-20T00:00:00] with format [yyyy-MM-dd'T'HH:mm:ss.SSSSSS]]]; nested: IllegalArgumentException[failed to parse date field [2021-07-20T00:00:00] with format [yyyy-MM-dd'T'HH:mm:ss.SSSSSS]]; nested: NotSerializableExceptionWrapper[date_time_parse_exception: Text '2021-07-20T00:00:00' could not be parsed at index 19]; For more details, please send
request for Json format to see the raw response from elasticsearch engine.
```
#### Screenshots
#### How to reproduce the bug
1. Set up ES index with a field of datetime with {
"type": "date",
"format": "yyyy-MM-dd'T'HH:mm:ss.SSSSSS'Z'"
},
2. Insert some data into ES
3. Setup superset with elasticsearch-dbapi==0.2.4
3. Create database in Superset and wait it to fetch column information
4. Edit dataset columns:
* The column should be marked as DATETIME and Is temporal.
* Set datetime format as `%Y-%m-%dT%H:%M:%S.%fZ`
5. Run explore
You should get error logs:
```
Traceback (most recent call last):
File "/superset/.venv/lib/python3.8/site-packages/superset/connectors/sqla/models.py", line 1454, in query
df = self.database.get_df(sql, self.schema, mutator=assign_column_label)
File "/superset/.venv/lib/python3.8/site-packages/superset/models/core.py", line 388, in get_df
self.db_engine_spec.execute(cursor, sqls[-1])
File "/superset/.venv/lib/python3.8/site-packages/superset/db_engine_specs/base.py", line 1064, in execute
raise cls.get_dbapi_mapped_exception(ex)
File "/superset/.venv/lib/python3.8/site-packages/superset/db_engine_specs/base.py", line 1062, in execute
cursor.execute(query)
File "/superset/.venv/lib/python3.8/site-packages/es/baseapi.py", line 37, in wrap
return f(self, *args, **kwargs)
File "/superset/.venv/lib/python3.8/site-packages/es/opendistro/api.py", line 276, in execute
results = self.elastic_query(query)
File "/superset/.venv/lib/python3.8/site-packages/es/baseapi.py", line 325, in elastic_query
raise exceptions.ProgrammingError(
es.exceptions.ProgrammingError: (Error occurred in Elasticsearch engine: all shards failed): Shard[0]: ElasticsearchParseException[failed to parse date field [2021-07-20T00:00:00] with format [yyyy-MM-dd'T'HH:mm:ss.SSSSSS]: [failed to parse date field [2021-07-20T00:00:00] with format [yyyy-MM-dd'T'HH:mm:ss.SSSSSS]]]; nested: IllegalArgumentException[failed to parse date field [2021-07-20T00:00:00] with format [yyyy-MM-dd'T'HH:mm:ss.SSSSSS]]; nested: NotSerializableExceptionWrapper[date_time_parse_exception: Text '2021-07-20T00:00:00' could not be parsed at index 19];
Shard[1]: ElasticsearchParseException[failed to parse date field [2021-07-20T00:00:00] with format [yyyy-MM-dd'T'HH:mm:ss.SSSSSS]: [failed to parse date field [2021-07-20T00:00:00] with format [yyyy-MM-dd'T'HH:mm:ss.SSSSSS]]]; nested: IllegalArgumentException[failed to parse date field [2021-07-20T00:00:00] with format [yyyy-MM-dd'T'HH:mm:ss.SSSSSS]]; nested: DateTimeParseException[Text '2021-07-20T00:00:00' could not be parsed at index 19];
Shard[2]: ElasticsearchParseException[failed to parse date field [2021-07-20T00:00:00] with format [yyyy-MM-dd'T'HH:mm:ss.SSSSSS]: [failed to parse date field [2021-07-20T00:00:00] with format [yyyy-MM-dd'T'HH:mm:ss.SSSSSS]]]; nested: IllegalArgumentException[failed to parse date field [2021-07-20T00:00:00] with format [yyyy-MM-dd'T'HH:mm:ss.SSSSSS]]; nested: NotSerializableExceptionWrapper[date_time_parse_exception: Text '2021-07-20T00:00:00' could not be parsed at index 19];
Shard[3]: ElasticsearchParseException[failed to parse date field [2021-07-20T00:00:00] with format [yyyy-MM-dd'T'HH:mm:ss.SSSSSS]: [failed to parse date field [2021-07-20T00:00:00] with format [yyyy-MM-dd'T'HH:mm:ss.SSSSSS]]]; nested: IllegalArgumentException[failed to parse date field [2021-07-20T00:00:00] with format [yyyy-MM-dd'T'HH:mm:ss.SSSSSS]]; nested: NotSerializableExceptionWrapper[date_time_parse_exception: Text '2021-07-20T00:00:00' could not be parsed at index 19];
For more details, please send request for Json format to see the raw response from elasticsearch engine.
Query SELECT COUNT(*) AS count
FROM "default".offerings
WHERE updated_at >= '2021-07-20T00:00:00'
AND updated_at < '2021-07-27T00:00:00'
ORDER BY count DESC
LIMIT 10000 on schema default failed
method: POST
timestamp: 2021-07-27T11:18:02.695954+00:00
url: /api/v1/chart/data
```
### Environment
- superset version: 1.2.0
- python version: 3.8.8
- node.js version: not installed
### Checklist
Make sure to follow these steps before submitting your issue - thank you!
- [x] I have checked the superset logs for python stacktraces and included it here as text if there are any.
- [x] I have reproduced the issue with at least the latest released version of superset.
- [x] I have checked the issue tracker for the same issue and I haven't found one similar.
### Additional context
I setup a dockerfile with superset and its db connectors with pip.
requirements.in with latest pip:
apache-superset[cors]
elasticsearch-dbapi[opendistro]
shillelagh[gsheetsapi]
pyhive
gunicorn[gevent]>=20.0.0,<21
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org