You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@superset.apache.org by GitBox <gi...@apache.org> on 2020/01/10 16:43:06 UTC
[GitHub] [incubator-superset] robdiciuccio opened a new pull request #8946:
Ensure proper JSON serialization of numpy.ndarray
robdiciuccio opened a new pull request #8946: Ensure proper JSON serialization of numpy.ndarray
URL: https://github.com/apache/incubator-superset/pull/8946
### CATEGORY
Choose one
- [X] Bug Fix
- [ ] Enhancement (new features, refinement)
- [ ] Refactor
- [ ] Add tests
- [ ] Build / Development Environment
- [ ] Documentation
### SUMMARY
When querying a Presto datasource with a column of type `ARRAY` asynchronously, the following error would be raised:
```
TypeError: Unserializable object [] of type <class 'numpy.ndarray'>
```
This fix ensures proper conversion of the `numpy.ndarray` to a list for JSON serialization.
### BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF
<!--- Skip this if not applicable -->
### TEST PLAN
Ensure queries against a Presto table containing nested data type columns work correctly. Repeat for other supported DBs having nested data type columns.
This fix was tested against Presto 0.227 in both async and sync queries.
### ADDITIONAL INFORMATION
<!--- Check any relevant boxes with "x" -->
<!--- HINT: Include "Fixes #nnn" if you are fixing an existing issue -->
- [ ] Has associated issue:
- [ ] Changes UI
- [ ] Requires DB Migration.
- [ ] Confirm DB Migration upgrade and downgrade tested.
- [ ] Introduces new feature or API
- [ ] Removes existing feature or API
### REVIEWERS
@graceguo-supercat @craig-rueda @dpgaspar @villebro
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [incubator-superset] robdiciuccio commented on issue #8946: Ensure
proper JSON serialization of numpy.ndarray
Posted by GitBox <gi...@apache.org>.
robdiciuccio commented on issue #8946: Ensure proper JSON serialization of numpy.ndarray
URL: https://github.com/apache/incubator-superset/pull/8946#issuecomment-573924221
@graceguo-supercat I tested with the following table and data in our Presto (Hive) cluster, and was unable to replicate the error.
```
CREATE TABLE default.superset_test (
id bigint,
name varchar,
list_ints array(bigint),
list_maps array(map(varchar, varchar)),
map1 map(varchar, varchar),
map2 map(varchar, boolean),
ds date
);
INSERT INTO default.superset_test VALUES (1, 'San Francisco', ARRAY[1,2,3], ARRAY[MAP(ARRAY['foo'], ARRAY['bar']), MAP(ARRAY['fad'], ARRAY['baz'])], MAP(ARRAY['foo'], ARRAY['bar']), MAP(ARRAY['foo'], ARRAY[True]), date '2019-01-09');
select * from default.superset_test;
```
<img width="791" alt="Screen Shot 2020-01-13 at 3 29 56 PM" src="https://user-images.githubusercontent.com/296227/72300764-9465c000-3619-11ea-8402-94de5549782a.png">
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [incubator-superset] mistercrunch merged pull request #8946: Ensure
proper JSON serialization of numpy.ndarray
Posted by GitBox <gi...@apache.org>.
mistercrunch merged pull request #8946: Ensure proper JSON serialization of numpy.ndarray
URL: https://github.com/apache/incubator-superset/pull/8946
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [incubator-superset] graceguo-supercat commented on issue #8946:
Ensure proper JSON serialization of numpy.ndarray
Posted by GitBox <gi...@apache.org>.
graceguo-supercat commented on issue #8946: Ensure proper JSON serialization of numpy.ndarray
URL: https://github.com/apache/incubator-superset/pull/8946#issuecomment-573261231
Hi @robdiciuccio Thanks very much for the fix. I run a few queries in different tables. Here is another error message also related to `pyarrow`:
```
File "/usr/local/lib/python3.6/dist-packages/flask_cors/extension.py", line 161, in wrapped_function
return cors_after_request(app.make_response(f(*args, **kwargs)))
File "/usr/local/lib/python3.6/dist-packages/newrelic/hooks/framework_flask.py", line 108, in _nr_wrapper_Flask_handle_exception_
return wrapped(*args, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/flask/app.py", line 1820, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/usr/local/lib/python3.6/dist-packages/flask/_compat.py", line 39, in reraise
raise value
File "/usr/local/lib/python3.6/dist-packages/flask/app.py", line 1949, in full_dispatch_request
rv = self.dispatch_request()
File "/usr/local/lib/python3.6/dist-packages/flask/app.py", line 1935, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "/usr/local/lib/python3.6/dist-packages/newrelic/hooks/framework_flask.py", line 45, in _nr_wrapper_handler_
return wrapped(*args, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/flask_appbuilder/security/decorators.py", line 168, in wraps
return f(self, *args, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/superset/utils/log.py", line 59, in wrapper
value = f(*args, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/superset/views/core.py", line 2291, in results
return self.results_exec(key)
File "/usr/local/lib/python3.6/dist-packages/superset/views/core.py", line 2331, in results_exec
payload, query, cast(bool, results_backend_use_msgpack)
File "/usr/local/lib/python3.6/dist-packages/superset/views/core.py", line 233, in _deserialize_results_payload
df = result_set.SupersetResultSet.convert_table_to_df(pa_table)
File "/usr/local/lib/python3.6/dist-packages/superset/result_set.py", line 129, in convert_table_to_df
return table.to_pandas(integer_object_nulls=True)
File "pyarrow/array.pxi", line 468, in pyarrow.lib._PandasConvertible.to_pandas
File "pyarrow/table.pxi", line 1238, in pyarrow.lib.Table._to_pandas
File "/usr/local/lib/python3.6/dist-packages/pyarrow/pandas_compat.py", line 704, in table_to_blockmanager
blocks = _table_to_blocks(options, table, categories)
File "/usr/local/lib/python3.6/dist-packages/pyarrow/pandas_compat.py", line 976, in _table_to_blocks
result = pa.lib.table_to_blocks(options, block_table, categories)
File "pyarrow/table.pxi", line 780, in pyarrow.lib.table_to_blocks
File "pyarrow/error.pxi", line 86, in pyarrow.lib.check_status
pyarrow.lib.ArrowNotImplementedError: Not implemented type for list in DataFrameBlock: struct<author_type: string, locale: string>
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [incubator-superset] robdiciuccio commented on issue #8946: Ensure
proper JSON serialization of numpy.ndarray
Posted by GitBox <gi...@apache.org>.
robdiciuccio commented on issue #8946: Ensure proper JSON serialization of numpy.ndarray
URL: https://github.com/apache/incubator-superset/pull/8946#issuecomment-573489315
Thanks @graceguo-supercat. Is this in Presto? What are the data types of the columns being queried?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [incubator-superset] codecov-io commented on issue #8946: Ensure
proper JSON serialization of numpy.ndarray
Posted by GitBox <gi...@apache.org>.
codecov-io commented on issue #8946: Ensure proper JSON serialization of numpy.ndarray
URL: https://github.com/apache/incubator-superset/pull/8946#issuecomment-573122211
# [Codecov](https://codecov.io/gh/apache/incubator-superset/pull/8946?src=pr&el=h1) Report
> Merging [#8946](https://codecov.io/gh/apache/incubator-superset/pull/8946?src=pr&el=desc) into [master](https://codecov.io/gh/apache/incubator-superset/commit/3bedee75d37af0ed137a14e3f739ed3e09cc227d?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/incubator-superset/pull/8946?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #8946 +/- ##
=======================================
Coverage 58.97% 58.97%
=======================================
Files 359 359
Lines 11333 11333
Branches 2787 2787
=======================================
Hits 6684 6684
Misses 4471 4471
Partials 178 178
```
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-superset/pull/8946?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/incubator-superset/pull/8946?src=pr&el=footer). Last update [3bedee7...8926e22](https://codecov.io/gh/apache/incubator-superset/pull/8946?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [incubator-superset] graceguo-supercat commented on issue #8946:
Ensure proper JSON serialization of numpy.ndarray
Posted by GitBox <gi...@apache.org>.
graceguo-supercat commented on issue #8946: Ensure proper JSON serialization of numpy.ndarray
URL: https://github.com/apache/incubator-superset/pull/8946#issuecomment-574454658
@robdiciuccio Do you test this PR on Hive `Struct` data type? otherwise it may break extra cases.
cc @john-bodley
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [incubator-superset] graceguo-supercat edited a comment on issue
#8946: Ensure proper JSON serialization of numpy.ndarray
Posted by GitBox <gi...@apache.org>.
graceguo-supercat edited a comment on issue #8946: Ensure proper JSON serialization of numpy.ndarray
URL: https://github.com/apache/incubator-superset/pull/8946#issuecomment-574454658
@robdiciuccio Do you test this PR with Hive `Struct` data type? otherwise it may break extra cases.
cc @john-bodley
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [incubator-superset] graceguo-supercat commented on issue #8946:
Ensure proper JSON serialization of numpy.ndarray
Posted by GitBox <gi...@apache.org>.
graceguo-supercat commented on issue #8946: Ensure proper JSON serialization of numpy.ndarray
URL: https://github.com/apache/incubator-superset/pull/8946#issuecomment-573821514
Presto version=0.188.
I feel the problem might be caused by `MAP<STRING,BOOLEAN>` type or `ARRAY<MAP<STRING,STRING>>`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org