You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@superset.apache.org by GitBox <gi...@apache.org> on 2022/08/09 16:39:33 UTC
[GitHub] [superset] hughhhh opened a new pull request, #21029: fix: take form_data reference for metrics for pivot_v2 table reports
hughhhh opened a new pull request, #21029:
URL: https://github.com/apache/superset/pull/21029
<!---
Please write the PR title following the conventions at https://www.conventionalcommits.org/en/v1.0.0/
Example:
fix(dashboard): load charts correctly
-->
### SUMMARY
<!--- Describe the change below, including rationale and design decisions -->
When executing pivot v2 table report if the `verbose_map` doesn't match up with the df columns we are throwing this error for metric/values. To fix this for pivot v2 table we won't be using the verbose map to change the values/metrics names to have better alignment.
```
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1516, in full_dispatch_request
rv = self.dispatch_request()
File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1502, in dispatch_request
return self.ensure_sync(self.view_functions[rule.endpoint])(*req.view_args)
File "/usr/local/lib/python3.8/site-packages/flask_appbuilder/security/decorators.py", line 89, in wraps
return f(self, *args, *kwargs)
File "/usr/local/lib/python3.8/site-packages/superset/views/base_api.py", line 113, in wraps
raise ex
File "/usr/local/lib/python3.8/site-packages/superset/views/base_api.py", line 110, in wraps
duration, response = time_function(f, self, args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/superset/utils/core.py", line 1533, in time_function
response = func(*args, *kwargs)
File "/usr/local/lib/python3.8/site-packages/superset/utils/log.py", line 244, in wrapper
value = f(args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/superset/charts/data/api.py", line 160, in get_data
return self._get_data_response(
File "/usr/local/lib/python3.8/site-packages/superset/charts/data/api.py", line 396, in _get_data_response
return self._send_chart_response(result, form_data, datasource)
File "/usr/local/lib/python3.8/site-packages/superset/charts/data/api.py", line 343, in _send_chart_response
result = apply_post_process(result, form_data, datasource)
File "/usr/local/lib/python3.8/site-packages/superset/charts/post_processing.py", line 339, in apply_post_process
processed_df = post_processor(df, form_data, datasource)
File "/usr/local/lib/python3.8/site-packages/superset/charts/post_processing.py", line 234, in pivot_table_v2
return pivot_df(
File "/usr/local/lib/python3.8/site-packages/superset/charts/post_processing.py", line 89, in pivot_df
df = df.pivot_table(
File "/usr/local/lib/python3.8/site-packages/pandas/core/frame.py", line 7951, in pivot_table
return pivot_table(
File "/usr/local/lib/python3.8/site-packages/pandas/core/reshape/pivot.py", line 95, in pivot_table
table = __internal_pivot_table(
File "/usr/local/lib/python3.8/site-packages/pandas/core/reshape/pivot.py", line 141, in __internal_pivot_table
raise KeyError(i)
KeyError: 'COUNT()'
```
[Click here to view exception in pandas](https://github.com/pandas-dev/pandas/commit/8f7ba1baa7514379dbb0a70d6b032fbef9a88eb2)
### BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF
<!--- Skip this if not applicable -->
### TESTING INSTRUCTIONS
<!--- Required! What steps can be taken to manually verify the changes? -->
### ADDITIONAL INFORMATION
<!--- Check any relevant boxes with "x" -->
<!--- HINT: Include "Fixes #nnn" if you are fixing an existing issue -->
- [ ] Has associated issue:
- [ ] Required feature flags:
- [ ] Changes UI
- [ ] Includes DB Migration (follow approval process in [SIP-59](https://github.com/apache/superset/issues/13351))
- [ ] Migration is atomic, supports rollback & is backwards-compatible
- [ ] Confirm DB migration upgrade and downgrade tested
- [ ] Runtime estimates and downtime expectations provided
- [ ] Introduces new feature or API
- [ ] Removes existing feature or API
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] betodealmeida commented on a diff in pull request #21029: fix: take form_data reference for metrics for pivot_v2 table reports
Posted by GitBox <gi...@apache.org>.
betodealmeida commented on code in PR #21029:
URL: https://github.com/apache/superset/pull/21029#discussion_r942902080
##########
tests/unit_tests/charts/test_post_processing.py:
##########
@@ -1960,3 +1961,64 @@ def test_apply_post_process_json_format_data_is_none():
assert apply_post_process(result, form_data) == {
"queries": [{"result_format": ChartDataResultFormat.JSON, "data": None}]
}
+
+
+def test_apply_post_process_verbose_map(session: Session):
+ from superset.connectors.sqla.models import SqlaTable, SqlMetric
+ from superset.models.core import Database
+
+ engine = session.get_bind()
+ SqlaTable.metadata.create_all(engine) # pylint: disable=no-member
+ db = Database(database_name="my_database", sqlalchemy_uri="sqlite://")
+ sqla_table = SqlaTable(
+ table_name="my_sqla_table",
+ columns=[],
+ metrics=[
+ SqlMetric(
+ metric_name="count",
+ verbose_name="COUNT(*)",
+ metric_type="count",
+ expression="COUNT(*)",
+ )
+ ],
+ database=db,
+ )
+
+ result = {
+ "queries": [
+ {
+ "result_format": ChartDataResultFormat.JSON,
+ "data": [{"count": 4725}],
+ }
+ ]
+ }
+ form_data = {
+ "datasource": "19__table",
+ "viz_type": "pivot_table_v2",
+ "slice_id": 69,
+ "url_params": {},
+ "granularity_sqla": "time_start",
+ "time_grain_sqla": "P1D",
+ "time_range": "No filter",
+ "groupbyColumns": [],
+ "groupbyRows": [],
+ "metrics": ["COUNT(*)"],
+ "metricsLayout": "COLUMNS",
+ "row_limit": 10000,
+ "order_desc": True,
+ "valueFormat": "SMART_NUMBER",
+ "date_format": "smart_date",
+ "rowOrder": "key_a_to_z",
+ "colOrder": "key_a_to_z",
+ "extra_form_data": {},
+ "force": False,
+ "result_format": "json",
+ "result_type": "results",
+ }
+
+ assert (
+ "COUNT(*)"
+ in apply_post_process(result, form_data, datasource=sqla_table)["queries"][0][
+ "data"
+ ].keys()
+ )
Review Comment:
Nit: I would make this assertion more generic, since it might be useful to capture future regressions in other parts of the code. Something like:
```python
assert apply_post_process(result, form_data, datasource=sqla_table) = {
"queries": [
{
"data": { "COUNT(*)": ..., ... },
},
]
)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] hughhhh merged pull request #21029: fix: take form_data reference for metrics for pivot_v2 table reports
Posted by GitBox <gi...@apache.org>.
hughhhh merged PR #21029:
URL: https://github.com/apache/superset/pull/21029
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] codecov[bot] commented on pull request #21029: fix: take form_data reference for metrics for pivot_v2 table reports
Posted by GitBox <gi...@apache.org>.
codecov[bot] commented on PR #21029:
URL: https://github.com/apache/superset/pull/21029#issuecomment-1209629180
# [Codecov](https://codecov.io/gh/apache/superset/pull/21029?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#21029](https://codecov.io/gh/apache/superset/pull/21029?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (6f1d516) into [master](https://codecov.io/gh/apache/superset/commit/665007622845cf4303b50e57d11549a162dbbb1b?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (6650076) will **decrease** coverage by `11.66%`.
> The diff coverage is `n/a`.
```diff
@@ Coverage Diff @@
## master #21029 +/- ##
===========================================
- Coverage 66.35% 54.68% -11.67%
===========================================
Files 1767 1767
Lines 67356 67356
Branches 7147 7147
===========================================
- Hits 44694 36835 -7859
- Misses 20834 28693 +7859
Partials 1828 1828
```
| Flag | Coverage Δ | |
|---|---|---|
| hive | `?` | |
| mysql | `?` | |
| postgres | `?` | |
| presto | `53.05% <ø> (ø)` | |
| python | `57.42% <ø> (-24.08%)` | :arrow_down: |
| sqlite | `?` | |
| unit | `50.47% <ø> (ø)` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/superset/pull/21029?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [superset/charts/post\_processing.py](https://codecov.io/gh/apache/superset/pull/21029/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvY2hhcnRzL3Bvc3RfcHJvY2Vzc2luZy5weQ==) | `89.55% <ø> (ø)` | |
| [superset/utils/dashboard\_import\_export.py](https://codecov.io/gh/apache/superset/pull/21029/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvdXRpbHMvZGFzaGJvYXJkX2ltcG9ydF9leHBvcnQucHk=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [superset/key\_value/commands/update.py](https://codecov.io/gh/apache/superset/pull/21029/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQva2V5X3ZhbHVlL2NvbW1hbmRzL3VwZGF0ZS5weQ==) | `0.00% <0.00%> (-88.89%)` | :arrow_down: |
| [superset/key\_value/commands/delete.py](https://codecov.io/gh/apache/superset/pull/21029/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQva2V5X3ZhbHVlL2NvbW1hbmRzL2RlbGV0ZS5weQ==) | `0.00% <0.00%> (-85.30%)` | :arrow_down: |
| [superset/db\_engines/hive.py](https://codecov.io/gh/apache/superset/pull/21029/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvZGJfZW5naW5lcy9oaXZlLnB5) | `0.00% <0.00%> (-85.19%)` | :arrow_down: |
| [superset/key\_value/commands/delete\_expired.py](https://codecov.io/gh/apache/superset/pull/21029/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQva2V5X3ZhbHVlL2NvbW1hbmRzL2RlbGV0ZV9leHBpcmVkLnB5) | `0.00% <0.00%> (-80.77%)` | :arrow_down: |
| [superset/dashboards/commands/importers/v0.py](https://codecov.io/gh/apache/superset/pull/21029/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvZGFzaGJvYXJkcy9jb21tYW5kcy9pbXBvcnRlcnMvdjAucHk=) | `15.62% <0.00%> (-76.25%)` | :arrow_down: |
| [superset/datasets/commands/update.py](https://codecov.io/gh/apache/superset/pull/21029/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvZGF0YXNldHMvY29tbWFuZHMvdXBkYXRlLnB5) | `25.00% <0.00%> (-69.05%)` | :arrow_down: |
| [superset/datasets/commands/create.py](https://codecov.io/gh/apache/superset/pull/21029/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvZGF0YXNldHMvY29tbWFuZHMvY3JlYXRlLnB5) | `29.41% <0.00%> (-68.63%)` | :arrow_down: |
| [superset/datasets/commands/importers/v0.py](https://codecov.io/gh/apache/superset/pull/21029/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvZGF0YXNldHMvY29tbWFuZHMvaW1wb3J0ZXJzL3YwLnB5) | `24.03% <0.00%> (-67.45%)` | :arrow_down: |
| ... and [280 more](https://codecov.io/gh/apache/superset/pull/21029/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
:mega: Codecov can now indicate which changes are the most critical in Pull Requests. [Learn more](https://about.codecov.io/product/feature/runtime-insights/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] betodealmeida commented on a diff in pull request #21029: fix: take form_data reference for metrics for pivot_v2 table reports
Posted by GitBox <gi...@apache.org>.
betodealmeida commented on code in PR #21029:
URL: https://github.com/apache/superset/pull/21029#discussion_r942721968
##########
superset/charts/post_processing.py:
##########
@@ -336,6 +336,10 @@ def apply_post_process(
elif query["result_format"] == ChartDataResultFormat.CSV:
df = pd.read_csv(StringIO(query["data"]))
+ # convert metrics to verbose (label) name
Review Comment:
```suggestion
# convert all columns to verbose (label) name
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org