You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@superset.apache.org by GitBox <gi...@apache.org> on 2022/09/08 13:15:39 UTC
[GitHub] [superset] zhaoyongjie opened a new pull request, #21378: fix: Time Grain in SQLite and Time Shift decopule from granularity
zhaoyongjie opened a new pull request, #21378:
URL: https://github.com/apache/superset/pull/21378
### SUMMARY
<!--- Describe the change below, including rationale and design decisions -->
TBD
### BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF
<!--- Skip this if not applicable -->
### TESTING INSTRUCTIONS
<!--- Required! What steps can be taken to manually verify the changes? -->
### ADDITIONAL INFORMATION
<!--- Check any relevant boxes with "x" -->
<!--- HINT: Include "Fixes #nnn" if you are fixing an existing issue -->
- [ ] Has associated issue:
- [ ] Required feature flags:
- [ ] Changes UI
- [ ] Includes DB Migration (follow approval process in [SIP-59](https://github.com/apache/superset/issues/13351))
- [ ] Migration is atomic, supports rollback & is backwards-compatible
- [ ] Confirm DB migration upgrade and downgrade tested
- [ ] Runtime estimates and downtime expectations provided
- [ ] Introduces new feature or API
- [ ] Removes existing feature or API
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] zhaoyongjie commented on a diff in pull request #21378: fix: Time Offset in SQLite
Posted by GitBox <gi...@apache.org>.
zhaoyongjie commented on code in PR #21378:
URL: https://github.com/apache/superset/pull/21378#discussion_r971093757
##########
superset/common/query_context_processor.py:
##########
@@ -245,11 +246,21 @@ def normalize_df(self, df: pd.DataFrame, query_object: QueryObject) -> pd.DataFr
if dttm_col:
timestamp_format = dttm_col.python_date_format
+ base_axis_label = None
+ if (
+ (label := get_base_axis_label(query_object.columns))
+ and query_object.datasource is not None
+ and (base_column := query_object.datasource.get_column(label))
+ and base_column.is_dttm
+ ):
+ base_axis_label = label
+
normalize_dttm_col(
df=df,
timestamp_format=timestamp_format,
offset=datasource.offset,
time_shift=query_object.time_shift,
+ dttm_alias=base_axis_label,
Review Comment:
nice! it's also treating multiple DateTime columns. I have addressed this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] zhaoyongjie commented on a diff in pull request #21378: fix: Time Offset in SQLite
Posted by GitBox <gi...@apache.org>.
zhaoyongjie commented on code in PR #21378:
URL: https://github.com/apache/superset/pull/21378#discussion_r971084059
##########
superset/common/query_context_processor.py:
##########
@@ -344,10 +354,7 @@ def processing_time_offsets( # pylint: disable=too-many-locals,too-many-stateme
offset_metrics_df = offset_metrics_df.rename(columns=metrics_mapping)
# 3. set time offset for index
- # TODO: add x-axis to QueryObject, potentially as an array for
- # multi-dimensional charts
- granularity = query_object.granularity
- index = granularity if granularity in df.columns else DTTM_ALIAS
+ index = (get_base_axis_labels(query_object.columns) or [DTTM_ALIAS])[0]
Review Comment:
`dataframe_utils.is_datetime_series` has processed `None` value case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] villebro commented on a diff in pull request #21378: fix: Time Offset in SQLite
Posted by GitBox <gi...@apache.org>.
villebro commented on code in PR #21378:
URL: https://github.com/apache/superset/pull/21378#discussion_r971577301
##########
superset/common/query_context_processor.py:
##########
@@ -245,11 +246,20 @@ def normalize_df(self, df: pd.DataFrame, query_object: QueryObject) -> pd.DataFr
if dttm_col:
timestamp_format = dttm_col.python_date_format
+ dttm_cols = tuple(
+ label
+ for label in get_base_axis_labels(query_object.columns)
+ if query_object.datasource
+ and (col := query_object.datasource.get_column(label))
+ and col.is_dttm
+ )
+
normalize_dttm_col(
df=df,
timestamp_format=timestamp_format,
offset=datasource.offset,
time_shift=query_object.time_shift,
+ dttm_cols=dttm_cols,
Review Comment:
As `timestamp_format` is defined per column, we would probably need to pass a `dict` in that maps the respective column to its `timestamp_format`. So instead of `dttm_cols` being `Optional[Tuple[str, ...]]`, maybe it should be a `Optional[Dict[str, Optional[str]]]` where the key would be the respective `timestamp_format` for said column?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] zhaoyongjie merged pull request #21378: fix: Time Offset in SQLite and refine logic in Date Type conversion
Posted by GitBox <gi...@apache.org>.
zhaoyongjie merged PR #21378:
URL: https://github.com/apache/superset/pull/21378
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] zhaoyongjie commented on a diff in pull request #21378: fix: Time Offset in SQLite
Posted by GitBox <gi...@apache.org>.
zhaoyongjie commented on code in PR #21378:
URL: https://github.com/apache/superset/pull/21378#discussion_r971086556
##########
superset/utils/core.py:
##########
@@ -1271,15 +1271,15 @@ def is_adhoc_column(column: Column) -> TypeGuard[AdhocColumn]:
return isinstance(column, dict)
-def get_base_axis_column(columns: Optional[List[Column]]) -> Optional[AdhocColumn]:
- if columns is None:
- return None
+def get_base_axis_label(columns: Optional[List[Column]]) -> Optional[str]:
Review Comment:
Good point, I have addressed these suggestions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] codecov[bot] commented on pull request #21378: [wip]fix: Time Grain in SQLite and Time Shift decopule from granularity
Posted by GitBox <gi...@apache.org>.
codecov[bot] commented on PR #21378:
URL: https://github.com/apache/superset/pull/21378#issuecomment-1240863248
# [Codecov](https://codecov.io/gh/apache/superset/pull/21378?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#21378](https://codecov.io/gh/apache/superset/pull/21378?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (f49bde3) into [master](https://codecov.io/gh/apache/superset/commit/b3028185225fdc8ee3eb8b9058f3867d1a28ea86?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (b302818) will **decrease** coverage by `11.48%`.
> The diff coverage is `81.72%`.
> :exclamation: Current head f49bde3 differs from pull request most recent head 69f2687. Consider uploading reports for the commit 69f2687 to get more accurate results
```diff
@@ Coverage Diff @@
## master #21378 +/- ##
===========================================
- Coverage 66.61% 55.12% -11.49%
===========================================
Files 1791 1791
Lines 68538 68565 +27
Branches 7310 7319 +9
===========================================
- Hits 45655 37797 -7858
- Misses 21002 28882 +7880
- Partials 1881 1886 +5
```
| Flag | Coverage Δ | |
|---|---|---|
| hive | `52.98% <50.00%> (+0.01%)` | :arrow_up: |
| mysql | `?` | |
| postgres | `?` | |
| presto | `?` | |
| python | `57.59% <54.54%> (-23.77%)` | :arrow_down: |
| sqlite | `?` | |
| unit | `50.82% <22.72%> (-0.01%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/superset/pull/21378?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...ntend/src/components/MetadataBar/ContentConfig.tsx](https://codecov.io/gh/apache/superset/pull/21378/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQtZnJvbnRlbmQvc3JjL2NvbXBvbmVudHMvTWV0YWRhdGFCYXIvQ29udGVudENvbmZpZy50c3g=) | `85.00% <ø> (ø)` | |
| [superset/config.py](https://codecov.io/gh/apache/superset/pull/21378/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvY29uZmlnLnB5) | `90.96% <ø> (-0.65%)` | :arrow_down: |
| [superset/datasets/api.py](https://codecov.io/gh/apache/superset/pull/21378/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvZGF0YXNldHMvYXBpLnB5) | `49.59% <ø> (-37.71%)` | :arrow_down: |
| [superset/common/query\_context\_processor.py](https://codecov.io/gh/apache/superset/pull/21378/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvY29tbW9uL3F1ZXJ5X2NvbnRleHRfcHJvY2Vzc29yLnB5) | `62.66% <40.00%> (-26.04%)` | :arrow_down: |
| [superset/utils/core.py](https://codecov.io/gh/apache/superset/pull/21378/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvdXRpbHMvY29yZS5weQ==) | `63.12% <58.82%> (-26.88%)` | :arrow_down: |
| [...ard/components/DrillDetailPane/DrillDetailPane.tsx](https://codecov.io/gh/apache/superset/pull/21378/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQtZnJvbnRlbmQvc3JjL2Rhc2hib2FyZC9jb21wb25lbnRzL0RyaWxsRGV0YWlsUGFuZS9EcmlsbERldGFpbFBhbmUudHN4) | `81.01% <76.19%> (-1.75%)` | :arrow_down: |
| [...rontend/src/components/MetadataBar/MetadataBar.tsx](https://codecov.io/gh/apache/superset/pull/21378/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQtZnJvbnRlbmQvc3JjL2NvbXBvbmVudHMvTWV0YWRhdGFCYXIvTWV0YWRhdGFCYXIudHN4) | `96.00% <96.00%> (ø)` | |
| [superset/utils/dashboard\_import\_export.py](https://codecov.io/gh/apache/superset/pull/21378/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvdXRpbHMvZGFzaGJvYXJkX2ltcG9ydF9leHBvcnQucHk=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [superset/key\_value/commands/update.py](https://codecov.io/gh/apache/superset/pull/21378/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQva2V5X3ZhbHVlL2NvbW1hbmRzL3VwZGF0ZS5weQ==) | `0.00% <0.00%> (-88.89%)` | :arrow_down: |
| [superset/key\_value/commands/delete.py](https://codecov.io/gh/apache/superset/pull/21378/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQva2V5X3ZhbHVlL2NvbW1hbmRzL2RlbGV0ZS5weQ==) | `0.00% <0.00%> (-85.30%)` | :arrow_down: |
| ... and [287 more](https://codecov.io/gh/apache/superset/pull/21378/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
:mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] zhaoyongjie commented on a diff in pull request #21378: fix: Time Offset in SQLite
Posted by GitBox <gi...@apache.org>.
zhaoyongjie commented on code in PR #21378:
URL: https://github.com/apache/superset/pull/21378#discussion_r971601407
##########
superset/common/query_context_processor.py:
##########
@@ -245,11 +246,20 @@ def normalize_df(self, df: pd.DataFrame, query_object: QueryObject) -> pd.DataFr
if dttm_col:
timestamp_format = dttm_col.python_date_format
+ dttm_cols = tuple(
+ label
+ for label in get_base_axis_labels(query_object.columns)
+ if query_object.datasource
+ and (col := query_object.datasource.get_column(label))
+ and col.is_dttm
+ )
+
normalize_dttm_col(
df=df,
timestamp_format=timestamp_format,
offset=datasource.offset,
time_shift=query_object.time_shift,
+ dttm_cols=dttm_cols,
Review Comment:
Thanks for pointing that out. I am going to refactor `normalize_dttm_col`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] villebro commented on a diff in pull request #21378: fix: Time Offset in SQLite
Posted by GitBox <gi...@apache.org>.
villebro commented on code in PR #21378:
URL: https://github.com/apache/superset/pull/21378#discussion_r970841998
##########
superset/utils/core.py:
##########
@@ -1271,15 +1271,15 @@ def is_adhoc_column(column: Column) -> TypeGuard[AdhocColumn]:
return isinstance(column, dict)
-def get_base_axis_column(columns: Optional[List[Column]]) -> Optional[AdhocColumn]:
- if columns is None:
- return None
+def get_base_axis_label(columns: Optional[List[Column]]) -> Optional[str]:
Review Comment:
Could this be:
```python
def get_base_axis_labels(columns: Optional[List[Column]]) -> List[str]:
```
i.e. an empty list if there are none, and otherwise all axis column labels?
##########
superset/common/query_context_processor.py:
##########
@@ -245,11 +246,21 @@ def normalize_df(self, df: pd.DataFrame, query_object: QueryObject) -> pd.DataFr
if dttm_col:
timestamp_format = dttm_col.python_date_format
+ base_axis_label = None
+ if (
+ (label := get_base_axis_label(query_object.columns))
+ and query_object.datasource is not None
+ and (base_column := query_object.datasource.get_column(label))
+ and base_column.is_dttm
+ ):
+ base_axis_label = label
+
normalize_dttm_col(
df=df,
timestamp_format=timestamp_format,
offset=datasource.offset,
time_shift=query_object.time_shift,
+ dttm_alias=base_axis_label,
Review Comment:
In reference to my comment about changing `get_base_axis_label` to plural form, could we loop through all `base_axis_labels` here to future proof for multi-dimensional charts?
##########
superset/utils/core.py:
##########
@@ -1850,28 +1853,31 @@ def normalize_dttm_col(
timestamp_format: Optional[str],
offset: int,
time_shift: Optional[timedelta],
+ dttm_alias: Optional[str] = DTTM_ALIAS,
Review Comment:
nit: could we call this `dttm_col` to be more in line with the method name?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] villebro commented on a diff in pull request #21378: fix: Time Offset in SQLite
Posted by GitBox <gi...@apache.org>.
villebro commented on code in PR #21378:
URL: https://github.com/apache/superset/pull/21378#discussion_r970963527
##########
superset/utils/core.py:
##########
@@ -1850,28 +1853,31 @@ def normalize_dttm_col(
timestamp_format: Optional[str],
offset: int,
time_shift: Optional[timedelta],
+ dttm_alias: Optional[str] = DTTM_ALIAS,
Review Comment:
nit: could we call this `dttm_col`, `column_name` or other to be more in line with the method name?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org