You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@superset.apache.org by GitBox <gi...@apache.org> on 2022/09/08 13:15:39 UTC

[GitHub] [superset] zhaoyongjie opened a new pull request, #21378: fix: Time Grain in SQLite and Time Shift decopule from granularity

zhaoyongjie opened a new pull request, #21378:
URL: https://github.com/apache/superset/pull/21378

   ### SUMMARY
   <!--- Describe the change below, including rationale and design decisions -->
   TBD
   
   ### BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF
   <!--- Skip this if not applicable -->
   
   ### TESTING INSTRUCTIONS
   <!--- Required! What steps can be taken to manually verify the changes? -->
   
   ### ADDITIONAL INFORMATION
   <!--- Check any relevant boxes with "x" -->
   <!--- HINT: Include "Fixes #nnn" if you are fixing an existing issue -->
   - [ ] Has associated issue:
   - [ ] Required feature flags:
   - [ ] Changes UI
   - [ ] Includes DB Migration (follow approval process in [SIP-59](https://github.com/apache/superset/issues/13351))
     - [ ] Migration is atomic, supports rollback & is backwards-compatible
     - [ ] Confirm DB migration upgrade and downgrade tested
     - [ ] Runtime estimates and downtime expectations provided
   - [ ] Introduces new feature or API
   - [ ] Removes existing feature or API
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


[GitHub] [superset] zhaoyongjie commented on a diff in pull request #21378: fix: Time Offset in SQLite

Posted by GitBox <gi...@apache.org>.
zhaoyongjie commented on code in PR #21378:
URL: https://github.com/apache/superset/pull/21378#discussion_r971093757


##########
superset/common/query_context_processor.py:
##########
@@ -245,11 +246,21 @@ def normalize_df(self, df: pd.DataFrame, query_object: QueryObject) -> pd.DataFr
             if dttm_col:
                 timestamp_format = dttm_col.python_date_format
 
+        base_axis_label = None
+        if (
+            (label := get_base_axis_label(query_object.columns))
+            and query_object.datasource is not None
+            and (base_column := query_object.datasource.get_column(label))
+            and base_column.is_dttm
+        ):
+            base_axis_label = label
+
         normalize_dttm_col(
             df=df,
             timestamp_format=timestamp_format,
             offset=datasource.offset,
             time_shift=query_object.time_shift,
+            dttm_alias=base_axis_label,

Review Comment:
   nice! it's also treating multiple DateTime columns. I have addressed this.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


[GitHub] [superset] zhaoyongjie commented on a diff in pull request #21378: fix: Time Offset in SQLite

Posted by GitBox <gi...@apache.org>.
zhaoyongjie commented on code in PR #21378:
URL: https://github.com/apache/superset/pull/21378#discussion_r971084059


##########
superset/common/query_context_processor.py:
##########
@@ -344,10 +354,7 @@ def processing_time_offsets(  # pylint: disable=too-many-locals,too-many-stateme
                 offset_metrics_df = offset_metrics_df.rename(columns=metrics_mapping)
 
                 # 3. set time offset for index
-                # TODO: add x-axis to QueryObject, potentially as an array for
-                #  multi-dimensional charts
-                granularity = query_object.granularity
-                index = granularity if granularity in df.columns else DTTM_ALIAS
+                index = (get_base_axis_labels(query_object.columns) or [DTTM_ALIAS])[0]

Review Comment:
   `dataframe_utils.is_datetime_series` has processed `None` value case.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


[GitHub] [superset] villebro commented on a diff in pull request #21378: fix: Time Offset in SQLite

Posted by GitBox <gi...@apache.org>.
villebro commented on code in PR #21378:
URL: https://github.com/apache/superset/pull/21378#discussion_r971577301


##########
superset/common/query_context_processor.py:
##########
@@ -245,11 +246,20 @@ def normalize_df(self, df: pd.DataFrame, query_object: QueryObject) -> pd.DataFr
             if dttm_col:
                 timestamp_format = dttm_col.python_date_format
 
+        dttm_cols = tuple(
+            label
+            for label in get_base_axis_labels(query_object.columns)
+            if query_object.datasource
+            and (col := query_object.datasource.get_column(label))
+            and col.is_dttm
+        )
+
         normalize_dttm_col(
             df=df,
             timestamp_format=timestamp_format,
             offset=datasource.offset,
             time_shift=query_object.time_shift,
+            dttm_cols=dttm_cols,

Review Comment:
   As `timestamp_format` is defined per column, we would probably need to pass a `dict` in that maps the respective column to its `timestamp_format`. So instead of `dttm_cols` being `Optional[Tuple[str, ...]]`, maybe it should be a `Optional[Dict[str, Optional[str]]]` where the key would be the respective `timestamp_format` for said column?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


[GitHub] [superset] zhaoyongjie merged pull request #21378: fix: Time Offset in SQLite and refine logic in Date Type conversion

Posted by GitBox <gi...@apache.org>.
zhaoyongjie merged PR #21378:
URL: https://github.com/apache/superset/pull/21378


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


[GitHub] [superset] zhaoyongjie commented on a diff in pull request #21378: fix: Time Offset in SQLite

Posted by GitBox <gi...@apache.org>.
zhaoyongjie commented on code in PR #21378:
URL: https://github.com/apache/superset/pull/21378#discussion_r971086556


##########
superset/utils/core.py:
##########
@@ -1271,15 +1271,15 @@ def is_adhoc_column(column: Column) -> TypeGuard[AdhocColumn]:
     return isinstance(column, dict)
 
 
-def get_base_axis_column(columns: Optional[List[Column]]) -> Optional[AdhocColumn]:
-    if columns is None:
-        return None
+def get_base_axis_label(columns: Optional[List[Column]]) -> Optional[str]:

Review Comment:
   Good point, I have addressed these suggestions.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


[GitHub] [superset] codecov[bot] commented on pull request #21378: [wip]fix: Time Grain in SQLite and Time Shift decopule from granularity

Posted by GitBox <gi...@apache.org>.
codecov[bot] commented on PR #21378:
URL: https://github.com/apache/superset/pull/21378#issuecomment-1240863248

   # [Codecov](https://codecov.io/gh/apache/superset/pull/21378?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#21378](https://codecov.io/gh/apache/superset/pull/21378?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (f49bde3) into [master](https://codecov.io/gh/apache/superset/commit/b3028185225fdc8ee3eb8b9058f3867d1a28ea86?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (b302818) will **decrease** coverage by `11.48%`.
   > The diff coverage is `81.72%`.
   
   > :exclamation: Current head f49bde3 differs from pull request most recent head 69f2687. Consider uploading reports for the commit 69f2687 to get more accurate results
   
   ```diff
   @@             Coverage Diff             @@
   ##           master   #21378       +/-   ##
   ===========================================
   - Coverage   66.61%   55.12%   -11.49%     
   ===========================================
     Files        1791     1791               
     Lines       68538    68565       +27     
     Branches     7310     7319        +9     
   ===========================================
   - Hits        45655    37797     -7858     
   - Misses      21002    28882     +7880     
   - Partials     1881     1886        +5     
   ```
   
   | Flag | Coverage Δ | |
   |---|---|---|
   | hive | `52.98% <50.00%> (+0.01%)` | :arrow_up: |
   | mysql | `?` | |
   | postgres | `?` | |
   | presto | `?` | |
   | python | `57.59% <54.54%> (-23.77%)` | :arrow_down: |
   | sqlite | `?` | |
   | unit | `50.82% <22.72%> (-0.01%)` | :arrow_down: |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   | [Impacted Files](https://codecov.io/gh/apache/superset/pull/21378?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [...ntend/src/components/MetadataBar/ContentConfig.tsx](https://codecov.io/gh/apache/superset/pull/21378/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQtZnJvbnRlbmQvc3JjL2NvbXBvbmVudHMvTWV0YWRhdGFCYXIvQ29udGVudENvbmZpZy50c3g=) | `85.00% <ø> (ø)` | |
   | [superset/config.py](https://codecov.io/gh/apache/superset/pull/21378/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvY29uZmlnLnB5) | `90.96% <ø> (-0.65%)` | :arrow_down: |
   | [superset/datasets/api.py](https://codecov.io/gh/apache/superset/pull/21378/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvZGF0YXNldHMvYXBpLnB5) | `49.59% <ø> (-37.71%)` | :arrow_down: |
   | [superset/common/query\_context\_processor.py](https://codecov.io/gh/apache/superset/pull/21378/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvY29tbW9uL3F1ZXJ5X2NvbnRleHRfcHJvY2Vzc29yLnB5) | `62.66% <40.00%> (-26.04%)` | :arrow_down: |
   | [superset/utils/core.py](https://codecov.io/gh/apache/superset/pull/21378/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvdXRpbHMvY29yZS5weQ==) | `63.12% <58.82%> (-26.88%)` | :arrow_down: |
   | [...ard/components/DrillDetailPane/DrillDetailPane.tsx](https://codecov.io/gh/apache/superset/pull/21378/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQtZnJvbnRlbmQvc3JjL2Rhc2hib2FyZC9jb21wb25lbnRzL0RyaWxsRGV0YWlsUGFuZS9EcmlsbERldGFpbFBhbmUudHN4) | `81.01% <76.19%> (-1.75%)` | :arrow_down: |
   | [...rontend/src/components/MetadataBar/MetadataBar.tsx](https://codecov.io/gh/apache/superset/pull/21378/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQtZnJvbnRlbmQvc3JjL2NvbXBvbmVudHMvTWV0YWRhdGFCYXIvTWV0YWRhdGFCYXIudHN4) | `96.00% <96.00%> (ø)` | |
   | [superset/utils/dashboard\_import\_export.py](https://codecov.io/gh/apache/superset/pull/21378/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvdXRpbHMvZGFzaGJvYXJkX2ltcG9ydF9leHBvcnQucHk=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | [superset/key\_value/commands/update.py](https://codecov.io/gh/apache/superset/pull/21378/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQva2V5X3ZhbHVlL2NvbW1hbmRzL3VwZGF0ZS5weQ==) | `0.00% <0.00%> (-88.89%)` | :arrow_down: |
   | [superset/key\_value/commands/delete.py](https://codecov.io/gh/apache/superset/pull/21378/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQva2V5X3ZhbHVlL2NvbW1hbmRzL2RlbGV0ZS5weQ==) | `0.00% <0.00%> (-85.30%)` | :arrow_down: |
   | ... and [287 more](https://codecov.io/gh/apache/superset/pull/21378/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
   
   :mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


[GitHub] [superset] zhaoyongjie commented on a diff in pull request #21378: fix: Time Offset in SQLite

Posted by GitBox <gi...@apache.org>.
zhaoyongjie commented on code in PR #21378:
URL: https://github.com/apache/superset/pull/21378#discussion_r971601407


##########
superset/common/query_context_processor.py:
##########
@@ -245,11 +246,20 @@ def normalize_df(self, df: pd.DataFrame, query_object: QueryObject) -> pd.DataFr
             if dttm_col:
                 timestamp_format = dttm_col.python_date_format
 
+        dttm_cols = tuple(
+            label
+            for label in get_base_axis_labels(query_object.columns)
+            if query_object.datasource
+            and (col := query_object.datasource.get_column(label))
+            and col.is_dttm
+        )
+
         normalize_dttm_col(
             df=df,
             timestamp_format=timestamp_format,
             offset=datasource.offset,
             time_shift=query_object.time_shift,
+            dttm_cols=dttm_cols,

Review Comment:
   Thanks for pointing that out. I am going to refactor `normalize_dttm_col`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


[GitHub] [superset] villebro commented on a diff in pull request #21378: fix: Time Offset in SQLite

Posted by GitBox <gi...@apache.org>.
villebro commented on code in PR #21378:
URL: https://github.com/apache/superset/pull/21378#discussion_r970841998


##########
superset/utils/core.py:
##########
@@ -1271,15 +1271,15 @@ def is_adhoc_column(column: Column) -> TypeGuard[AdhocColumn]:
     return isinstance(column, dict)
 
 
-def get_base_axis_column(columns: Optional[List[Column]]) -> Optional[AdhocColumn]:
-    if columns is None:
-        return None
+def get_base_axis_label(columns: Optional[List[Column]]) -> Optional[str]:

Review Comment:
   Could this be:
   ```python
   def get_base_axis_labels(columns: Optional[List[Column]]) -> List[str]:
   ```
   i.e. an empty list if there are none, and otherwise all axis column labels?



##########
superset/common/query_context_processor.py:
##########
@@ -245,11 +246,21 @@ def normalize_df(self, df: pd.DataFrame, query_object: QueryObject) -> pd.DataFr
             if dttm_col:
                 timestamp_format = dttm_col.python_date_format
 
+        base_axis_label = None
+        if (
+            (label := get_base_axis_label(query_object.columns))
+            and query_object.datasource is not None
+            and (base_column := query_object.datasource.get_column(label))
+            and base_column.is_dttm
+        ):
+            base_axis_label = label
+
         normalize_dttm_col(
             df=df,
             timestamp_format=timestamp_format,
             offset=datasource.offset,
             time_shift=query_object.time_shift,
+            dttm_alias=base_axis_label,

Review Comment:
   In reference to my comment about changing `get_base_axis_label` to plural form, could we loop through all `base_axis_labels` here to future proof for multi-dimensional charts?



##########
superset/utils/core.py:
##########
@@ -1850,28 +1853,31 @@ def normalize_dttm_col(
     timestamp_format: Optional[str],
     offset: int,
     time_shift: Optional[timedelta],
+    dttm_alias: Optional[str] = DTTM_ALIAS,

Review Comment:
   nit: could we call this `dttm_col` to be more in line with the method name?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


[GitHub] [superset] villebro commented on a diff in pull request #21378: fix: Time Offset in SQLite

Posted by GitBox <gi...@apache.org>.
villebro commented on code in PR #21378:
URL: https://github.com/apache/superset/pull/21378#discussion_r970963527


##########
superset/utils/core.py:
##########
@@ -1850,28 +1853,31 @@ def normalize_dttm_col(
     timestamp_format: Optional[str],
     offset: int,
     time_shift: Optional[timedelta],
+    dttm_alias: Optional[str] = DTTM_ALIAS,

Review Comment:
   nit: could we call this `dttm_col`, `column_name` or other to be more in line with the method name?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org