You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@superset.apache.org by "michael-s-molina (via GitHub)" <gi...@apache.org> on 2024/05/10 14:52:30 UTC
[PR] fix: Time shifts calculation for ECharts plugins [superset]
michael-s-molina opened a new pull request, #28432:
URL: https://github.com/apache/superset/pull/28432
### SUMMARY
TODO
### BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF
TODO
### TESTING INSTRUCTIONS
TODO
### ADDITIONAL INFORMATION
- [ ] Has associated issue:
- [ ] Required feature flags:
- [ ] Changes UI
- [ ] Includes DB Migration (follow approval process in [SIP-59](https://github.com/apache/superset/issues/13351))
- [ ] Migration is atomic, supports rollback & is backwards-compatible
- [ ] Confirm DB migration upgrade and downgrade tested
- [ ] Runtime estimates and downtime expectations provided
- [ ] Introduces new feature or API
- [ ] Removes existing feature or API
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix: Time shifts calculation for ECharts plugins [superset]
Posted by "michael-s-molina (via GitHub)" <gi...@apache.org>.
michael-s-molina commented on code in PR #28432:
URL: https://github.com/apache/superset/pull/28432#discussion_r1600455839
##########
tests/unit_tests/common/test_time_shifts.py:
##########
@@ -0,0 +1,187 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from pandas import DataFrame, Series, Timestamp
+from pandas.testing import assert_frame_equal
+from pytest import fixture, mark
+
+from superset.common.chart_data import ChartDataResultFormat, ChartDataResultType
+from superset.common.query_context import QueryContext
+from superset.common.query_context_processor import QueryContextProcessor
+from superset.connectors.sqla.models import BaseDatasource
+from superset.constants import TimeGrain
+
+query_context_processor = QueryContextProcessor(
+ QueryContext(
+ datasource=BaseDatasource(),
+ queries=[],
+ result_type=ChartDataResultType.COLUMNS,
+ form_data={},
+ slice_=None,
+ result_format=ChartDataResultFormat.CSV,
+ cache_values={},
+ )
+)
+
+
+@fixture
+def make_join_column_producer():
+ def join_column_producer(row: Series, column_index: int) -> str:
+ return "CUSTOM_FORMAT"
+
+ return join_column_producer
+
+
+@mark.parametrize(
+ ("time_grain", "expected"),
+ [
+ (TimeGrain.WEEK, "2020-W01"),
+ (TimeGrain.MONTH, "2020-01"),
+ (TimeGrain.QUARTER, "2020-Q1"),
+ (TimeGrain.YEAR, "2020"),
+ ],
+)
+def test_join_column(time_grain: str, expected: str):
+ df = DataFrame({"ds": [Timestamp("2020-01-07")]})
+ column_name = "join_column"
+ query_context_processor.add_offset_join_column(df, column_name, time_grain)
+ result = DataFrame({"ds": [Timestamp("2020-01-07")], column_name: [expected]})
+ assert_frame_equal(df, result)
+
+
+def test_join_column_producer(make_join_column_producer):
+ df = DataFrame({"ds": [Timestamp("2020-01-07")]})
+ column_name = "join_column"
+ query_context_processor.add_offset_join_column(
+ df, column_name, TimeGrain.YEAR, None, make_join_column_producer
+ )
+ result = DataFrame(
+ {"ds": [Timestamp("2020-01-07")], column_name: ["CUSTOM_FORMAT"]}
+ )
+ assert_frame_equal(df, result)
+
+
+def test_join_offset_dfs_no_offsets():
+ df = DataFrame({"A": ["2021-01-01", "2021-02-01", "2021-03-01"]})
+ offset_dfs = {}
+ time_grain = "YEAR"
+ join_keys = ["A"]
+
+ result = query_context_processor.join_offset_dfs(
+ df, offset_dfs, time_grain, join_keys
+ )
+
+ assert_frame_equal(df, result)
+
+
+def test_join_offset_dfs_with_offsets():
+ df = DataFrame({"A": ["2021-01-01", "2021-02-01", "2021-03-01"]})
Review Comment:
Their content depends on the test case for most of them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix: Time shifts calculation for ECharts plugins [superset]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #28432:
URL: https://github.com/apache/superset/pull/28432#issuecomment-2105075458
@eschutho Ephemeral environment spinning up at http://54.184.7.128:8080. Credentials are `admin`/`admin`. Please allow several minutes for bootstrapping and startup.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix: Time shifts calculation for ECharts plugins [superset]
Posted by "john-bodley (via GitHub)" <gi...@apache.org>.
john-bodley commented on code in PR #28432:
URL: https://github.com/apache/superset/pull/28432#discussion_r1600433534
##########
superset/common/query_context_processor.py:
##########
@@ -339,19 +339,31 @@ def get_time_grain(query_object: QueryObject) -> Any | None:
return query_object.extras.get("time_grain_sqla")
- def add_aggregated_join_column(
+ # pylint: disable=too-many-arguments
+ def add_offset_join_column(
self,
df: pd.DataFrame,
+ name: str,
time_grain: str,
+ time_offset: str | None = None,
join_column_producer: Any = None,
) -> None:
+ """
+ Adds an offset join column to the provided DataFrame.
+
Review Comment:
```suggestion
The function modifies the DataFrame in-place.
```
##########
superset/common/query_context_processor.py:
##########
@@ -519,51 +510,112 @@ def processing_time_offsets( # pylint: disable=too-many-locals,too-many-stateme
datasource_uid=query_context.datasource.uid,
region=CacheRegion.DATA,
)
- offset_dfs.append(offset_metrics_df)
+ offset_dfs[offset] = offset_metrics_df
if offset_dfs:
- # iterate on offset_dfs, left join each with df
- for offset_df in offset_dfs:
- df = dataframe_utils.left_join_df(
- left_df=df,
- right_df=offset_df,
- join_keys=join_keys,
- rsuffix=R_SUFFIX,
- )
+ df = self.join_offset_dfs(
+ df,
+ offset_dfs,
+ time_grain,
+ join_keys,
+ )
+
+ return CachedTimeOffset(df=df, queries=queries, cache_keys=cache_keys)
+
+ def join_offset_dfs(
+ self,
+ df: pd.DataFrame,
+ offset_dfs: dict[str, pd.DataFrame],
+ time_grain: str,
+ join_keys: list[str],
+ ) -> pd.DataFrame:
+ """
+ Join offset DataFrames with the main DataFrame.
- # removes columns used for join
- df.drop(
- list(df.filter(regex=f"{AGGREGATED_JOIN_COLUMN}|{R_SUFFIX}")),
- axis=1,
- inplace=True,
+ :param df: The main DataFrame.
+ :param offset_dfs: A list of offset DataFrames.
+ :param time_grain: The time grain used to calculate the temporal join key.
+ :param join_keys: The keys to join on.
+ """
+ join_column_producer = config["TIME_GRAIN_JOIN_COLUMN_PRODUCERS"].get(
+ time_grain
)
- return CachedTimeOffset(df=df, queries=queries, cache_keys=cache_keys)
+ # iterate on offset_dfs, left join each with df
+ for offset, offset_df in offset_dfs.items():
+ # defines a column name for the offset join column
+ column_name = OFFSET_JOIN_COLUMN_SUFFIX + offset
+
+ # add offset join column to df
+ self.add_offset_join_column(
+ df, column_name, time_grain, offset, join_column_producer
+ )
+
+ # add artifoffseticial join column to offset_df
Review Comment:
```suggestion
# add offset join column to offset_df
```
##########
superset/common/query_context_processor.py:
##########
@@ -339,19 +339,31 @@ def get_time_grain(query_object: QueryObject) -> Any | None:
return query_object.extras.get("time_grain_sqla")
- def add_aggregated_join_column(
+ # pylint: disable=too-many-arguments
+ def add_offset_join_column(
self,
df: pd.DataFrame,
+ name: str,
time_grain: str,
+ time_offset: str | None = None,
join_column_producer: Any = None,
) -> None:
+ """
+ Adds an offset join column to the provided DataFrame.
+
+ :param df: pandas DataFrame to which the offset join column will be added.
+ :param name: The name of the new column to be added.
+ :param time_grain: The time grain used to calculate the new column.
+ :param time_offset: The time offset used to calculate the new column.
+ :param join_column_producer: A function to generate the join column.
+
+ :return: None. The function modifies the DataFrame in-place.
Review Comment:
```suggestion
```
I don't think we need to include a `return:` docstring for `None`. That said knowing this is in-place is helpful, but that statement could be included in the main docstring.
##########
tests/unit_tests/common/test_time_shifts.py:
##########
@@ -0,0 +1,187 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from pandas import DataFrame, Series, Timestamp
+from pandas.testing import assert_frame_equal
+from pytest import fixture, mark
+
+from superset.common.chart_data import ChartDataResultFormat, ChartDataResultType
+from superset.common.query_context import QueryContext
+from superset.common.query_context_processor import QueryContextProcessor
+from superset.connectors.sqla.models import BaseDatasource
+from superset.constants import TimeGrain
+
+query_context_processor = QueryContextProcessor(
+ QueryContext(
+ datasource=BaseDatasource(),
+ queries=[],
+ result_type=ChartDataResultType.COLUMNS,
+ form_data={},
+ slice_=None,
+ result_format=ChartDataResultFormat.CSV,
+ cache_values={},
+ )
+)
+
+
+@fixture
+def make_join_column_producer():
+ def join_column_producer(row: Series, column_index: int) -> str:
+ return "CUSTOM_FORMAT"
+
+ return join_column_producer
+
+
+@mark.parametrize(
+ ("time_grain", "expected"),
+ [
+ (TimeGrain.WEEK, "2020-W01"),
+ (TimeGrain.MONTH, "2020-01"),
+ (TimeGrain.QUARTER, "2020-Q1"),
+ (TimeGrain.YEAR, "2020"),
+ ],
+)
+def test_join_column(time_grain: str, expected: str):
+ df = DataFrame({"ds": [Timestamp("2020-01-07")]})
+ column_name = "join_column"
+ query_context_processor.add_offset_join_column(df, column_name, time_grain)
+ result = DataFrame({"ds": [Timestamp("2020-01-07")], column_name: [expected]})
+ assert_frame_equal(df, result)
+
+
+def test_join_column_producer(make_join_column_producer):
+ df = DataFrame({"ds": [Timestamp("2020-01-07")]})
+ column_name = "join_column"
+ query_context_processor.add_offset_join_column(
+ df, column_name, TimeGrain.YEAR, None, make_join_column_producer
+ )
+ result = DataFrame(
+ {"ds": [Timestamp("2020-01-07")], column_name: ["CUSTOM_FORMAT"]}
+ )
+ assert_frame_equal(df, result)
+
+
+def test_join_offset_dfs_no_offsets():
+ df = DataFrame({"A": ["2021-01-01", "2021-02-01", "2021-03-01"]})
+ offset_dfs = {}
+ time_grain = "YEAR"
+ join_keys = ["A"]
+
+ result = query_context_processor.join_offset_dfs(
+ df, offset_dfs, time_grain, join_keys
+ )
+
+ assert_frame_equal(df, result)
+
+
+def test_join_offset_dfs_with_offsets():
+ df = DataFrame({"A": ["2021-01-01", "2021-02-01", "2021-03-01"]})
Review Comment:
Are any of these dataframes reused? If so I wonder if they should be made into fixtures.
##########
superset/common/query_context_processor.py:
##########
@@ -519,51 +510,112 @@ def processing_time_offsets( # pylint: disable=too-many-locals,too-many-stateme
datasource_uid=query_context.datasource.uid,
region=CacheRegion.DATA,
)
- offset_dfs.append(offset_metrics_df)
+ offset_dfs[offset] = offset_metrics_df
if offset_dfs:
- # iterate on offset_dfs, left join each with df
- for offset_df in offset_dfs:
- df = dataframe_utils.left_join_df(
- left_df=df,
- right_df=offset_df,
- join_keys=join_keys,
- rsuffix=R_SUFFIX,
- )
+ df = self.join_offset_dfs(
+ df,
+ offset_dfs,
+ time_grain,
+ join_keys,
+ )
+
+ return CachedTimeOffset(df=df, queries=queries, cache_keys=cache_keys)
+
+ def join_offset_dfs(
+ self,
+ df: pd.DataFrame,
+ offset_dfs: dict[str, pd.DataFrame],
+ time_grain: str,
+ join_keys: list[str],
+ ) -> pd.DataFrame:
+ """
+ Join offset DataFrames with the main DataFrame.
- # removes columns used for join
- df.drop(
- list(df.filter(regex=f"{AGGREGATED_JOIN_COLUMN}|{R_SUFFIX}")),
- axis=1,
- inplace=True,
+ :param df: The main DataFrame.
+ :param offset_dfs: A list of offset DataFrames.
+ :param time_grain: The time grain used to calculate the temporal join key.
+ :param join_keys: The keys to join on.
+ """
+ join_column_producer = config["TIME_GRAIN_JOIN_COLUMN_PRODUCERS"].get(
+ time_grain
)
- return CachedTimeOffset(df=df, queries=queries, cache_keys=cache_keys)
+ # iterate on offset_dfs, left join each with df
+ for offset, offset_df in offset_dfs.items():
+ # defines a column name for the offset join column
+ column_name = OFFSET_JOIN_COLUMN_SUFFIX + offset
+
+ # add offset join column to df
+ self.add_offset_join_column(
+ df, column_name, time_grain, offset, join_column_producer
+ )
+
+ # add artifoffseticial join column to offset_df
+ self.add_offset_join_column(
+ offset_df, column_name, time_grain, None, join_column_producer
+ )
+
+ # the temporal column is the first column in the join keys
+ # so we use the join column instead of the temporal column
+ actual_join_keys = [column_name, *join_keys[1:]]
+
+ # left join df with offset_df
+ df = dataframe_utils.left_join_df(
+ left_df=df,
+ right_df=offset_df,
+ join_keys=actual_join_keys,
+ rsuffix=R_SUFFIX,
+ )
+
+ # move the temporal column to the first column in df
+ col = df.pop(join_keys[0])
Review Comment:
Is this guaranteed to exist?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix: Time shifts calculation for ECharts plugins [superset]
Posted by "michael-s-molina (via GitHub)" <gi...@apache.org>.
michael-s-molina commented on code in PR #28432:
URL: https://github.com/apache/superset/pull/28432#discussion_r1600457643
##########
superset/common/query_context_processor.py:
##########
@@ -339,19 +339,31 @@ def get_time_grain(query_object: QueryObject) -> Any | None:
return query_object.extras.get("time_grain_sqla")
- def add_aggregated_join_column(
+ # pylint: disable=too-many-arguments
+ def add_offset_join_column(
self,
df: pd.DataFrame,
+ name: str,
time_grain: str,
+ time_offset: str | None = None,
join_column_producer: Any = None,
) -> None:
+ """
+ Adds an offset join column to the provided DataFrame.
+
+ :param df: pandas DataFrame to which the offset join column will be added.
+ :param name: The name of the new column to be added.
+ :param time_grain: The time grain used to calculate the new column.
+ :param time_offset: The time offset used to calculate the new column.
+ :param join_column_producer: A function to generate the join column.
+
+ :return: None. The function modifies the DataFrame in-place.
Review Comment:
Done!
##########
superset/common/query_context_processor.py:
##########
@@ -339,19 +339,31 @@ def get_time_grain(query_object: QueryObject) -> Any | None:
return query_object.extras.get("time_grain_sqla")
- def add_aggregated_join_column(
+ # pylint: disable=too-many-arguments
+ def add_offset_join_column(
self,
df: pd.DataFrame,
+ name: str,
time_grain: str,
+ time_offset: str | None = None,
join_column_producer: Any = None,
) -> None:
+ """
+ Adds an offset join column to the provided DataFrame.
+
Review Comment:
Done!
##########
superset/common/query_context_processor.py:
##########
@@ -519,51 +510,112 @@ def processing_time_offsets( # pylint: disable=too-many-locals,too-many-stateme
datasource_uid=query_context.datasource.uid,
region=CacheRegion.DATA,
)
- offset_dfs.append(offset_metrics_df)
+ offset_dfs[offset] = offset_metrics_df
if offset_dfs:
- # iterate on offset_dfs, left join each with df
- for offset_df in offset_dfs:
- df = dataframe_utils.left_join_df(
- left_df=df,
- right_df=offset_df,
- join_keys=join_keys,
- rsuffix=R_SUFFIX,
- )
+ df = self.join_offset_dfs(
+ df,
+ offset_dfs,
+ time_grain,
+ join_keys,
+ )
+
+ return CachedTimeOffset(df=df, queries=queries, cache_keys=cache_keys)
+
+ def join_offset_dfs(
+ self,
+ df: pd.DataFrame,
+ offset_dfs: dict[str, pd.DataFrame],
+ time_grain: str,
+ join_keys: list[str],
+ ) -> pd.DataFrame:
+ """
+ Join offset DataFrames with the main DataFrame.
- # removes columns used for join
- df.drop(
- list(df.filter(regex=f"{AGGREGATED_JOIN_COLUMN}|{R_SUFFIX}")),
- axis=1,
- inplace=True,
+ :param df: The main DataFrame.
+ :param offset_dfs: A list of offset DataFrames.
+ :param time_grain: The time grain used to calculate the temporal join key.
+ :param join_keys: The keys to join on.
+ """
+ join_column_producer = config["TIME_GRAIN_JOIN_COLUMN_PRODUCERS"].get(
+ time_grain
)
- return CachedTimeOffset(df=df, queries=queries, cache_keys=cache_keys)
+ # iterate on offset_dfs, left join each with df
+ for offset, offset_df in offset_dfs.items():
+ # defines a column name for the offset join column
+ column_name = OFFSET_JOIN_COLUMN_SUFFIX + offset
+
+ # add offset join column to df
+ self.add_offset_join_column(
+ df, column_name, time_grain, offset, join_column_producer
+ )
+
+ # add artifoffseticial join column to offset_df
Review Comment:
Done!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix: Time shifts calculation for ECharts plugins [superset]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #28432:
URL: https://github.com/apache/superset/pull/28432#issuecomment-2112238048
Ephemeral environment shutdown and build artifacts deleted.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix: Time shifts calculation for ECharts plugins [superset]
Posted by "michael-s-molina (via GitHub)" <gi...@apache.org>.
michael-s-molina merged PR #28432:
URL: https://github.com/apache/superset/pull/28432
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix: Time shifts calculation for ECharts plugins [superset]
Posted by "eschutho (via GitHub)" <gi...@apache.org>.
eschutho commented on PR #28432:
URL: https://github.com/apache/superset/pull/28432#issuecomment-2105064080
/testenv up
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix: Time shifts calculation for ECharts plugins [superset]
Posted by "michael-s-molina (via GitHub)" <gi...@apache.org>.
michael-s-molina commented on code in PR #28432:
URL: https://github.com/apache/superset/pull/28432#discussion_r1600452645
##########
superset/common/query_context_processor.py:
##########
@@ -519,51 +510,112 @@ def processing_time_offsets( # pylint: disable=too-many-locals,too-many-stateme
datasource_uid=query_context.datasource.uid,
region=CacheRegion.DATA,
)
- offset_dfs.append(offset_metrics_df)
+ offset_dfs[offset] = offset_metrics_df
if offset_dfs:
- # iterate on offset_dfs, left join each with df
- for offset_df in offset_dfs:
- df = dataframe_utils.left_join_df(
- left_df=df,
- right_df=offset_df,
- join_keys=join_keys,
- rsuffix=R_SUFFIX,
- )
+ df = self.join_offset_dfs(
+ df,
+ offset_dfs,
+ time_grain,
+ join_keys,
+ )
+
+ return CachedTimeOffset(df=df, queries=queries, cache_keys=cache_keys)
+
+ def join_offset_dfs(
+ self,
+ df: pd.DataFrame,
+ offset_dfs: dict[str, pd.DataFrame],
+ time_grain: str,
+ join_keys: list[str],
+ ) -> pd.DataFrame:
+ """
+ Join offset DataFrames with the main DataFrame.
- # removes columns used for join
- df.drop(
- list(df.filter(regex=f"{AGGREGATED_JOIN_COLUMN}|{R_SUFFIX}")),
- axis=1,
- inplace=True,
+ :param df: The main DataFrame.
+ :param offset_dfs: A list of offset DataFrames.
+ :param time_grain: The time grain used to calculate the temporal join key.
+ :param join_keys: The keys to join on.
+ """
+ join_column_producer = config["TIME_GRAIN_JOIN_COLUMN_PRODUCERS"].get(
+ time_grain
)
- return CachedTimeOffset(df=df, queries=queries, cache_keys=cache_keys)
+ # iterate on offset_dfs, left join each with df
+ for offset, offset_df in offset_dfs.items():
+ # defines a column name for the offset join column
+ column_name = OFFSET_JOIN_COLUMN_SUFFIX + offset
+
+ # add offset join column to df
+ self.add_offset_join_column(
+ df, column_name, time_grain, offset, join_column_producer
+ )
+
+ # add artifoffseticial join column to offset_df
+ self.add_offset_join_column(
+ offset_df, column_name, time_grain, None, join_column_producer
+ )
+
+ # the temporal column is the first column in the join keys
+ # so we use the join column instead of the temporal column
+ actual_join_keys = [column_name, *join_keys[1:]]
+
+ # left join df with offset_df
+ df = dataframe_utils.left_join_df(
+ left_df=df,
+ right_df=offset_df,
+ join_keys=actual_join_keys,
+ rsuffix=R_SUFFIX,
+ )
+
+ # move the temporal column to the first column in df
+ col = df.pop(join_keys[0])
Review Comment:
Yep
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix: Time shifts calculation for ECharts plugins [superset]
Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #28432:
URL: https://github.com/apache/superset/pull/28432#issuecomment-2110358179
## [Codecov](https://app.codecov.io/gh/apache/superset/pull/28432?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
Attention: Patch coverage is `49.01961%` with `26 lines` in your changes are missing coverage. Please review.
> Project coverage is 60.98%. Comparing base [(`76d897e`)](https://app.codecov.io/gh/apache/superset/commit/76d897eaa2f9e137102bc194c2e3109c29d0348f?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) to head [(`8c664ac`)](https://app.codecov.io/gh/apache/superset/pull/28432?dropdown=coverage&src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
> Report is 108 commits behind head on master.
| [Files](https://app.codecov.io/gh/apache/superset/pull/28432?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [superset/common/query\_context\_processor.py](https://app.codecov.io/gh/apache/superset/pull/28432?src=pr&el=tree&filepath=superset%2Fcommon%2Fquery_context_processor.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c3VwZXJzZXQvY29tbW9uL3F1ZXJ5X2NvbnRleHRfcHJvY2Vzc29yLnB5) | 47.36% | [20 Missing :warning: ](https://app.codecov.io/gh/apache/superset/pull/28432?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...gin-chart-echarts/src/Timeseries/transformProps.ts](https://app.codecov.io/gh/apache/superset/pull/28432?src=pr&el=tree&filepath=superset-frontend%2Fplugins%2Fplugin-chart-echarts%2Fsrc%2FTimeseries%2FtransformProps.ts&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c3VwZXJzZXQtZnJvbnRlbmQvcGx1Z2lucy9wbHVnaW4tY2hhcnQtZWNoYXJ0cy9zcmMvVGltZXNlcmllcy90cmFuc2Zvcm1Qcm9wcy50cw==) | 33.33% | [5 Missing and 1 partial :warning: ](https://app.codecov.io/gh/apache/superset/pull/28432?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #28432 +/- ##
==========================================
+ Coverage 60.48% 60.98% +0.49%
==========================================
Files 1931 1944 +13
Lines 76236 77301 +1065
Branches 8568 8653 +85
==========================================
+ Hits 46114 47140 +1026
- Misses 28017 28042 +25
- Partials 2105 2119 +14
```
| [Flag](https://app.codecov.io/gh/apache/superset/pull/28432/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [hive](https://app.codecov.io/gh/apache/superset/pull/28432/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `49.09% <10.52%> (-0.08%)` | :arrow_down: |
| [presto](https://app.codecov.io/gh/apache/superset/pull/28432/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `53.64% <10.52%> (-0.16%)` | :arrow_down: |
| [python](https://app.codecov.io/gh/apache/superset/pull/28432/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `64.43% <47.36%> (+0.94%)` | :arrow_up: |
| [unit](https://app.codecov.io/gh/apache/superset/pull/28432/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `58.81% <47.36%> (+1.19%)` | :arrow_up: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
</details>
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/superset/pull/28432?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org