You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@superset.apache.org by "SEPURI-SAI-KRISHNA (via GitHub)" <gi...@apache.org> on 2026/10/17 06:38:32 UTC
[PR] fix(reports): apply d3TimeFormat in table post-processing [superset]
SEPURI-SAI-KRISHNA opened a new pull request, #45410:
URL: https://github.com/apache/superset/pull/45410
### SUMMARY
Fixes #45409.
`table()` in `superset/charts/client_processing.py`, the backend mirror of the
Table plugin used for `POST_PROCESSED` report data, applied `d3NumberFormat` and
`currencyFormat` but had no `d3TimeFormat` branch, so a column with an explicit
time format was left raw in email and Slack reports while the browser formatted
it.
This adds the explicit `d3TimeFormat` branch as the first check in the per-column
loop, mirroring the order in `plugin-chart-table/src/transformProps.ts`
(`if (isTime || config.d3TimeFormat)`). It is scoped to datetime columns with an
explicit format and applied with `strftime`, which shares its common directives
(`%Y`, `%m`, `%d`, `%H`, `%M`, `%S`, ...) with d3-time-format. A non-datetime
column carrying the key is left untouched, and null (`NaT`) is preserved.
The wider browser behaviour (bare `isTime`, the `tableTimestampFormat` default,
`SMART_DATE` granularity, and string temporal columns) is intentionally out of
scope; it needs a full server-side d3-time-format port and is a larger change
for reports.
### BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF
With `column_config = {"ds": {"d3TimeFormat": "%Y"}, "SUM(num)": {"d3NumberFormat": ",.2f"}}`:
Before:
```
ds SUM(num)
2024-01-15 13:45:00 1,234,567.00
```
After:
```
ds SUM(num)
2024 1,234,567.00
```
### TESTING INSTRUCTIONS
```
pytest tests/unit_tests/charts/test_client_processing.py -k table
```
New `test_table_applies_time_format_alongside_number_format` fails without the
branch (the timestamp is left raw) and passes with it; the number column is
formatted on the same call, confirming the branch does not disturb number
formatting.
### ADDITIONAL INFORMATION
- [x] Has associated issue: #45409
- [ ] Required feature flags:
- [ ] Changes UI
- [ ] Includes DB Migration (follow approval process in [SIP-59](https://github.com/apache/superset/issues/13351))
- [ ] Migration is atomic, supports rollback & is backwards-compatible
- [ ] Confirm DB migration upgrade and downgrade tested
- [ ] Runtime estimates and downtime expectations provided
- [ ] Introduces new feature or API
- [ ] Removes existing feature or API
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(reports): apply d3TimeFormat in table post-processing [superset]
Posted by "SEPURI-SAI-KRISHNA (via GitHub)" <gi...@apache.org>.
SEPURI-SAI-KRISHNA commented on code in PR #45410:
URL: https://github.com/apache/superset/pull/45410#discussion_r4403758587
##########
superset/charts/client_processing.py:
##########
@@ -1568,6 +1676,44 @@ def _read_excel_for_client_processing(
return df.set_index(first_column)
+def payload_coltypes(
+ df: pd.DataFrame,
+ datasource: Optional[Union["BaseDatasource", "Query"]],
+) -> list[GenericDataType]:
+ """The payload's column types, reporting a text-rendered column as text.
+
+ ``extract_dataframe_dtypes`` lets the dataset's ``is_dttm`` flag decide the
+ type, so a temporal column a post-processor has rendered as text is still
+ reported as ``TEMPORAL``. ``chart_data_to_dataframe`` casts every
+ ``TEMPORAL`` column back to ``datetime64[ms]`` to build the notification
+ table, which would re-parse those strings and lose the configured format,
+ and would read ``02/01/2024`` month-first while reading ``15/01/2024``
+ day-first.
+
+ Only a column whose values really are strings is reported differently. The
+ XLSX path hands these types to ``apply_column_types`` to get date cells out
+ of a frame whose dates are still dates, and that frame is untouched here.
+
+ Nulls are dropped before the values are read. ``format_time_column`` leaves
+ them in place, and ``infer_dtype`` counts a ``NaT`` as a value of its own
+ type rather than one ``skipna`` drops, so it calls a formatted column
+ holding one ``"mixed"``. A column of nothing but nulls reads as neither
+ text nor mixed and keeps whatever type the dataset gives it.
+
+ The index is positional because ``extract_dataframe_dtypes`` walks the
+ frame the same way, so the two line up even where a label repeats.
+ """
+ coltypes = extract_dataframe_dtypes(df, datasource)
+ return [
+ GenericDataType.STRING
+ if coltype == GenericDataType.TEMPORAL
+ and pd.api.types.infer_dtype(df.iloc[:, index].dropna(), skipna=False)
Review Comment:
Confirmed. `in ("string", "empty")` keeps all 315 green, and `infer_dtype` on
the dropped series returns `"empty"` for an all-null column against `"string"`
for the mixed one, which is why the existing null test passes either way.
Added `test_payload_coltypes_keeps_an_all_null_column_temporal`: an all-`None`
object column flagged `is_dttm`, asserting `[GenericDataType.TEMPORAL]`. It is
the only failure under that widening.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(reports): apply d3TimeFormat in table post-processing [superset]
Posted by "sadpandajoe (via GitHub)" <gi...@apache.org>.
sadpandajoe commented on code in PR #45410:
URL: https://github.com/apache/superset/pull/45410#discussion_r4336546033
##########
superset/charts/client_processing.py:
##########
@@ -73,6 +75,32 @@
# The Excel equivalent, applied as a cell format so the value stays numeric.
EXCEL_PERCENT_FORMAT = "0.0%"
+# ``strftime`` copies a directive it does not recognize into its output
+# verbatim, so a d3-only one would reach the report as literal text: ``%L`` and
+# ``%q`` stay ``%L`` and ``%q`` rather than becoming milliseconds and a quarter.
+# These are the directives that carry the same meaning in d3-time-format and in
+# ``strftime``. Left out are:
+#
+# * the ones that name a weekday, a month or a period (``%a``, ``%A``, ``%b``,
+# ``%B``, ``%p``). The browser reads those names from the d3 locale
+# definition, which ``D3_TIME_FORMAT`` overrides key by key, while
+# ``strftime`` reads them from the worker's ``LC_TIME``. The two agree only
+# when nothing is configured and the worker runs in English.
+# * the locale-composite ones (``%c``, ``%x``, ``%X``), whose output follows
+# the server's locale rather than the chart's.
+# * ``%f``. d3 derives it from ``getUTCMilliseconds() * 1000``, so the browser's
+# microseconds always end in ``000``, while a warehouse column with sub-second
+# precision reaches ``strftime`` with the real value.
+# * the platform-dependent or d3-only ones (``%e``, ``%g``, ``%L``, ``%q``,
+# ``%Q``, ``%s``, ``%Z``, ``%z``).
+SHARED_TIME_DIRECTIVES = frozenset("dGHIjmMSuUVwWyY%")
Review Comment:
Nothing in the suite pins most of this allowlist. The successful-format tests only use `%Y`, `%d`, `%m`, `%H` and `%M`, so removing `%V` (or `%G`, `%j`, `%U`, ...) from `SHARED_TIME_DIRECTIVES` would leave every test green while silently turning ISO-week formats back into raw timestamps, and the d3 parity run described in the PR was manual. Could a parametrized `table()` case assert literal output for the remaining directives, e.g. `2021-01-01` with `%G-W%V-%u` giving `2020-W53-5`, plus a `%%` case?
##########
superset/charts/client_processing.py:
##########
@@ -835,6 +863,45 @@ def union_currency_context(
}
+def is_strftime_expressible(d3_time_format: str) -> bool:
+ """Whether every directive in a d3 time format means the same to strftime."""
+ if LOCAL_TIME_PREFIX in d3_time_format:
+ # ``TimeFormatterRegistry.get`` strips the marker before building the
+ # formatter, so leaving it in would render it as literal text.
+ return False
+ directives = re.findall(r"%(.)", d3_time_format)
Review Comment:
A format with a stray trailing `%` passes this check: `re.findall(r"%(.)", ...)` only sees a `%` that has a character after it, so `%Y%` yields just `Y`. `strftime` then copies the trailing `%` (`2024%` for `2024-01-15`) while d3 drops it (`2024`), so the report no longer matches the chart for that format. Should a `%` that isn't consumed as part of a directive make the format non-expressible?
##########
superset/charts/client_processing.py:
##########
@@ -1304,6 +1391,44 @@ def _read_excel_for_client_processing(
return df.set_index(first_column)
+def payload_coltypes(
+ df: pd.DataFrame,
+ datasource: Optional[Union["BaseDatasource", "Query"]],
+) -> list[GenericDataType]:
+ """The payload's column types, reporting a text-rendered column as text.
+
+ ``extract_dataframe_dtypes`` lets the dataset's ``is_dttm`` flag decide the
+ type, so a temporal column a post-processor has rendered as text is still
+ reported as ``TEMPORAL``. ``chart_data_to_dataframe`` casts every
+ ``TEMPORAL`` column back to ``datetime64[ms]`` to build the notification
+ table, which would re-parse those strings and lose the configured format,
+ and would read ``02/01/2024`` month-first while reading ``15/01/2024``
+ day-first.
+
+ Only a column whose values really are strings is reported differently. The
+ XLSX path hands these types to ``apply_column_types`` to get date cells out
+ of a frame whose dates are still dates, and that frame is untouched here.
+
+ Nulls are dropped before the values are read. ``format_time_column`` leaves
+ them in place, and ``infer_dtype`` counts a ``NaT`` as a value of its own
+ type rather than one ``skipna`` drops, so it calls a formatted column
+ holding one ``"mixed"``. A column of nothing but nulls reads as neither
+ text nor mixed and keeps whatever type the dataset gives it.
+
+ The index is positional because ``extract_dataframe_dtypes`` walks the
+ frame the same way, so the two line up even where a label repeats.
+ """
+ coltypes = extract_dataframe_dtypes(df, datasource)
+ return [
+ GenericDataType.STRING
+ if coltype == GenericDataType.TEMPORAL
+ and pd.api.types.infer_dtype(df.iloc[:, index].dropna(), skipna=False)
Review Comment:
The positional `iloc` read is what keeps repeated column labels lined up with `extract_dataframe_dtypes`, but no test pins it. If this became `df[df.columns[index]]`, a frame with two `ds` columns (one holding `"02/01/2024"`, the other a real timestamp) would type both as `STRING`, and the real timestamp column would lose `TEMPORAL` in the payload. Could a direct `payload_coltypes` case with two columns labelled `ds` assert `[STRING, TEMPORAL]`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(reports): apply d3TimeFormat in table post-processing [superset]
Posted by "sadpandajoe (via GitHub)" <gi...@apache.org>.
sadpandajoe commented on code in PR #45410:
URL: https://github.com/apache/superset/pull/45410#discussion_r4402941108
##########
superset/charts/client_processing.py:
##########
@@ -1568,6 +1676,44 @@ def _read_excel_for_client_processing(
return df.set_index(first_column)
+def payload_coltypes(
+ df: pd.DataFrame,
+ datasource: Optional[Union["BaseDatasource", "Query"]],
+) -> list[GenericDataType]:
+ """The payload's column types, reporting a text-rendered column as text.
+
+ ``extract_dataframe_dtypes`` lets the dataset's ``is_dttm`` flag decide the
+ type, so a temporal column a post-processor has rendered as text is still
+ reported as ``TEMPORAL``. ``chart_data_to_dataframe`` casts every
+ ``TEMPORAL`` column back to ``datetime64[ms]`` to build the notification
+ table, which would re-parse those strings and lose the configured format,
+ and would read ``02/01/2024`` month-first while reading ``15/01/2024``
+ day-first.
+
+ Only a column whose values really are strings is reported differently. The
+ XLSX path hands these types to ``apply_column_types`` to get date cells out
+ of a frame whose dates are still dates, and that frame is untouched here.
+
+ Nulls are dropped before the values are read. ``format_time_column`` leaves
+ them in place, and ``infer_dtype`` counts a ``NaT`` as a value of its own
+ type rather than one ``skipna`` drops, so it calls a formatted column
+ holding one ``"mixed"``. A column of nothing but nulls reads as neither
+ text nor mixed and keeps whatever type the dataset gives it.
+
+ The index is positional because ``extract_dataframe_dtypes`` walks the
+ frame the same way, so the two line up even where a label repeats.
+ """
+ coltypes = extract_dataframe_dtypes(df, datasource)
+ return [
+ GenericDataType.STRING
+ if coltype == GenericDataType.TEMPORAL
+ and pd.api.types.infer_dtype(df.iloc[:, index].dropna(), skipna=False)
Review Comment:
The docstring says a column of nothing but nulls keeps the type the dataset gives it, but no test pins that. Changing `== "string"` to `in ("string", "empty")` leaves every current test green, because the null test mixes a string in with the `NaT`. An all-`None` object column flagged `is_dttm` would then be typed `STRING` instead of `TEMPORAL`. Could a direct `payload_coltypes` case with only null values assert `[GenericDataType.TEMPORAL]`?
##########
tests/unit_tests/charts/test_client_processing.py:
##########
@@ -2875,6 +2877,551 @@ def test_table_applies_currency_format() -> None:
assert formatted["amount"].tolist() == ["$ 1,234.50"]
+def test_table_applies_time_format_alongside_number_format() -> None:
+ """
+ Table reports honor an explicit ``d3TimeFormat`` on a temporal column,
+ mirroring the Table plugin's first formatter branch, while still applying
+ number formats on the same call. Without the time branch the timestamp was
+ left raw even though the browser formats it.
+ """
+ df = pd.DataFrame.from_dict(
+ {
+ "ds": {
+ 0: pd.Timestamp("2024-01-15 13:45:00"),
+ 1: pd.Timestamp("2024-06-30 08:00:00"),
+ },
+ "SUM(num)": {0: 1234567.0, 1: 42.0},
+ }
+ )
+ form_data = {
+ "viz_type": "table",
+ "metrics": ["SUM(num)"],
+ "column_config": {
+ "ds": {"d3TimeFormat": "%Y"},
+ "SUM(num)": {"d3NumberFormat": ",.2f"},
+ },
+ }
+ formatted = table(df, form_data)
+ assert formatted["ds"].tolist() == ["2024", "2024"]
+ assert formatted["SUM(num)"].tolist() == ["1,234,567.00", "42.00"]
+
+
+def test_table_leaves_a_null_time_format_alone() -> None:
+ """
+ A ``d3TimeFormat`` of JSON ``null`` leaves the column alone.
+
+ ``column_config`` is free-form JSON out of the saved chart params, so the
+ key can be present and null. ``config.get("d3TimeFormat", "").strip()``
+ would raise ``AttributeError: 'NoneType' object has no attribute 'strip'``
+ and fail the whole report, and every other table test here passes either
+ way because none of them puts a null in the key.
+ """
+ df = pd.DataFrame.from_dict(
+ {
+ "ds": {0: pd.Timestamp("2024-04-02 13:45:00")},
+ "SUM(num)": {0: 1234.5678},
+ }
+ )
+ form_data = {
+ "viz_type": "table",
+ "metrics": ["SUM(num)"],
+ "column_config": {
+ "ds": {"d3TimeFormat": None},
+ "SUM(num)": {"d3NumberFormat": ",.1f"},
+ },
+ }
+ formatted = table(df, form_data)
+ assert formatted["ds"].tolist() == [pd.Timestamp("2024-04-02 13:45:00")]
+ assert formatted["SUM(num)"].tolist() == ["1,234.6"]
+
+
+@pytest.mark.parametrize(
+ ("time_format", "expected"),
+ [
+ ("%d", "01"),
+ ("%G", "2020"),
+ ("%H", "13"),
+ ("%I", "01"),
+ ("%j", "001"),
+ ("%m", "01"),
+ ("%M", "05"),
+ ("%S", "09"),
+ ("%u", "5"),
+ ("%U", "00"),
+ ("%V", "53"),
+ ("%w", "5"),
+ ("%W", "00"),
+ ("%y", "21"),
+ ("%Y", "2021"),
+ ("%%", "%"),
+ ("%G-W%V-%u", "2020-W53-5"),
+ ],
+)
+def test_table_time_format_matches_d3_for_each_shared_directive(
+ time_format: str, expected: str
+) -> None:
+ """
+ Every directive in ``SHARED_TIME_DIRECTIVES`` is held to d3's own output.
+
+ The expected strings are what ``utcFormat`` from ``d3-time-format`` returns
+ for this timestamp, so taking a directive out of the allowlist fails the
+ case that names it rather than passing unnoticed. 1 January 2021 is a
+ Friday in ISO week 53 of 2020, which is what separates ``%G`` from ``%Y``
+ and ``%V`` from ``%U`` and ``%W``; 13:05:09 separates ``%I`` from ``%H``.
+ """
+ df = pd.DataFrame.from_dict({"ds": {0: pd.Timestamp("2021-01-01 13:05:09")}})
+ form_data = {
+ "viz_type": "table",
+ "column_config": {"ds": {"d3TimeFormat": time_format}},
+ }
+
+ formatted = table(df, form_data)
+
+ assert formatted["ds"].tolist() == [expected]
+
+
+@pytest.mark.parametrize(
+ "time_format",
+ [
+ "%Y%",
+ "%d/%m/%Y%",
+ "%%%",
+ "%Y%%%",
+ ],
+)
+def test_table_skips_a_time_format_with_an_unconsumed_percent(
+ time_format: str,
+) -> None:
+ """
+ A ``%`` that no directive consumes is dropped by d3 and copied by
+ ``strftime``.
+
+ ``%Y%`` is ``2021`` in the chart and would be ``2021%`` in the report, and
+ ``%%%`` is ``%`` against ``%%``. Every directive in these is shared, so the
+ allowlist alone accepts them and the column has to be left to the browser
+ on the stray ``%`` instead.
+ """
+ df = pd.DataFrame.from_dict({"ds": {0: pd.Timestamp("2021-01-01 13:05:09")}})
+ form_data = {
+ "viz_type": "table",
+ "column_config": {"ds": {"d3TimeFormat": time_format}},
+ }
+
+ formatted = table(df, form_data)
+
+ assert formatted["ds"].tolist() == [pd.Timestamp("2021-01-01 13:05:09")]
+
+
+def test_table_leaves_adaptive_time_format_alone() -> None:
+ """
+ ``smart_date`` ("Adaptive formatting") is an alias the browser resolves to a
+ formatter, not a format string. It carries no ``%`` directive, so the rule
+ that a format has to have one is what keeps it away from ``strftime``,
+ which would hand the alias back unchanged and print it in every cell.
+ """
+ df = pd.DataFrame.from_dict(
+ {
+ "ds": {
+ 0: pd.Timestamp("2024-01-15 13:45:00"),
+ 1: pd.Timestamp("2024-06-30 08:00:00"),
+ },
+ }
+ )
+ form_data = {
+ "viz_type": "table",
+ "column_config": {"ds": {"d3TimeFormat": "smart_date"}},
+ }
+ formatted = table(df, form_data)
+ assert formatted["ds"].tolist() == [
+ pd.Timestamp("2024-01-15 13:45:00"),
+ pd.Timestamp("2024-06-30 08:00:00"),
+ ]
+
+
+@pytest.mark.parametrize(
+ "time_format",
+ [
+ "yyyy-mm-dd",
+ "Q1",
+ "Date",
+ "100%",
+ ],
+)
+def test_table_skips_a_time_format_with_no_directive(time_format: str) -> None:
+ """
+ A format carrying no ``%`` directive is not a format string.
+
+ ``strftime`` has nothing to substitute and returns the string it was given,
+ so accepting one of these would put ``yyyy-mm-dd`` in every cell of the
+ column instead of a date. This is the rule that also rejects the adaptive
+ alias.
+ """
+ df = pd.DataFrame.from_dict({"ds": {0: pd.Timestamp("2024-04-02 13:45:00")}})
+ form_data = {
+ "viz_type": "table",
+ "column_config": {"ds": {"d3TimeFormat": time_format}},
+ }
+
+ formatted = table(df, form_data)
+
+ assert formatted["ds"].tolist() == [pd.Timestamp("2024-04-02 13:45:00")]
+
+
+@pytest.mark.parametrize(
+ ("values", "expected"),
+ [
+ pytest.param(
+ {0: "2024-01-15", 1: "2024-06-30"},
+ ["2024-01-15", "2024-06-30"],
+ id="string column",
+ ),
+ pytest.param({0: 2024, 1: 2025}, [2024, 2025], id="integer column"),
+ ],
+)
+def test_table_skips_a_time_format_on_a_non_datetime_column(
+ values: dict[int, Any], expected: list[Any]
+) -> None:
+ """
+ ``d3TimeFormat`` on a column that is not datetime leaves the column alone.
+
+ A string or a number has no ``strftime`` to call, so formatting one would
+ raise ``AttributeError`` in the report worker. The browser reaches these
+ through a d3-time-format port, which parses the value first, and that is
+ what the follow-up needs rather than something ``strftime`` can stand in
+ for.
+ """
+ df = pd.DataFrame.from_dict({"ds": values})
+ form_data = {
+ "viz_type": "table",
+ "column_config": {"ds": {"d3TimeFormat": "%Y"}},
+ }
+
+ formatted = table(df, form_data)
+
+ assert formatted["ds"].tolist() == expected
+
+
+def test_table_time_format_keeps_nulls() -> None:
+ """
+ A missing timestamp stays missing, and does not fail the report.
+
+ ``strftime`` on a ``NaT`` does not produce the string ``"NaT"``; it raises
+ ``ValueError: NaTType does not support strftime``, so without the
+ ``pd.notna`` guard one null would fail the whole report rather than print
+ anything.
+ """
+ df = pd.DataFrame.from_dict(
+ {"ds": {0: pd.Timestamp("2024-01-15"), 1: pd.NaT}},
+ )
+ form_data = {
+ "viz_type": "table",
+ "column_config": {"ds": {"d3TimeFormat": "%Y"}},
+ }
+
+ formatted = table(df, form_data)
+
+ assert formatted["ds"].tolist()[0] == "2024"
+ assert pd.isna(formatted["ds"].tolist()[1])
+
+
+@pytest.mark.parametrize(
+ "time_format",
+ [" %Y-%m-%d ", " %Y-%m-%d", "%Y-%m-%d ", "\t%Y-%m-%d\n", "\r%Y-%m-%d\x0b"],
+)
+def test_table_time_format_strips_surrounding_whitespace(time_format: str) -> None:
+ """
+ A padded format renders like the chart, which trims before formatting.
+
+ ``TimeFormatterRegistry.get`` trims the format string before building the
+ formatter, while d3 itself does not: ``utcFormat(" %Y ")`` returns
+ ``" 2024 "``. Without the strip the report kept the padding the chart had
+ already dropped. Every character here is whitespace to both ``str.strip``
+ and ``String.prototype.trim``; the handful they disagree on is in
+ ``format_time_column``.
+ """
+ df = pd.DataFrame.from_dict({"ds": {0: pd.Timestamp("2024-01-15")}})
+ form_data = {
+ "viz_type": "table",
+ "column_config": {"ds": {"d3TimeFormat": time_format}},
+ }
+
+ formatted = table(df, form_data)
+
+ assert formatted["ds"].tolist() == ["2024-01-15"]
+
+
+@pytest.mark.parametrize(
+ "time_format",
+ [
+ "%Y Q%q",
+ "%H:%M:%S.%L",
+ "%Q",
+ "%-d %b",
+ "%x",
+ "local!%Y-%m-%d",
+ "%Y local!",
+ "%H:%M:%S.%f",
Review Comment:
Nothing in this rejection list exercises `%Z`, so adding `Z` to `SHARED_TIME_DIRECTIVES` would keep every test green. After the UTC conversion `strftime("%Y %Z")` renders `2024 UTC`, while d3's `%Z` renders `2024 +0000`, so the report would stop matching the chart for any zone-bearing format. Could `"%Y %Z"` be added here, asserting the timestamp comes back unchanged?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(reports): apply d3TimeFormat in table post-processing [superset]
Posted by "sadpandajoe (via GitHub) via notifications" <no...@superset.apache.org>.
sadpandajoe merged PR #45410:
URL: https://github.com/apache/superset/pull/45410
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(reports): apply d3TimeFormat in table post-processing [superset]
Posted by "SEPURI-SAI-KRISHNA (via GitHub)" <gi...@apache.org>.
SEPURI-SAI-KRISHNA commented on code in PR #45410:
URL: https://github.com/apache/superset/pull/45410#discussion_r4356554018
##########
superset/charts/client_processing.py:
##########
@@ -835,6 +863,45 @@ def union_currency_context(
}
+def is_strftime_expressible(d3_time_format: str) -> bool:
+ """Whether every directive in a d3 time format means the same to strftime."""
+ if LOCAL_TIME_PREFIX in d3_time_format:
+ # ``TimeFormatterRegistry.get`` strips the marker before building the
+ # formatter, so leaving it in would render it as literal text.
+ return False
+ directives = re.findall(r"%(.)", d3_time_format)
Review Comment:
Confirmed against d3. `utcFormat` on 2021-01-01 13:05:09 UTC:
```
format d3 strftime
%Y% 2021 2021%
%d/%m/%Y% 01/01/2021 01/01/2021%
%%% % %%
%Y%%% 2021% 2021%%
```
So a `%` that no directive consumes has to make the format non-expressible. `%%` is not that case, both read it as one escaped `%` and agree, so the allowlist keeps `%`.
Added after the no-directive rule so the existing cases keep their path:
```python
if "%" in re.sub(r"%.", "", d3_time_format):
return False
```
Stripping every `%` a directive consumes leaves behind exactly the stray ones. It rejects all four formats above and accepts `%Y%%`, `100%%` and the ordinary ones. Pinned by `test_table_skips_a_time_format_with_an_unconsumed_percent` over the four.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(reports): apply d3TimeFormat in table post-processing [superset]
Posted by "SEPURI-SAI-KRISHNA (via GitHub)" <gi...@apache.org>.
SEPURI-SAI-KRISHNA commented on code in PR #45410:
URL: https://github.com/apache/superset/pull/45410#discussion_r4326580767
##########
superset/charts/client_processing.py:
##########
@@ -1304,6 +1390,34 @@ def _read_excel_for_client_processing(
return df.set_index(first_column)
+def payload_coltypes(
+ df: pd.DataFrame,
+ datasource: Optional[Union["BaseDatasource", "Query"]],
+) -> list[GenericDataType]:
+ """The payload's column types, reporting a text-rendered column as text.
+
+ ``extract_dataframe_dtypes`` lets the dataset's ``is_dttm`` flag decide the
+ type, so a temporal column a post-processor has rendered as text is still
+ reported as ``TEMPORAL``. ``chart_data_to_dataframe`` casts every
+ ``TEMPORAL`` column back to ``datetime64[ms]`` to build the notification
+ table, which would re-parse those strings and lose the configured format,
+ and would read ``02/01/2024`` month-first while reading ``15/01/2024``
+ day-first.
+
+ Only a column whose values really are strings is reported differently. The
+ XLSX path hands these types to ``apply_column_types`` to get date cells out
+ of a frame whose dates are still dates, and that frame is untouched here.
+ """
+ coltypes = extract_dataframe_dtypes(df, datasource)
+ return [
+ GenericDataType.STRING
Review Comment:
You are right, and the mutation you describe is worse than passing: it is
silent. Returning `STRING` for every temporal column instead of only a
text-valued one passed all 126 tests on the previous commit.
```
unformatted datetime column
as written coltypes = ['TEMPORAL', 'STRING']
STRING for every temporal coltypes = ['STRING', 'STRING']
formatted datetime column
both coltypes = ['STRING', 'STRING']
```
Added both cases you asked for.
`test_report_payload_keeps_an_unformatted_temporal_column_temporal` runs a
payload with no `d3TimeFormat` through `apply_client_processing` and asserts the
column stays `TEMPORAL` and its values stay `Timestamp`s. That is the test the
mutation now fails.
`test_report_payload_types_a_csv_read_temporal_column_as_text` is the other
half, and checking it turned up something I had not registered: `coltypes` is
built for every result format, while `apply_number_format` is only true for
JSON. So the CSV branch never formats anything, its temporal column arrives as
text straight from `read_csv`, and the override fires on a column no
post-processor touched:
```
CSV payload "ds,cnt\n2024-01-02,1\n2024-01-15,2\n"
coltypes = ['STRING', 'NUMERIC'] and the CSV text is handed back untouched
```
Which is the behavior I want, since the type should describe the values rather
than record whether this module wrote them, but it was worth pinning precisely
because the obvious "fix" if someone saw that in isolation would be to key the
override on whether a format was applied. The test says so in as many words.
Between the two, each direction is now caught: dropping the override fails the
formatted case and the CSV case, and widening it to every temporal column fails
the unformatted one.
One thing I checked while in there and left alone: on the XLSX branch the
override is inert, because `_read_excel_for_client_processing` takes the first
column as the index, so that payload reports `['NUMERIC']` with
`colnames: ['cnt']` both with and without this change. Pre-existing and
unrelated, just noting it so it does not look like a gap this introduces.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(reports): apply d3TimeFormat in table post-processing [superset]
Posted by "codeant-ai-for-open-source[bot] (via GitHub)" <gi...@apache.org>.
codeant-ai-for-open-source[bot] commented on code in PR #45410:
URL: https://github.com/apache/superset/pull/45410#discussion_r4326589459
##########
superset/charts/client_processing.py:
##########
@@ -1237,6 +1250,20 @@ def label_of(name: str) -> str:
# left untouched, matching the browser.
for column in df.columns:
config = config_by_label.get(column) or {}
+
+ # Mirror the Table plugin's first formatter branch
+ # (``if (isTime || config.d3TimeFormat)`` in ``transformProps.ts``):
+ # an explicit per-column ``d3TimeFormat`` formats a temporal column and
+ # takes precedence over number formatting. Scoped to datetime columns
+ # with an explicit format; the bare ``isTime`` fallback,
+ # ``tableTimestampFormat`` default, SMART_DATE granularity, and string
+ # temporal columns are intentionally left to a follow-up, since they
+ # need a full server-side d3-time-format port rather than ``strftime``.
+ time_format = config.get("d3TimeFormat")
+ if time_format and pd.api.types.is_datetime64_any_dtype(df[column]):
+ format_time_column(df, column, time_format)
Review Comment:
โ
**CodeAnt verified this suggestion was addressed in subsequent commits and marked this thread resolved** as of `8956664`.
The datetime-format branch now skips `SMART_DATE` and formats only formats recognized as `strftime`-expressible, so `smart_date` is not passed to `format_time_column`.
<sub>If that's not right, unresolve this thread and CodeAnt will leave it open.</sub>
<!-- codeant-auto-resolve-reply -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(reports): apply d3TimeFormat in table post-processing [superset]
Posted by "sadpandajoe (via GitHub)" <gi...@apache.org>.
sadpandajoe commented on code in PR #45410:
URL: https://github.com/apache/superset/pull/45410#discussion_r4321248725
##########
superset/charts/client_processing.py:
##########
@@ -835,6 +852,26 @@ def union_currency_context(
}
+def is_strftime_expressible(d3_time_format: str) -> bool:
+ """Whether every directive in a d3 time format means the same to strftime."""
+ directives = re.findall(r"%(.)", d3_time_format)
+ return bool(directives) and all(
+ directive in SHARED_TIME_DIRECTIVES for directive in directives
+ )
+
+
+def format_time_column(df: pd.DataFrame, column: Any, d3_time_format: str) -> None:
+ """Format a temporal column in place with an explicit d3 time format.
+
+ The caller gates this on ``is_strftime_expressible``, so every directive in
+ ``d3_time_format`` means the same thing here as it does in the browser. Null
+ values are left as-is rather than rendered as the string ``"NaT"``.
+ """
+ df[column] = df[column].apply(
+ lambda value: value.strftime(d3_time_format) if pd.notna(value) else value
Review Comment:
For a timezone-aware column, `strftime` formats in the value's own offset, whereas the chart's default formatter (`utcFormat`) formats in UTC. With `%Y-%m-%d %H:%M`, `2024-01-01 00:30+05:30` renders as `2024-01-01 00:30` in the report but `2023-12-31 19:00` in the chart. Should aware values be converted to UTC before formatting so the report matches what the chart displays?
##########
superset/charts/client_processing.py:
##########
@@ -835,6 +852,26 @@ def union_currency_context(
}
+def is_strftime_expressible(d3_time_format: str) -> bool:
+ """Whether every directive in a d3 time format means the same to strftime."""
+ directives = re.findall(r"%(.)", d3_time_format)
Review Comment:
A `local!`-prefixed format (the prefix `TimeFormatterRegistry` strips before choosing local-time formatting) passes this check, since `%Y` etc. are all in the allowlist, and then goes to `strftime` unchanged. `d3TimeFormat: "local!%Y-%m-%d"` would render as the literal text `local!2024-01-15` in the report, while the chart shows `2024-01-15`. Could the prefix be rejected here, so the column is left raw like the other unsupported formats?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(reports): apply d3TimeFormat in table post-processing [superset]
Posted by "codeant-ai-for-open-source[bot] (via GitHub)" <gi...@apache.org>.
codeant-ai-for-open-source[bot] commented on code in PR #45410:
URL: https://github.com/apache/superset/pull/45410#discussion_r4333971781
##########
superset/charts/client_processing.py:
##########
@@ -1237,6 +1250,20 @@ def label_of(name: str) -> str:
# left untouched, matching the browser.
for column in df.columns:
config = config_by_label.get(column) or {}
+
+ # Mirror the Table plugin's first formatter branch
+ # (``if (isTime || config.d3TimeFormat)`` in ``transformProps.ts``):
+ # an explicit per-column ``d3TimeFormat`` formats a temporal column and
+ # takes precedence over number formatting. Scoped to datetime columns
+ # with an explicit format; the bare ``isTime`` fallback,
+ # ``tableTimestampFormat`` default, SMART_DATE granularity, and string
+ # temporal columns are intentionally left to a follow-up, since they
+ # need a full server-side d3-time-format port rather than ``strftime``.
+ time_format = config.get("d3TimeFormat")
+ if time_format and pd.api.types.is_datetime64_any_dtype(df[column]):
+ format_time_column(df, column, time_format)
Review Comment:
โ
**CodeAnt verified this suggestion was addressed in subsequent commits and marked this thread resolved** as of `a7d45f2`.
Formatting now runs only when `is_strftime_expressible(time_format)` passes, and the code explicitly leaves adaptive granularity for a future d3-time-format implementation.
<sub>If that's not right, unresolve this thread and CodeAnt will leave it open.</sub>
<!-- codeant-auto-resolve-reply -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(reports): apply d3TimeFormat in table post-processing [superset]
Posted by "codeant-ai-for-open-source[bot] (via GitHub)" <gi...@apache.org>.
codeant-ai-for-open-source[bot] commented on code in PR #45410:
URL: https://github.com/apache/superset/pull/45410#discussion_r4301781594
##########
superset/charts/client_processing.py:
##########
@@ -1237,6 +1250,20 @@ def label_of(name: str) -> str:
# left untouched, matching the browser.
for column in df.columns:
config = config_by_label.get(column) or {}
+
+ # Mirror the Table plugin's first formatter branch
+ # (``if (isTime || config.d3TimeFormat)`` in ``transformProps.ts``):
+ # an explicit per-column ``d3TimeFormat`` formats a temporal column and
+ # takes precedence over number formatting. Scoped to datetime columns
+ # with an explicit format; the bare ``isTime`` fallback,
+ # ``tableTimestampFormat`` default, SMART_DATE granularity, and string
+ # temporal columns are intentionally left to a follow-up, since they
+ # need a full server-side d3-time-format port rather than ``strftime``.
+ time_format = config.get("d3TimeFormat")
+ if time_format and pd.api.types.is_datetime64_any_dtype(df[column]):
+ format_time_column(df, column, time_format)
Review Comment:
**Suggestion:** `smart_date` is a valid adaptive-format option, but this passes it to `strftime`, so every datetime cell becomes the literal `smart_date` instead of a date.
**Assessment:** ๐ `Major` ยท ๐ `Occurrence: Sometimes` ยท ๐ท๏ธ `Api mismatch`
[](https://docs.codeant.ai/cli/resolve-pr-comments-skill) [](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=ba8feec9f0474f798b27773d9ccf8d41&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=ba8feec9f0474f798b27773d9ccf8d41&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset)
<details>
<summary><b>Prompt for AI Agent ๐ค </b></summary>
```mdx
This is a comment left during a code review.
**Path:** superset/charts/client_processing.py
**Line:** 1262:1264
**Comment:**
*Api Mismatch: `smart_date` is a valid adaptive-format option, but this passes it to `strftime`, so every datetime cell becomes the literal `smart_date` instead of a date.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F45410&comment_hash=9735c96d93146ff67ffd774a99ca0816c8e3fc281b616195437e0fae65cc6cd3&reaction=like'>๐</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F45410&comment_hash=9735c96d93146ff67ffd774a99ca0816c8e3fc281b616195437e0fae65cc6cd3&reaction=dislike'>๐</a>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(reports): apply d3TimeFormat in table post-processing [superset]
Posted by "sadpandajoe (via GitHub)" <gi...@apache.org>.
sadpandajoe commented on code in PR #45410:
URL: https://github.com/apache/superset/pull/45410#discussion_r4386900424
##########
superset/charts/client_processing.py:
##########
@@ -78,6 +80,37 @@
# column meta and formatter selection, so it shares this post-processor.
AG_GRID_TABLE = "ag-grid-table"
+# "Adaptive formatting" in the time format controls, mirroring ``SMART_DATE_ID``
+# in ``superset-ui-core/time-format``. It is an alias the browser resolves to a
+# formatter that picks a representation per value, not a format string.
+SMART_DATE = "smart_date"
Review Comment:
`SMART_DATE` is defined here but nothing reads it: `is_strftime_expressible` rejects `smart_date` through the no-directive check, and the test hardcodes the string. The constant and its comment read as if they gate that behavior, so someone changing one would expect the other to follow. Should it be removed, or is it meant to be compared against in `is_strftime_expressible`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(reports): apply d3TimeFormat in table post-processing [superset]
Posted by "sadpandajoe (via GitHub)" <gi...@apache.org>.
sadpandajoe commented on code in PR #45410:
URL: https://github.com/apache/superset/pull/45410#discussion_r4385565304
##########
superset/charts/client_processing.py:
##########
@@ -1372,6 +1455,31 @@ def label_of(name: str) -> str:
# left untouched, matching the browser.
for column in df.columns:
config = config_by_label.get(column) or {}
+
+ # Mirror the Table plugin's first formatter branch
+ # (``if (isTime || config.d3TimeFormat)`` in ``transformProps.ts``):
+ # an explicit per-column ``d3TimeFormat`` formats a temporal column and
+ # takes precedence over number formatting. Scoped to a datetime column
+ # carrying a format every directive of which ``strftime`` reads the same
+ # way. A column of another dtype has no ``strftime`` to call, and the
+ # bare ``isTime`` fallback, the ``tableTimestampFormat`` default,
+ # adaptive granularity and string temporal columns are left to a
+ # follow-up: they need a full server-side d3-time-format port rather
+ # than ``strftime``.
+ # ``TimeFormatterRegistry.get`` trims the format before building the
+ # formatter while d3 itself does not, so ``utcFormat(" %Y ")`` returns
+ # ``" 2024 "`` and without the strip the report keeps padding the chart
+ # dropped. One that is only whitespace strips to empty and falls
+ # through unformatted, like one whose directives cannot be reproduced.
+ time_format = (config.get("d3TimeFormat") or "").strip()
Review Comment:
Nothing in the suite pins the `or ""` here. Every table test either omits `d3TimeFormat` or passes a string, so swapping this for `config.get("d3TimeFormat", "").strip()` keeps them all green while a column whose format was cleared and saved as JSON `null` raises `AttributeError: 'NoneType' object has no attribute 'strip'` and fails the whole report. Could you add a `table()` test with `{"ds": {"d3TimeFormat": None}}` on a datetime column next to a number-formatted metric, asserting the timestamp is untouched and the metric still formats?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(reports): apply d3TimeFormat in table post-processing [superset]
Posted by "SEPURI-SAI-KRISHNA (via GitHub)" <gi...@apache.org>.
SEPURI-SAI-KRISHNA commented on code in PR #45410:
URL: https://github.com/apache/superset/pull/45410#discussion_r4333965527
##########
superset/charts/client_processing.py:
##########
@@ -1304,6 +1390,34 @@ def _read_excel_for_client_processing(
return df.set_index(first_column)
+def payload_coltypes(
+ df: pd.DataFrame,
+ datasource: Optional[Union["BaseDatasource", "Query"]],
+) -> list[GenericDataType]:
+ """The payload's column types, reporting a text-rendered column as text.
+
+ ``extract_dataframe_dtypes`` lets the dataset's ``is_dttm`` flag decide the
+ type, so a temporal column a post-processor has rendered as text is still
+ reported as ``TEMPORAL``. ``chart_data_to_dataframe`` casts every
+ ``TEMPORAL`` column back to ``datetime64[ms]`` to build the notification
+ table, which would re-parse those strings and lose the configured format,
+ and would read ``02/01/2024`` month-first while reading ``15/01/2024``
+ day-first.
+
+ Only a column whose values really are strings is reported differently. The
+ XLSX path hands these types to ``apply_column_types`` to get date cells out
+ of a frame whose dates are still dates, and that frame is untouched here.
+ """
+ coltypes = extract_dataframe_dtypes(df, datasource)
+ return [
+ GenericDataType.STRING
+ if coltype == GenericDataType.TEMPORAL
+ and pd.api.types.infer_dtype(df.iloc[:, index], skipna=True) == "string"
Review Comment:
Confirmed, and the wrong date reproduces. On pandas 2.3.3 it is the `NaT`
specifically:
infer_dtype(["02/01/2024", pd.NaT], skipna=True) -> "mixed"
infer_dtype(["02/01/2024", None], skipna=True) -> "string"
`skipna` drops `None` and `NaN` but counts a `NaT` as a value of its own type,
so the column stays `TEMPORAL` and the cast reads the string month-first:
pd.Series(["02/01/2024", pd.NaT]).astype("datetime64[ms]")
-> [Timestamp("2024-02-01 00:00:00"), NaT]
With `%d/%m/%Y` on 2 January that is 1 February in the report, and any null in
the column was enough to undo the format.
Fixed by reading only the non-null values:
pd.api.types.infer_dtype(df.iloc[:, index].dropna(), skipna=False)
I kept it keyed on the values rather than tracking which columns `table()`
formatted, because the type has to follow the values for the CSV branch too,
where `read_csv` hands back text this module never wrote. A column of nothing
but nulls reads as empty rather than text and keeps the type the dataset gives
it, which is what the XLSX branch needs to get date cells.
New test `test_table_report_time_format_survives_a_null_in_the_column`. It
fails on the previous head with `TEMPORAL`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(reports): apply d3TimeFormat in table post-processing [superset]
Posted by "codeant-ai-for-open-source[bot] (via GitHub)" <gi...@apache.org>.
codeant-ai-for-open-source[bot] commented on code in PR #45410:
URL: https://github.com/apache/superset/pull/45410#discussion_r4303476632
##########
superset/charts/client_processing.py:
##########
@@ -1237,6 +1250,20 @@ def label_of(name: str) -> str:
# left untouched, matching the browser.
for column in df.columns:
config = config_by_label.get(column) or {}
+
+ # Mirror the Table plugin's first formatter branch
+ # (``if (isTime || config.d3TimeFormat)`` in ``transformProps.ts``):
+ # an explicit per-column ``d3TimeFormat`` formats a temporal column and
+ # takes precedence over number formatting. Scoped to datetime columns
+ # with an explicit format; the bare ``isTime`` fallback,
+ # ``tableTimestampFormat`` default, SMART_DATE granularity, and string
+ # temporal columns are intentionally left to a follow-up, since they
+ # need a full server-side d3-time-format port rather than ``strftime``.
+ time_format = config.get("d3TimeFormat")
+ if time_format and pd.api.types.is_datetime64_any_dtype(df[column]):
+ format_time_column(df, column, time_format)
Review Comment:
โ
**CodeAnt verified this suggestion was addressed in subsequent commits and marked this thread resolved** as of `983ccd9`.
The datetime-format branch now skips `SMART_DATE` and only passes formats containing `%` to `format_time_column`, preventing `smart_date` from being sent to `strftime`.
<sub>If that's not right, unresolve this thread and CodeAnt will leave it open.</sub>
<!-- codeant-auto-resolve-reply -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(reports): apply d3TimeFormat in table post-processing [superset]
Posted by "sadpandajoe (via GitHub)" <gi...@apache.org>.
sadpandajoe commented on code in PR #45410:
URL: https://github.com/apache/superset/pull/45410#discussion_r4363709709
##########
superset/charts/client_processing.py:
##########
@@ -835,6 +863,51 @@ def union_currency_context(
}
+def is_strftime_expressible(d3_time_format: str) -> bool:
+ """Whether ``strftime`` renders a d3 time format the way the browser does."""
+ if LOCAL_TIME_PREFIX in d3_time_format:
+ # ``TimeFormatterRegistry.get`` strips the marker before building the
+ # formatter, so leaving it in would render it as literal text.
+ return False
+ directives = re.findall(r"%(.)", d3_time_format)
+ if not directives:
+ # Nothing for ``strftime`` to substitute, so it hands the string back
+ # unchanged and every cell would read the format itself. This is also
+ # what rejects the aliases the time format controls offer, which the
+ # browser resolves to a formatter that picks a representation per value
+ # rather than to a format string: ``smart_date`` ("Adaptive
+ # formatting", ``SMART_DATE_ID`` in ``superset-ui-core/time-format``)
+ # among them.
+ return False
+ if "%" in re.sub(r"%.", "", d3_time_format):
+ # A ``%`` no directive consumed, as the trailing one in ``%Y%``. d3
+ # drops it (``2021``) and ``strftime`` copies it (``2021%``), so the
+ # two disagree even where every directive is shared. ``%%`` is not
+ # this case: both read it as one escaped ``%``.
+ return False
+ return all(directive in SHARED_TIME_DIRECTIVES for directive in directives)
+
+
+def format_time_column(df: pd.DataFrame, column: Any, d3_time_format: str) -> None:
+ """Format a temporal column in place with an explicit d3 time format.
+
+ The caller gates this on ``is_strftime_expressible``, so every directive in
+ ``d3_time_format`` means the same thing here as it does in the browser. Null
+ values are left as-is rather than rendered as the string ``"NaT"``.
+
+ An aware value is read in UTC. ``createD3TimeFormatter`` builds the
+ formatter with ``utcFormat`` unless the format asks for local time, which
+ ``is_strftime_expressible`` rejects, whereas ``strftime`` would use the
+ value's own offset and put a different instant in the report.
+ """
+ values = df[column]
+ if isinstance(values.dtype, pd.DatetimeTZDtype):
+ values = values.dt.tz_convert("UTC")
+ df[column] = values.apply(
+ lambda value: value.strftime(d3_time_format) if pd.notna(value) else value
Review Comment:
Passing the format through unchanged leaves surrounding whitespace in the cell: the browser's formatter registry trims the format before building the formatter, so `d3TimeFormat: " %Y "` shows `2024` in the chart but `" 2024 "` in the report. Would it make sense to `.strip()` the format before validating and formatting it, as the registry does?
##########
superset/charts/client_processing.py:
##########
@@ -835,6 +863,51 @@ def union_currency_context(
}
+def is_strftime_expressible(d3_time_format: str) -> bool:
+ """Whether ``strftime`` renders a d3 time format the way the browser does."""
+ if LOCAL_TIME_PREFIX in d3_time_format:
+ # ``TimeFormatterRegistry.get`` strips the marker before building the
+ # formatter, so leaving it in would render it as literal text.
+ return False
+ directives = re.findall(r"%(.)", d3_time_format)
+ if not directives:
+ # Nothing for ``strftime`` to substitute, so it hands the string back
+ # unchanged and every cell would read the format itself. This is also
+ # what rejects the aliases the time format controls offer, which the
+ # browser resolves to a formatter that picks a representation per value
+ # rather than to a format string: ``smart_date`` ("Adaptive
+ # formatting", ``SMART_DATE_ID`` in ``superset-ui-core/time-format``)
+ # among them.
+ return False
+ if "%" in re.sub(r"%.", "", d3_time_format):
+ # A ``%`` no directive consumed, as the trailing one in ``%Y%``. d3
+ # drops it (``2021``) and ``strftime`` copies it (``2021%``), so the
+ # two disagree even where every directive is shared. ``%%`` is not
+ # this case: both read it as one escaped ``%``.
+ return False
+ return all(directive in SHARED_TIME_DIRECTIVES for directive in directives)
+
+
+def format_time_column(df: pd.DataFrame, column: Any, d3_time_format: str) -> None:
+ """Format a temporal column in place with an explicit d3 time format.
+
+ The caller gates this on ``is_strftime_expressible``, so every directive in
+ ``d3_time_format`` means the same thing here as it does in the browser. Null
+ values are left as-is rather than rendered as the string ``"NaT"``.
Review Comment:
The docstring says a null would be rendered as the string `"NaT"`, but `pd.NaT.strftime(...)` raises `ValueError: NaTType does not support strftime`, so without the `pd.notna` guard a single null timestamp fails the whole report rather than printing text. Could the comment describe that instead? (The same wording appears in `test_table_time_format_keeps_nulls`.)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(reports): apply d3TimeFormat in table post-processing [superset]
Posted by "sadpandajoe (via GitHub)" <gi...@apache.org>.
sadpandajoe commented on code in PR #45410:
URL: https://github.com/apache/superset/pull/45410#discussion_r4320330036
##########
superset/charts/client_processing.py:
##########
@@ -1237,6 +1255,27 @@ def label_of(name: str) -> str:
# left untouched, matching the browser.
for column in df.columns:
config = config_by_label.get(column) or {}
+
+ # Mirror the Table plugin's first formatter branch
+ # (``if (isTime || config.d3TimeFormat)`` in ``transformProps.ts``):
+ # an explicit per-column ``d3TimeFormat`` formats a temporal column and
+ # takes precedence over number formatting. Scoped to datetime columns
+ # with an explicit ``strftime``-style format; ``smart_date`` and any
+ # other alias is skipped so it is never rendered as literal text. The
+ # bare ``isTime`` fallback, ``tableTimestampFormat`` default,
+ # SMART_DATE granularity, and string temporal columns are left to a
+ # follow-up, since they need a full server-side d3-time-format port
+ # rather than ``strftime``.
+ time_format = config.get("d3TimeFormat")
+ if (
+ time_format
+ and time_format != SMART_DATE
+ and "%" in time_format
+ and pd.api.types.is_datetime64_any_dtype(df[column])
+ ):
+ format_time_column(df, column, time_format)
Review Comment:
The formatted strings don't survive to the report: `extract_dataframe_dtypes` still tags this column `TEMPORAL` (it goes by the dataset's `is_dttm`, not the new dtype), and `chart_data_to_dataframe` then runs `astype("datetime64[ms]")` on it. With `%Y` the cells come back as `2024-01-01`, and with `%d/%m/%Y` the string `02/01/2024` is re-parsed month-first as Feb 1 instead of Jan 2. Should the formatted column be excluded from that re-cast (or tagged non-temporal in `coltypes`) so the notification actually shows the configured format?
##########
superset/charts/client_processing.py:
##########
@@ -835,6 +840,19 @@ def union_currency_context(
}
+def format_time_column(df: pd.DataFrame, column: Any, d3_time_format: str) -> None:
+ """Format a temporal column in place with an explicit d3 time format.
+
+ Only the directives shared between d3-time-format and ``strftime``
+ (``%Y``, ``%m``, ``%d``, ``%H``, ``%M``, ``%S`` and similar) are supported,
+ which covers the formats users configure in practice. Null values are left
+ as-is rather than rendered as the string ``"NaT"``.
+ """
+ df[column] = df[column].apply(
+ lambda value: value.strftime(d3_time_format) if pd.notna(value) else value
Review Comment:
The `%` check doesn't restrict the format to the directives shared with d3, so d3-only ones are emitted literally: `%Y Q%q` renders as `2024 Q%q` and `%L` stays `%L`, whereas the browser shows `2024 Q2` / milliseconds. Would it be safer to leave the column raw when the format contains a directive `strftime` can't express, like the number path does for formats it can't handle?
##########
tests/unit_tests/charts/test_client_processing.py:
##########
@@ -2488,6 +2488,60 @@ def test_table_applies_currency_format() -> None:
assert formatted["amount"].tolist() == ["$ 1,234.50"]
+def test_table_applies_time_format_alongside_number_format() -> None:
Review Comment:
The PR description and `format_time_column` docstring promise that null (`NaT`) is preserved, but both new tests use only valid timestamps, so removing the `pd.notna` guard would pass them. Could you add a `table()` case with `ds=[pd.Timestamp("2024-01-15"), pd.NaT]` and `d3TimeFormat="%Y"` asserting `"2024"` and a null in the second cell?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(reports): apply d3TimeFormat in table post-processing [superset]
Posted by "SEPURI-SAI-KRISHNA (via GitHub)" <gi...@apache.org>.
SEPURI-SAI-KRISHNA commented on code in PR #45410:
URL: https://github.com/apache/superset/pull/45410#discussion_r4321209217
##########
superset/charts/client_processing.py:
##########
@@ -835,6 +840,19 @@ def union_currency_context(
}
+def format_time_column(df: pd.DataFrame, column: Any, d3_time_format: str) -> None:
+ """Format a temporal column in place with an explicit d3 time format.
+
+ Only the directives shared between d3-time-format and ``strftime``
+ (``%Y``, ``%m``, ``%d``, ``%H``, ``%M``, ``%S`` and similar) are supported,
+ which covers the formats users configure in practice. Null values are left
+ as-is rather than rendered as the string ``"NaT"``.
+ """
+ df[column] = df[column].apply(
+ lambda value: value.strftime(d3_time_format) if pd.notna(value) else value
Review Comment:
Agreed, and it is worse than a mismatch: `strftime` does not raise on a directive it does not know, it copies it into the output.
```
%L '%L'
%q '%q'
%Q '%Q'
```
So the column is gated on an allowlist now, and anything outside it leaves the column raw, the way the number path leaves a format it cannot handle:
```python
SHARED_TIME_DIRECTIVES = frozenset("aAbBdfGHIjmMpSuUVwWyY%")
def is_strftime_expressible(d3_time_format: str) -> bool:
"""Whether every directive in a d3 time format means the same to strftime."""
directives = re.findall(r"%(.)", d3_time_format)
return bool(directives) and all(
directive in SHARED_TIME_DIRECTIVES for directive in directives
)
```
What I left out, with reasons, since the list is the whole argument:
`%c`, `%x`, `%X` are locale-composite. Python builds them from the server's locale and d3 from the chart's, so `%x` gives `04/02/24` here and a different order elsewhere. Those are a silent mismatch rather than a visible one, which is worse.
`%L`, `%q`, `%Q` are d3-only, the ones above.
`%e` and `%s` work through the platform's C library rather than CPython, and `%g` is not d3's two-digit ISO year. `%Z` and `%z` render empty for a naive datetime here and offsets in the browser.
`%f`, `%G`, `%u`, `%V`, `%U`, `%W` and `%j` are in, having checked each against d3-time-format's table rather than assuming: microseconds, ISO year, ISO weekday, ISO week, the two week-of-year variants and day-of-year all agree.
`test_table_skips_a_time_format_strftime_cannot_express` covers `%Y Q%q`, `%H:%M:%S.%L`, `%Q`, `%-d %b` and `%x`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(reports): apply d3TimeFormat in table post-processing [superset]
Posted by "SEPURI-SAI-KRISHNA (via GitHub)" <gi...@apache.org>.
SEPURI-SAI-KRISHNA commented on code in PR #45410:
URL: https://github.com/apache/superset/pull/45410#discussion_r4385015825
##########
superset/charts/client_processing.py:
##########
@@ -835,6 +863,51 @@ def union_currency_context(
}
+def is_strftime_expressible(d3_time_format: str) -> bool:
+ """Whether ``strftime`` renders a d3 time format the way the browser does."""
+ if LOCAL_TIME_PREFIX in d3_time_format:
+ # ``TimeFormatterRegistry.get`` strips the marker before building the
+ # formatter, so leaving it in would render it as literal text.
+ return False
+ directives = re.findall(r"%(.)", d3_time_format)
+ if not directives:
+ # Nothing for ``strftime`` to substitute, so it hands the string back
+ # unchanged and every cell would read the format itself. This is also
+ # what rejects the aliases the time format controls offer, which the
+ # browser resolves to a formatter that picks a representation per value
+ # rather than to a format string: ``smart_date`` ("Adaptive
+ # formatting", ``SMART_DATE_ID`` in ``superset-ui-core/time-format``)
+ # among them.
+ return False
+ if "%" in re.sub(r"%.", "", d3_time_format):
+ # A ``%`` no directive consumed, as the trailing one in ``%Y%``. d3
+ # drops it (``2021``) and ``strftime`` copies it (``2021%``), so the
+ # two disagree even where every directive is shared. ``%%`` is not
+ # this case: both read it as one escaped ``%``.
+ return False
+ return all(directive in SHARED_TIME_DIRECTIVES for directive in directives)
+
+
+def format_time_column(df: pd.DataFrame, column: Any, d3_time_format: str) -> None:
+ """Format a temporal column in place with an explicit d3 time format.
+
+ The caller gates this on ``is_strftime_expressible``, so every directive in
+ ``d3_time_format`` means the same thing here as it does in the browser. Null
+ values are left as-is rather than rendered as the string ``"NaT"``.
+
+ An aware value is read in UTC. ``createD3TimeFormatter`` builds the
+ formatter with ``utcFormat`` unless the format asks for local time, which
+ ``is_strftime_expressible`` rejects, whereas ``strftime`` would use the
+ value's own offset and put a different instant in the report.
+ """
+ values = df[column]
+ if isinstance(values.dtype, pd.DatetimeTZDtype):
+ values = values.dt.tz_convert("UTC")
+ df[column] = values.apply(
+ lambda value: value.strftime(d3_time_format) if pd.notna(value) else value
Review Comment:
Confirmed, and the trim is the registry's, not d3's. `utcFormat(" %Y ")`
returns `" 2024 "` called directly; `TimeFormatterRegistry.get` trims at
`TimeFormatterRegistry.ts:51` before building the formatter. The gate accepts
`" %Y "`, so the report did render `' 2024 '`.
Stripping at the call site so the gate and the formatting both see the trimmed
format. The new test parametrizes five whitespace shapes and is the only
failure when the strip is reverted, and when it is narrowed to `lstrip`.
Two limits, in the docstring rather than papered over. `str.strip` is not
`String.prototype.trim`: a leading `U+FEFF` is trimmed in the browser and kept
here, `U+001C`-`U+001F` and `U+0085` the other way round. And a whitespace-only
format strips to empty and falls through unformatted, so the report keeps the
timestamp where the chart renders empty. That is the same exit as a format this
cannot reproduce, but say so if you would rather it matched the registry.
Sibling left out: `NumberFormatterRegistry.get` trims at the same spot while
the number path does not strip, so `format_number_with_config(" .1f ", None,
1234.5678)` returns `"1234.5678"`, not `"1234.6"`. Predates this PR, different
config key, happy to send separately.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(reports): apply d3TimeFormat in table post-processing [superset]
Posted by "SEPURI-SAI-KRISHNA (via GitHub)" <gi...@apache.org>.
SEPURI-SAI-KRISHNA commented on code in PR #45410:
URL: https://github.com/apache/superset/pull/45410#discussion_r4333967110
##########
superset/charts/client_processing.py:
##########
@@ -1237,6 +1302,27 @@ def label_of(name: str) -> str:
# left untouched, matching the browser.
for column in df.columns:
config = config_by_label.get(column) or {}
+
+ # Mirror the Table plugin's first formatter branch
+ # (``if (isTime || config.d3TimeFormat)`` in ``transformProps.ts``):
+ # an explicit per-column ``d3TimeFormat`` formats a temporal column and
+ # takes precedence over number formatting. Scoped to datetime columns
+ # with an explicit ``strftime``-style format; ``smart_date`` and any
+ # other alias is skipped so it is never rendered as literal text. The
+ # bare ``isTime`` fallback, ``tableTimestampFormat`` default,
+ # SMART_DATE granularity, and string temporal columns are left to a
+ # follow-up, since they need a full server-side d3-time-format port
+ # rather than ``strftime``.
+ time_format = config.get("d3TimeFormat")
+ if (
+ time_format
+ and time_format != SMART_DATE
+ and is_strftime_expressible(time_format)
+ and pd.api.types.is_datetime64_any_dtype(df[column])
Review Comment:
You are right, and the guard is load-bearing: a string column carrying
`d3TimeFormat` reaches `.strftime` and raises `AttributeError` in the worker.
Added `test_table_skips_a_time_format_on_a_non_datetime_column`, parametrized
over a string column and an integer one, asserting both come back unchanged.
Deleting the `is_datetime64_any_dtype` line fails it.
Running that same check over the rest of the diff found a second gate with no
test behind it, so it is in the same commit. `is_strftime_expressible` requires
at least one `%` directive, and nothing pinned that. Without it a format like
`yyyy-mm-dd` or `Date` is accepted, `strftime` finds nothing to substitute and
hands the string back as it is, and every cell in the column reads
`yyyy-mm-dd`. `test_table_skips_a_time_format_with_no_directive` covers it.
That rule is also what rejects `smart_date`, which means the separate
`time_format != SMART_DATE` comparison in `table()` could never fire. I dropped
it and moved the explanation next to the rule that does the work, so
`test_table_leaves_adaptive_time_format_alone` now pins the real reason.
I also re-ran the directive set against the real `d3-time-format` rather than
reading it, over seven dates covering a leap day, both year boundaries and an
ISO week that belongs to the previous year. Every directive in
`SHARED_TIME_DIRECTIVES` matches `strftime` on all of them. The ones that agree
on those dates but are still excluded are there for the reasons in the comment:
`%a %A %b %B %p` agree only because the test ran in English with no
`D3_TIME_FORMAT` set, and `%f` agrees only up to millisecond precision, since
d3 scales what it has (`123456` microseconds in, `123000` out).
For the rest of the diff, every added condition now has a named test that fails
when it alone is reverted, except two that no input can separate: the `continue`
after `format_time_column` (the column is text by then, so `is_numeric_dtype` is
already false and no later branch can act on it) and the `coltype == TEMPORAL`
gate in `payload_coltypes` (`extract_dataframe_dtypes` only departs from the
values when `is_dttm` is set, so the override would be a no-op everywhere it
newly applied). Both are in the description as kept for what they say rather
than for what they do, in case you would rather they go.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(reports): apply d3TimeFormat in table post-processing [superset]
Posted by "sadpandajoe (via GitHub)" <gi...@apache.org>.
sadpandajoe commented on code in PR #45410:
URL: https://github.com/apache/superset/pull/45410#discussion_r4326159037
##########
superset/charts/client_processing.py:
##########
@@ -1304,6 +1390,34 @@ def _read_excel_for_client_processing(
return df.set_index(first_column)
+def payload_coltypes(
+ df: pd.DataFrame,
+ datasource: Optional[Union["BaseDatasource", "Query"]],
+) -> list[GenericDataType]:
+ """The payload's column types, reporting a text-rendered column as text.
+
+ ``extract_dataframe_dtypes`` lets the dataset's ``is_dttm`` flag decide the
+ type, so a temporal column a post-processor has rendered as text is still
+ reported as ``TEMPORAL``. ``chart_data_to_dataframe`` casts every
+ ``TEMPORAL`` column back to ``datetime64[ms]`` to build the notification
+ table, which would re-parse those strings and lose the configured format,
+ and would read ``02/01/2024`` month-first while reading ``15/01/2024``
+ day-first.
+
+ Only a column whose values really are strings is reported differently. The
+ XLSX path hands these types to ``apply_column_types`` to get date cells out
+ of a frame whose dates are still dates, and that frame is untouched here.
+ """
+ coltypes = extract_dataframe_dtypes(df, datasource)
+ return [
+ GenericDataType.STRING
Review Comment:
This STRING override is the only new branch in the payload types, but nothing pins its other half. The only assertion on `coltypes` is the formatted case, so if every temporal column were reported as `STRING`, an untouched `is_dttm` datetime column would lose its `TEMPORAL` type, and the XLSX path would stop writing date cells for it, and every test would still pass. Could you add a case through `apply_client_processing` with an unformatted datetime column asserting it stays `TEMPORAL`, and one with a CSV-read (string) temporal column to show what it reports?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(reports): apply d3TimeFormat in table post-processing [superset]
Posted by "codeant-ai-for-open-source[bot] (via GitHub)" <gi...@apache.org>.
codeant-ai-for-open-source[bot] commented on code in PR #45410:
URL: https://github.com/apache/superset/pull/45410#discussion_r4356557972
##########
superset/charts/client_processing.py:
##########
@@ -1237,6 +1250,20 @@ def label_of(name: str) -> str:
# left untouched, matching the browser.
for column in df.columns:
config = config_by_label.get(column) or {}
+
+ # Mirror the Table plugin's first formatter branch
+ # (``if (isTime || config.d3TimeFormat)`` in ``transformProps.ts``):
+ # an explicit per-column ``d3TimeFormat`` formats a temporal column and
+ # takes precedence over number formatting. Scoped to datetime columns
+ # with an explicit format; the bare ``isTime`` fallback,
+ # ``tableTimestampFormat`` default, SMART_DATE granularity, and string
+ # temporal columns are intentionally left to a follow-up, since they
+ # need a full server-side d3-time-format port rather than ``strftime``.
+ time_format = config.get("d3TimeFormat")
+ if time_format and pd.api.types.is_datetime64_any_dtype(df[column]):
+ format_time_column(df, column, time_format)
Review Comment:
โ
**CodeAnt verified this suggestion was addressed in subsequent commits and marked this thread resolved** as of `615d305`.
The datetime formatter now runs only when `is_strftime_expressible(time_format)` is true. Adaptive formats such as `smart_date` are excluded from this `strftime` path.
<sub>If that's not right, unresolve this thread and CodeAnt will leave it open.</sub>
<!-- codeant-auto-resolve-reply -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(reports): apply d3TimeFormat in table post-processing [superset]
Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #45410:
URL: https://github.com/apache/superset/pull/45410#issuecomment-6230911034
## [Codecov](https://app.codecov.io/gh/apache/superset/pull/45410?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `16.66667%` with `5 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 64.92%. Comparing base ([`a5d29a9`](https://app.codecov.io/gh/apache/superset/commit/a5d29a9d27d2951e203ce65174d1af7cb5443d2f?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`31824eb`](https://app.codecov.io/gh/apache/superset/commit/31824eb2e7665f543380fc73c3ad73e310ed2013?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
| [Files with missing lines](https://app.codecov.io/gh/apache/superset/pull/45410?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [superset/charts/client\_processing.py](https://app.codecov.io/gh/apache/superset/pull/45410?src=pr&el=tree&filepath=superset%2Fcharts%2Fclient_processing.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c3VwZXJzZXQvY2hhcnRzL2NsaWVudF9wcm9jZXNzaW5nLnB5) | 16.66% | [5 Missing :warning: ](https://app.codecov.io/gh/apache/superset/pull/45410?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #45410 +/- ##
==========================================
- Coverage 65.15% 64.92% -0.24%
==========================================
Files 3048 3048
Lines 199619 199625 +6
Branches 46578 46579 +1
==========================================
- Hits 130063 129603 -460
- Misses 66629 67091 +462
- Partials 2927 2931 +4
```
| [Flag](https://app.codecov.io/gh/apache/superset/pull/45410/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage ฮ | |
|---|---|---|
| [hive](https://app.codecov.io/gh/apache/superset/pull/45410/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `34.98% <16.66%> (-0.01%)` | :arrow_down: |
| [mysql](https://app.codecov.io/gh/apache/superset/pull/45410/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [postgres](https://app.codecov.io/gh/apache/superset/pull/45410/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [presto](https://app.codecov.io/gh/apache/superset/pull/45410/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `36.73% <16.66%> (-0.01%)` | :arrow_down: |
| [python](https://app.codecov.io/gh/apache/superset/pull/45410/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `52.81% <16.66%> (-0.42%)` | :arrow_down: |
| [sqlite](https://app.codecov.io/gh/apache/superset/pull/45410/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `52.70% <16.66%> (-0.01%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
</details>
[:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/superset/pull/45410?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(reports): apply d3TimeFormat in table post-processing [superset]
Posted by "codeant-ai-for-open-source[bot] (via GitHub) via notifications" <no...@superset.apache.org>.
codeant-ai-for-open-source[bot] commented on code in PR #45410:
URL: https://github.com/apache/superset/pull/45410#discussion_r4630397936
##########
superset/charts/client_processing.py:
##########
@@ -1237,6 +1250,20 @@ def label_of(name: str) -> str:
# left untouched, matching the browser.
for column in df.columns:
config = config_by_label.get(column) or {}
+
+ # Mirror the Table plugin's first formatter branch
+ # (``if (isTime || config.d3TimeFormat)`` in ``transformProps.ts``):
+ # an explicit per-column ``d3TimeFormat`` formats a temporal column and
+ # takes precedence over number formatting. Scoped to datetime columns
+ # with an explicit format; the bare ``isTime`` fallback,
+ # ``tableTimestampFormat`` default, SMART_DATE granularity, and string
+ # temporal columns are intentionally left to a follow-up, since they
+ # need a full server-side d3-time-format port rather than ``strftime``.
+ time_format = config.get("d3TimeFormat")
+ if time_format and pd.api.types.is_datetime64_any_dtype(df[column]):
+ format_time_column(df, column, time_format)
Review Comment:
โ
**CodeAnt verified this suggestion was addressed in subsequent commits and marked this thread resolved** as of `dfd7845`.
This location now builds currency-context pivot options and disables value transforms with `show_values_as: None`; the datetime `strftime` call is no longer present.
<sub>If that's not right, unresolve this thread and CodeAnt will leave it open.</sub>
<!-- codeant-auto-resolve-reply -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(reports): apply d3TimeFormat in table post-processing [superset]
Posted by "SEPURI-SAI-KRISHNA (via GitHub)" <gi...@apache.org>.
SEPURI-SAI-KRISHNA commented on code in PR #45410:
URL: https://github.com/apache/superset/pull/45410#discussion_r4321181830
##########
superset/charts/client_processing.py:
##########
@@ -1237,6 +1255,27 @@ def label_of(name: str) -> str:
# left untouched, matching the browser.
for column in df.columns:
config = config_by_label.get(column) or {}
+
+ # Mirror the Table plugin's first formatter branch
+ # (``if (isTime || config.d3TimeFormat)`` in ``transformProps.ts``):
+ # an explicit per-column ``d3TimeFormat`` formats a temporal column and
+ # takes precedence over number formatting. Scoped to datetime columns
+ # with an explicit ``strftime``-style format; ``smart_date`` and any
+ # other alias is skipped so it is never rendered as literal text. The
+ # bare ``isTime`` fallback, ``tableTimestampFormat`` default,
+ # SMART_DATE granularity, and string temporal columns are left to a
+ # follow-up, since they need a full server-side d3-time-format port
+ # rather than ``strftime``.
+ time_format = config.get("d3TimeFormat")
+ if (
+ time_format
+ and time_format != SMART_DATE
+ and "%" in time_format
+ and pd.api.types.is_datetime64_any_dtype(df[column])
+ ):
+ format_time_column(df, column, time_format)
Review Comment:
You are right, and this was the finding that mattered most. The formatted strings did not reach the recipient at all. I measured it before fixing anything:
```
dtype of ds column : object
coltypes : ['TEMPORAL', 'NUMERIC']
notification table ->
ds cnt
0 2024-02-01 1
1 2024-01-15 2
```
That is `%d/%m/%Y` over `02/01/2024` and `15/01/2024`. The first is re-parsed month-first and the second day-first, because `15` cannot be a month, so a single column comes out with its dates inconsistent rather than uniformly shifted. `%Y` collapses to `2024-01-01` as you said.
The type now follows the values:
```python
coltypes = extract_dataframe_dtypes(df, datasource)
return [
GenericDataType.STRING
if coltype == GenericDataType.TEMPORAL
and pd.api.types.infer_dtype(df.iloc[:, index], skipna=True) == "string"
else coltype
for index, coltype in enumerate(coltypes)
]
```
Two things I checked before settling on that shape rather than changing the cast in `chart_data_to_dataframe`:
`coltypes` is not inert for every format. The XLSX branch passes it to `excel.apply_column_types` to get real date cells, so a blanket "a text column is not temporal" rule would have turned XLSX dates into strings. Testing the values rather than the format keeps that path untouched: the frame it hands over still holds datetimes, read back by `read_excel`, so nothing is downgraded. A plain temporal column on the JSON path is also untouched, since `DataFrame.from_dict` over the in-process rows gives it a `datetime64` dtype.
The cast itself has to stay. It exists because the payload's temporal values are not datetimes by the time the notification table reads them, which is still true for every column this PR does not format.
`test_table_report_time_format_survives_the_notification_table` now runs the whole path, from `apply_client_processing` through `chart_data_to_dataframe`, and asserts the cells are still `02/01/2024` and `15/01/2024` at the end. On the parent commit it fails at `coltypes[0] == STRING` with `TEMPORAL`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(reports): apply d3TimeFormat in table post-processing [superset]
Posted by "SEPURI-SAI-KRISHNA (via GitHub)" <gi...@apache.org>.
SEPURI-SAI-KRISHNA commented on code in PR #45410:
URL: https://github.com/apache/superset/pull/45410#discussion_r4322301027
##########
superset/charts/client_processing.py:
##########
@@ -835,6 +852,26 @@ def union_currency_context(
}
+def is_strftime_expressible(d3_time_format: str) -> bool:
+ """Whether every directive in a d3 time format means the same to strftime."""
+ directives = re.findall(r"%(.)", d3_time_format)
+ return bool(directives) and all(
+ directive in SHARED_TIME_DIRECTIVES for directive in directives
+ )
+
+
+def format_time_column(df: pd.DataFrame, column: Any, d3_time_format: str) -> None:
+ """Format a temporal column in place with an explicit d3 time format.
+
+ The caller gates this on ``is_strftime_expressible``, so every directive in
+ ``d3_time_format`` means the same thing here as it does in the browser. Null
+ values are left as-is rather than rendered as the string ``"NaT"``.
+ """
+ df[column] = df[column].apply(
+ lambda value: value.strftime(d3_time_format) if pd.notna(value) else value
Review Comment:
Right, and with your numbers:
```
strftime as is : 2024-01-01 00:30
strftime after UTC : 2023-12-31 19:00
```
`createD3TimeFormatter` is where it is decided, and UTC is the default:
```ts
if (typeof locale === 'undefined') {
const format = useLocalTime ? timeFormat : utcFormat;
formatFunc = format(formatString);
} else {
const localeObject = timeFormatLocale(locale);
formatFunc = useLocalTime
? localeObject.format(formatString)
: localeObject.utcFormat(formatString);
}
```
`useLocalTime` only comes from the `local!` marker, which is rejected in the sibling comment, so every format this post-processor accepts is a UTC one. Aware values are converted before formatting:
```python
values = df[column]
if isinstance(values.dtype, pd.DatetimeTZDtype):
values = values.dt.tz_convert("UTC")
```
The test on the dtype rather than on each value is deliberate: `is_datetime64_any_dtype` gates the caller, so the column is either `datetime64[ns]` with no offset or `datetime64[ns, tz]` with one offset for the whole column. A mixed-offset column arrives as object dtype and is never formatted here at all.
`test_table_time_format_reads_an_aware_value_in_utc` asserts `["2023-12-31 19:00"]` for `00:30+05:30`, and fails with `2024-01-01 00:30` without the conversion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(reports): apply d3TimeFormat in table post-processing [superset]
Posted by "SEPURI-SAI-KRISHNA (via GitHub)" <gi...@apache.org>.
SEPURI-SAI-KRISHNA commented on code in PR #45410:
URL: https://github.com/apache/superset/pull/45410#discussion_r4356559903
##########
superset/charts/client_processing.py:
##########
@@ -73,6 +75,32 @@
# The Excel equivalent, applied as a cell format so the value stays numeric.
EXCEL_PERCENT_FORMAT = "0.0%"
+# ``strftime`` copies a directive it does not recognize into its output
+# verbatim, so a d3-only one would reach the report as literal text: ``%L`` and
+# ``%q`` stay ``%L`` and ``%q`` rather than becoming milliseconds and a quarter.
+# These are the directives that carry the same meaning in d3-time-format and in
+# ``strftime``. Left out are:
+#
+# * the ones that name a weekday, a month or a period (``%a``, ``%A``, ``%b``,
+# ``%B``, ``%p``). The browser reads those names from the d3 locale
+# definition, which ``D3_TIME_FORMAT`` overrides key by key, while
+# ``strftime`` reads them from the worker's ``LC_TIME``. The two agree only
+# when nothing is configured and the worker runs in English.
+# * the locale-composite ones (``%c``, ``%x``, ``%X``), whose output follows
+# the server's locale rather than the chart's.
+# * ``%f``. d3 derives it from ``getUTCMilliseconds() * 1000``, so the browser's
+# microseconds always end in ``000``, while a warehouse column with sub-second
+# precision reaches ``strftime`` with the real value.
+# * the platform-dependent or d3-only ones (``%e``, ``%g``, ``%L``, ``%q``,
+# ``%Q``, ``%s``, ``%Z``, ``%z``).
+SHARED_TIME_DIRECTIVES = frozenset("dGHIjmMSuUVwWyY%")
Review Comment:
You are right, and this is the gap I should have caught: I ran the discrimination check over the branches in `is_strftime_expressible` and treated `SHARED_TIME_DIRECTIVES` as data rather than as fifteen separate claims. Dropping `%V` or `%G` does leave the suite green, and the d3 comparison was a script I ran rather than anything the suite holds.
Added `test_table_time_format_matches_d3_for_each_shared_directive`, one case per directive so a failure names the one that moved:
```
%d 01 %G 2020 %H 13 %I 01 %j 001 %m 01
%M 05 %S 09 %u 5 %U 00 %V 53 %w 5
%W 00 %y 21 %Y 2021 %% % %G-W%V-%u 2020-W53-5
```
Your suggested value is exactly right: 1 January 2021 is a Friday in ISO week 53 of 2020, and `%G-W%V-%u` is `2020-W53-5` in both. The timestamp is 2021-01-01 13:05:09 so `%I` separates from `%H` as well. Every expected string above is what `utcFormat` from `d3-time-format` returns for that instant, checked against `datetime.strftime` on the same instant.
Removing any single directive from the allowlist now fails its own case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(reports): apply d3TimeFormat in table post-processing [superset]
Posted by "SEPURI-SAI-KRISHNA (via GitHub)" <gi...@apache.org>.
SEPURI-SAI-KRISHNA commented on code in PR #45410:
URL: https://github.com/apache/superset/pull/45410#discussion_r4402832528
##########
superset/charts/client_processing.py:
##########
@@ -78,6 +80,37 @@
# column meta and formatter selection, so it shares this post-processor.
AG_GRID_TABLE = "ag-grid-table"
+# "Adaptive formatting" in the time format controls, mirroring ``SMART_DATE_ID``
+# in ``superset-ui-core/time-format``. It is an alias the browser resolves to a
+# formatter that picks a representation per value, not a format string.
+SMART_DATE = "smart_date"
Review Comment:
My leftover. I dropped the `time_format != SMART_DATE` comparison earlier in
this PR and left the constant behind. Removed, and all 314 stay green without
it.
Comparing against it instead would put back a branch no test can separate,
since the no-directive check already rejects `smart_date`. That is why the
comparison went. The explanation stays at the point of rejection.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(reports): apply d3TimeFormat in table post-processing [superset]
Posted by "SEPURI-SAI-KRISHNA (via GitHub)" <gi...@apache.org>.
SEPURI-SAI-KRISHNA commented on code in PR #45410:
URL: https://github.com/apache/superset/pull/45410#discussion_r4403744256
##########
tests/unit_tests/charts/test_client_processing.py:
##########
@@ -2875,6 +2877,551 @@ def test_table_applies_currency_format() -> None:
assert formatted["amount"].tolist() == ["$ 1,234.50"]
+def test_table_applies_time_format_alongside_number_format() -> None:
+ """
+ Table reports honor an explicit ``d3TimeFormat`` on a temporal column,
+ mirroring the Table plugin's first formatter branch, while still applying
+ number formats on the same call. Without the time branch the timestamp was
+ left raw even though the browser formats it.
+ """
+ df = pd.DataFrame.from_dict(
+ {
+ "ds": {
+ 0: pd.Timestamp("2024-01-15 13:45:00"),
+ 1: pd.Timestamp("2024-06-30 08:00:00"),
+ },
+ "SUM(num)": {0: 1234567.0, 1: 42.0},
+ }
+ )
+ form_data = {
+ "viz_type": "table",
+ "metrics": ["SUM(num)"],
+ "column_config": {
+ "ds": {"d3TimeFormat": "%Y"},
+ "SUM(num)": {"d3NumberFormat": ",.2f"},
+ },
+ }
+ formatted = table(df, form_data)
+ assert formatted["ds"].tolist() == ["2024", "2024"]
+ assert formatted["SUM(num)"].tolist() == ["1,234,567.00", "42.00"]
+
+
+def test_table_leaves_a_null_time_format_alone() -> None:
+ """
+ A ``d3TimeFormat`` of JSON ``null`` leaves the column alone.
+
+ ``column_config`` is free-form JSON out of the saved chart params, so the
+ key can be present and null. ``config.get("d3TimeFormat", "").strip()``
+ would raise ``AttributeError: 'NoneType' object has no attribute 'strip'``
+ and fail the whole report, and every other table test here passes either
+ way because none of them puts a null in the key.
+ """
+ df = pd.DataFrame.from_dict(
+ {
+ "ds": {0: pd.Timestamp("2024-04-02 13:45:00")},
+ "SUM(num)": {0: 1234.5678},
+ }
+ )
+ form_data = {
+ "viz_type": "table",
+ "metrics": ["SUM(num)"],
+ "column_config": {
+ "ds": {"d3TimeFormat": None},
+ "SUM(num)": {"d3NumberFormat": ",.1f"},
+ },
+ }
+ formatted = table(df, form_data)
+ assert formatted["ds"].tolist() == [pd.Timestamp("2024-04-02 13:45:00")]
+ assert formatted["SUM(num)"].tolist() == ["1,234.6"]
+
+
+@pytest.mark.parametrize(
+ ("time_format", "expected"),
+ [
+ ("%d", "01"),
+ ("%G", "2020"),
+ ("%H", "13"),
+ ("%I", "01"),
+ ("%j", "001"),
+ ("%m", "01"),
+ ("%M", "05"),
+ ("%S", "09"),
+ ("%u", "5"),
+ ("%U", "00"),
+ ("%V", "53"),
+ ("%w", "5"),
+ ("%W", "00"),
+ ("%y", "21"),
+ ("%Y", "2021"),
+ ("%%", "%"),
+ ("%G-W%V-%u", "2020-W53-5"),
+ ],
+)
+def test_table_time_format_matches_d3_for_each_shared_directive(
+ time_format: str, expected: str
+) -> None:
+ """
+ Every directive in ``SHARED_TIME_DIRECTIVES`` is held to d3's own output.
+
+ The expected strings are what ``utcFormat`` from ``d3-time-format`` returns
+ for this timestamp, so taking a directive out of the allowlist fails the
+ case that names it rather than passing unnoticed. 1 January 2021 is a
+ Friday in ISO week 53 of 2020, which is what separates ``%G`` from ``%Y``
+ and ``%V`` from ``%U`` and ``%W``; 13:05:09 separates ``%I`` from ``%H``.
+ """
+ df = pd.DataFrame.from_dict({"ds": {0: pd.Timestamp("2021-01-01 13:05:09")}})
+ form_data = {
+ "viz_type": "table",
+ "column_config": {"ds": {"d3TimeFormat": time_format}},
+ }
+
+ formatted = table(df, form_data)
+
+ assert formatted["ds"].tolist() == [expected]
+
+
+@pytest.mark.parametrize(
+ "time_format",
+ [
+ "%Y%",
+ "%d/%m/%Y%",
+ "%%%",
+ "%Y%%%",
+ ],
+)
+def test_table_skips_a_time_format_with_an_unconsumed_percent(
+ time_format: str,
+) -> None:
+ """
+ A ``%`` that no directive consumes is dropped by d3 and copied by
+ ``strftime``.
+
+ ``%Y%`` is ``2021`` in the chart and would be ``2021%`` in the report, and
+ ``%%%`` is ``%`` against ``%%``. Every directive in these is shared, so the
+ allowlist alone accepts them and the column has to be left to the browser
+ on the stray ``%`` instead.
+ """
+ df = pd.DataFrame.from_dict({"ds": {0: pd.Timestamp("2021-01-01 13:05:09")}})
+ form_data = {
+ "viz_type": "table",
+ "column_config": {"ds": {"d3TimeFormat": time_format}},
+ }
+
+ formatted = table(df, form_data)
+
+ assert formatted["ds"].tolist() == [pd.Timestamp("2021-01-01 13:05:09")]
+
+
+def test_table_leaves_adaptive_time_format_alone() -> None:
+ """
+ ``smart_date`` ("Adaptive formatting") is an alias the browser resolves to a
+ formatter, not a format string. It carries no ``%`` directive, so the rule
+ that a format has to have one is what keeps it away from ``strftime``,
+ which would hand the alias back unchanged and print it in every cell.
+ """
+ df = pd.DataFrame.from_dict(
+ {
+ "ds": {
+ 0: pd.Timestamp("2024-01-15 13:45:00"),
+ 1: pd.Timestamp("2024-06-30 08:00:00"),
+ },
+ }
+ )
+ form_data = {
+ "viz_type": "table",
+ "column_config": {"ds": {"d3TimeFormat": "smart_date"}},
+ }
+ formatted = table(df, form_data)
+ assert formatted["ds"].tolist() == [
+ pd.Timestamp("2024-01-15 13:45:00"),
+ pd.Timestamp("2024-06-30 08:00:00"),
+ ]
+
+
+@pytest.mark.parametrize(
+ "time_format",
+ [
+ "yyyy-mm-dd",
+ "Q1",
+ "Date",
+ "100%",
+ ],
+)
+def test_table_skips_a_time_format_with_no_directive(time_format: str) -> None:
+ """
+ A format carrying no ``%`` directive is not a format string.
+
+ ``strftime`` has nothing to substitute and returns the string it was given,
+ so accepting one of these would put ``yyyy-mm-dd`` in every cell of the
+ column instead of a date. This is the rule that also rejects the adaptive
+ alias.
+ """
+ df = pd.DataFrame.from_dict({"ds": {0: pd.Timestamp("2024-04-02 13:45:00")}})
+ form_data = {
+ "viz_type": "table",
+ "column_config": {"ds": {"d3TimeFormat": time_format}},
+ }
+
+ formatted = table(df, form_data)
+
+ assert formatted["ds"].tolist() == [pd.Timestamp("2024-04-02 13:45:00")]
+
+
+@pytest.mark.parametrize(
+ ("values", "expected"),
+ [
+ pytest.param(
+ {0: "2024-01-15", 1: "2024-06-30"},
+ ["2024-01-15", "2024-06-30"],
+ id="string column",
+ ),
+ pytest.param({0: 2024, 1: 2025}, [2024, 2025], id="integer column"),
+ ],
+)
+def test_table_skips_a_time_format_on_a_non_datetime_column(
+ values: dict[int, Any], expected: list[Any]
+) -> None:
+ """
+ ``d3TimeFormat`` on a column that is not datetime leaves the column alone.
+
+ A string or a number has no ``strftime`` to call, so formatting one would
+ raise ``AttributeError`` in the report worker. The browser reaches these
+ through a d3-time-format port, which parses the value first, and that is
+ what the follow-up needs rather than something ``strftime`` can stand in
+ for.
+ """
+ df = pd.DataFrame.from_dict({"ds": values})
+ form_data = {
+ "viz_type": "table",
+ "column_config": {"ds": {"d3TimeFormat": "%Y"}},
+ }
+
+ formatted = table(df, form_data)
+
+ assert formatted["ds"].tolist() == expected
+
+
+def test_table_time_format_keeps_nulls() -> None:
+ """
+ A missing timestamp stays missing, and does not fail the report.
+
+ ``strftime`` on a ``NaT`` does not produce the string ``"NaT"``; it raises
+ ``ValueError: NaTType does not support strftime``, so without the
+ ``pd.notna`` guard one null would fail the whole report rather than print
+ anything.
+ """
+ df = pd.DataFrame.from_dict(
+ {"ds": {0: pd.Timestamp("2024-01-15"), 1: pd.NaT}},
+ )
+ form_data = {
+ "viz_type": "table",
+ "column_config": {"ds": {"d3TimeFormat": "%Y"}},
+ }
+
+ formatted = table(df, form_data)
+
+ assert formatted["ds"].tolist()[0] == "2024"
+ assert pd.isna(formatted["ds"].tolist()[1])
+
+
+@pytest.mark.parametrize(
+ "time_format",
+ [" %Y-%m-%d ", " %Y-%m-%d", "%Y-%m-%d ", "\t%Y-%m-%d\n", "\r%Y-%m-%d\x0b"],
+)
+def test_table_time_format_strips_surrounding_whitespace(time_format: str) -> None:
+ """
+ A padded format renders like the chart, which trims before formatting.
+
+ ``TimeFormatterRegistry.get`` trims the format string before building the
+ formatter, while d3 itself does not: ``utcFormat(" %Y ")`` returns
+ ``" 2024 "``. Without the strip the report kept the padding the chart had
+ already dropped. Every character here is whitespace to both ``str.strip``
+ and ``String.prototype.trim``; the handful they disagree on is in
+ ``format_time_column``.
+ """
+ df = pd.DataFrame.from_dict({"ds": {0: pd.Timestamp("2024-01-15")}})
+ form_data = {
+ "viz_type": "table",
+ "column_config": {"ds": {"d3TimeFormat": time_format}},
+ }
+
+ formatted = table(df, form_data)
+
+ assert formatted["ds"].tolist() == ["2024-01-15"]
+
+
+@pytest.mark.parametrize(
+ "time_format",
+ [
+ "%Y Q%q",
+ "%H:%M:%S.%L",
+ "%Q",
+ "%-d %b",
+ "%x",
+ "local!%Y-%m-%d",
+ "%Y local!",
+ "%H:%M:%S.%f",
Review Comment:
Confirmed. Adding `Z` keeps all 315 green, and they diverge both ways: after
the UTC conversion `strftime("%Y %Z")` is `2024 UTC` against d3's `2024
+0000`, and on a naive value `strftime` writes nothing where d3 still writes
`+0000`.
The gap was coverage rather than one letter, so I ran every letter through
both and pinned each one they disagree on that was not already covered: `%z
%c %X %s %C %D %F %h %k %l %n %P %r %R %t %T`. Most are ones `strftime`
substitutes while d3 copies the bare letter, so `%T` would read `13:45:09`
against a literal `T` in the chart. `%s` is the odd one: d3 reads the epoch in
UTC and glibc in local time, so the two sit a server offset apart.
All 22 rejected directives now fail on their own when their letter is added to
the set. The seven outside the set that do agree are left unpinned on purpose,
so widening stays possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(reports): apply d3TimeFormat in table post-processing [superset]
Posted by "codeant-ai-for-open-source[bot] (via GitHub)" <gi...@apache.org>.
codeant-ai-for-open-source[bot] commented on PR #45410:
URL: https://github.com/apache/superset/pull/45410#issuecomment-6230856959
## CodeAnt PR Risk: Medium Risk
- The PR needs attention before merging.
- If `d3TimeFormat` is set to `smart_date`, `strftime` may render that literal text in every datetime cell instead of formatting dates.
- The added test covers `%Y` but does not verify handling of adaptive date formats.
**Assessed commit:** `31824eb2e766`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(reports): apply d3TimeFormat in table post-processing [superset]
Posted by "SEPURI-SAI-KRISHNA (via GitHub)" <gi...@apache.org>.
SEPURI-SAI-KRISHNA commented on code in PR #45410:
URL: https://github.com/apache/superset/pull/45410#discussion_r4321234329
##########
tests/unit_tests/charts/test_client_processing.py:
##########
@@ -2488,6 +2488,60 @@ def test_table_applies_currency_format() -> None:
assert formatted["amount"].tolist() == ["$ 1,234.50"]
+def test_table_applies_time_format_alongside_number_format() -> None:
Review Comment:
Fair, the guard was unpinned. Added `test_table_time_format_keeps_nulls`:
```python
df = pd.DataFrame.from_dict(
{"ds": {0: pd.Timestamp("2024-01-15"), 1: pd.NaT}},
)
...
assert formatted["ds"].tolist()[0] == "2024"
assert pd.isna(formatted["ds"].tolist()[1])
```
Dropping the `pd.notna` check makes it fail on the second cell with the string `NaT`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(reports): apply d3TimeFormat in table post-processing [superset]
Posted by "sadpandajoe (via GitHub)" <gi...@apache.org>.
sadpandajoe commented on code in PR #45410:
URL: https://github.com/apache/superset/pull/45410#discussion_r4333433561
##########
superset/charts/client_processing.py:
##########
@@ -1237,6 +1302,27 @@ def label_of(name: str) -> str:
# left untouched, matching the browser.
for column in df.columns:
config = config_by_label.get(column) or {}
+
+ # Mirror the Table plugin's first formatter branch
+ # (``if (isTime || config.d3TimeFormat)`` in ``transformProps.ts``):
+ # an explicit per-column ``d3TimeFormat`` formats a temporal column and
+ # takes precedence over number formatting. Scoped to datetime columns
+ # with an explicit ``strftime``-style format; ``smart_date`` and any
+ # other alias is skipped so it is never rendered as literal text. The
+ # bare ``isTime`` fallback, ``tableTimestampFormat`` default,
+ # SMART_DATE granularity, and string temporal columns are left to a
+ # follow-up, since they need a full server-side d3-time-format port
+ # rather than ``strftime``.
+ time_format = config.get("d3TimeFormat")
+ if (
+ time_format
+ and time_format != SMART_DATE
+ and is_strftime_expressible(time_format)
+ and pd.api.types.is_datetime64_any_dtype(df[column])
Review Comment:
The description says a non-datetime column carrying `d3TimeFormat` is left untouched, but every new `%Y` test passes datetime input, so deleting this `is_datetime64_any_dtype` guard would keep them all green while a string or numeric column with that key would reach `.strftime` and raise `AttributeError` in the report. Could you add a `table()` case with a string (or integer) column configured with `d3TimeFormat="%Y"` asserting it comes back unchanged?
##########
superset/charts/client_processing.py:
##########
@@ -1304,6 +1390,34 @@ def _read_excel_for_client_processing(
return df.set_index(first_column)
+def payload_coltypes(
+ df: pd.DataFrame,
+ datasource: Optional[Union["BaseDatasource", "Query"]],
+) -> list[GenericDataType]:
+ """The payload's column types, reporting a text-rendered column as text.
+
+ ``extract_dataframe_dtypes`` lets the dataset's ``is_dttm`` flag decide the
+ type, so a temporal column a post-processor has rendered as text is still
+ reported as ``TEMPORAL``. ``chart_data_to_dataframe`` casts every
+ ``TEMPORAL`` column back to ``datetime64[ms]`` to build the notification
+ table, which would re-parse those strings and lose the configured format,
+ and would read ``02/01/2024`` month-first while reading ``15/01/2024``
+ day-first.
+
+ Only a column whose values really are strings is reported differently. The
+ XLSX path hands these types to ``apply_column_types`` to get date cells out
+ of a frame whose dates are still dates, and that frame is untouched here.
+ """
+ coltypes = extract_dataframe_dtypes(df, datasource)
+ return [
+ GenericDataType.STRING
+ if coltype == GenericDataType.TEMPORAL
+ and pd.api.types.infer_dtype(df.iloc[:, index], skipna=True) == "string"
Review Comment:
A formatted temporal column that also contains a null never gets the `STRING` type: `infer_dtype(["02/01/2024", pd.NaT], skipna=True)` returns `"mixed"` (pandas 2.3.3), not `"string"`, because `format_time_column` leaves the `NaT` in place. The column stays `TEMPORAL`, and `chart_data_to_dataframe` then casts it back to `datetime64[ms]`, reading `02/01/2024` month-first as 2024-02-01 (I ran that cast). So with `%d/%m/%Y` on `[2024-01-02, null]` the report shows a different date, and any null in the column defeats the fix described above. Could the check look only at the non-null values, or could the formatted columns be tracked explicitly, with a test that has a `NaT` alongside the string?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(reports): apply d3TimeFormat in table post-processing [superset]
Posted by "netlify[bot] (via GitHub)" <gi...@apache.org>.
netlify[bot] commented on PR #45410:
URL: https://github.com/apache/superset/pull/45410#issuecomment-6236649347
### <span aria-hidden="true">โ
</span> Deploy Preview for *superset-docs-preview* ready!
| Name | Link |
|:-:|------------------------|
|<span aria-hidden="true">๐จ</span> Latest commit | 983ccd97edda3a90d29f9538c0c237bf35f22bb5 |
|<span aria-hidden="true">๐</span> Latest deploy log | https://app.netlify.com/projects/superset-docs-preview/deploys/6ad3a1d33f9c5588a8f1992c |
|<span aria-hidden="true">๐</span> Deploy Preview | [https://deploy-preview-45410--superset-docs-preview.netlify.app](https://deploy-preview-45410--superset-docs-preview.netlify.app) |
|<span aria-hidden="true">๐ฑ</span> Preview on mobile | <details><summary> Toggle QR Code... </summary><br /><br /><br /><br />_Use your smartphone camera to open QR code link._</details> |
|<span aria-hidden="true">๐ค</span> Make changes | [Run an agent on this branch](https://app.netlify.com/projects/superset-docs-preview/agent-runs#DcvBDYMwDADAXfwmqmgEStigY5jYaSMRHBk_qBC7k9997oKmUpvBAjDAqrinX3cu58tw3dgpN1Fz5buL8uHIW6mcRStaD8Rtk_-HepmRPI7kfY5pmkLAkMcY3wnuBw) |
---
<!-- [superset-docs-preview Preview](https://deploy-preview-45410--superset-docs-preview.netlify.app) -->
_To edit notification comments on pull requests, go to your [Netlify project configuration](https://app.netlify.com/projects/superset-docs-preview/configuration/notifications#deploy-notifications)._
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(reports): apply d3TimeFormat in table post-processing [superset]
Posted by "SEPURI-SAI-KRISHNA (via GitHub)" <gi...@apache.org>.
SEPURI-SAI-KRISHNA commented on PR #45410:
URL: https://github.com/apache/superset/pull/45410#issuecomment-6236616987
Good catch, this was a real defect. `smart_date` is the first entry in `D3_TIME_FORMAT_OPTIONS` and `transformProps.ts` has a dedicated branch for it, so a column config can legitimately carry it, and `strftime("smart_date")` returns the literal string for every row.
Fixed in the follow-up commit: the branch now only runs for a `strftime`-style format, so `smart_date` and any other alias falls through untouched instead of being written into the cells. Added `test_table_leaves_adaptive_time_format_alone` to cover it.
Rendering adaptive formatting server side still needs a real d3-time-format port, which is out of scope here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(reports): apply d3TimeFormat in table post-processing [superset]
Posted by "SEPURI-SAI-KRISHNA (via GitHub)" <gi...@apache.org>.
SEPURI-SAI-KRISHNA commented on code in PR #45410:
URL: https://github.com/apache/superset/pull/45410#discussion_r4402748534
##########
superset/charts/client_processing.py:
##########
@@ -1372,6 +1455,31 @@ def label_of(name: str) -> str:
# left untouched, matching the browser.
for column in df.columns:
config = config_by_label.get(column) or {}
+
+ # Mirror the Table plugin's first formatter branch
+ # (``if (isTime || config.d3TimeFormat)`` in ``transformProps.ts``):
+ # an explicit per-column ``d3TimeFormat`` formats a temporal column and
+ # takes precedence over number formatting. Scoped to a datetime column
+ # carrying a format every directive of which ``strftime`` reads the same
+ # way. A column of another dtype has no ``strftime`` to call, and the
+ # bare ``isTime`` fallback, the ``tableTimestampFormat`` default,
+ # adaptive granularity and string temporal columns are left to a
+ # follow-up: they need a full server-side d3-time-format port rather
+ # than ``strftime``.
+ # ``TimeFormatterRegistry.get`` trims the format before building the
+ # formatter while d3 itself does not, so ``utcFormat(" %Y ")`` returns
+ # ``" 2024 "`` and without the strip the report keeps padding the chart
+ # dropped. One that is only whitespace strips to empty and falls
+ # through unformatted, like one whose directives cannot be reproduced.
+ time_format = (config.get("d3TimeFormat") or "").strip()
Review Comment:
Confirmed. `config.get("d3TimeFormat", "").strip()` keeps all 314 green, and a
null raises `AttributeError: 'NoneType' object has no attribute 'strip'` right
there.
Added `test_table_leaves_a_null_time_format_alone`: a datetime `ds` carrying
`{"d3TimeFormat": None}` next to a `,.1f` metric, asserting the timestamp is
untouched and the metric still formats. It is the only failure under that swap.
`column_config` reaches here from `params`, which is a plain string field with
no schema over its contents, which is why `has_auto_currency_in_column_config`
and `security_manager` both isinstance-check it before reading.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(reports): apply d3TimeFormat in table post-processing [superset]
Posted by "codeant-ai-for-open-source[bot] (via GitHub)" <gi...@apache.org>.
codeant-ai-for-open-source[bot] commented on code in PR #45410:
URL: https://github.com/apache/superset/pull/45410#discussion_r4321075101
##########
superset/charts/client_processing.py:
##########
@@ -1237,6 +1250,20 @@ def label_of(name: str) -> str:
# left untouched, matching the browser.
for column in df.columns:
config = config_by_label.get(column) or {}
+
+ # Mirror the Table plugin's first formatter branch
+ # (``if (isTime || config.d3TimeFormat)`` in ``transformProps.ts``):
+ # an explicit per-column ``d3TimeFormat`` formats a temporal column and
+ # takes precedence over number formatting. Scoped to datetime columns
+ # with an explicit format; the bare ``isTime`` fallback,
+ # ``tableTimestampFormat`` default, SMART_DATE granularity, and string
+ # temporal columns are intentionally left to a follow-up, since they
+ # need a full server-side d3-time-format port rather than ``strftime``.
+ time_format = config.get("d3TimeFormat")
+ if time_format and pd.api.types.is_datetime64_any_dtype(df[column]):
+ format_time_column(df, column, time_format)
Review Comment:
โ
**CodeAnt verified this suggestion was addressed in subsequent commits and marked this thread resolved** as of `d21b2b3`.
Datetime formatting now skips `SMART_DATE` and other non-`strftime`-expressible aliases, so `smart_date` is not passed to `format_time_column` as a literal format string.
<sub>If that's not right, unresolve this thread and CodeAnt will leave it open.</sub>
<!-- codeant-auto-resolve-reply -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(reports): apply d3TimeFormat in table post-processing [superset]
Posted by "sadpandajoe (via GitHub)" <gi...@apache.org>.
sadpandajoe commented on code in PR #45410:
URL: https://github.com/apache/superset/pull/45410#discussion_r4322385141
##########
superset/charts/client_processing.py:
##########
@@ -73,6 +75,27 @@
# The Excel equivalent, applied as a cell format so the value stays numeric.
EXCEL_PERCENT_FORMAT = "0.0%"
+# "Adaptive formatting" in the time format controls, mirroring ``SMART_DATE_ID``
+# in ``superset-ui-core/time-format``. It is an alias the browser resolves to a
+# formatter that picks a representation per value, not a format string.
+SMART_DATE = "smart_date"
+
+# ``strftime`` copies a directive it does not recognize into its output
+# verbatim, so a d3-only one would reach the report as literal text: ``%L`` and
+# ``%q`` stay ``%L`` and ``%q`` rather than becoming milliseconds and a quarter.
+# These are the directives that carry the same meaning in d3-time-format and in
+# ``strftime``. Left out are the locale-composite ones (``%c``, ``%x``, ``%X``),
+# whose output follows the server's locale rather than the chart's, and the
+# platform-dependent or d3-only ones (``%e``, ``%g``, ``%L``, ``%q``, ``%Q``,
+# ``%s``, ``%Z``, ``%z``).
+SHARED_TIME_DIRECTIVES = frozenset("aAbBdfGHIjmMpSuUVwWyY%")
Review Comment:
`%a`, `%A`, `%b`, `%B` and `%p` are only the same in both implementations while the browser uses d3's English defaults. `D3_TIME_FORMAT` in `superset/config.py` can override `days`, `months`, `shortMonths` and `periods`, and the chart then uses those names while `strftime` uses the worker's `LC_TIME`. With French `months` configured on a C-locale worker, `2024-01-15` with `%B %Y` shows `janvier 2024` in the chart but `January 2024` in the report. Should these directives leave the column raw (as the locale-composite ones do), or should the configured names be applied?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(reports): apply d3TimeFormat in table post-processing [superset]
Posted by "SEPURI-SAI-KRISHNA (via GitHub)" <gi...@apache.org>.
SEPURI-SAI-KRISHNA commented on code in PR #45410:
URL: https://github.com/apache/superset/pull/45410#discussion_r4322298384
##########
superset/charts/client_processing.py:
##########
@@ -835,6 +852,26 @@ def union_currency_context(
}
+def is_strftime_expressible(d3_time_format: str) -> bool:
+ """Whether every directive in a d3 time format means the same to strftime."""
+ directives = re.findall(r"%(.)", d3_time_format)
Review Comment:
Confirmed, and rejected now. `TimeFormatterRegistry.get` takes the marker off before it builds anything:
```ts
const useLocalTime = targetFormat.startsWith(LOCAL_PREFIX);
const formatString = targetFormat.replace(LOCAL_PREFIX, '');
```
so `local!%Y-%m-%d` reached `strftime` with the marker still on it and gave `local!2024-04-02`.
```python
if LOCAL_TIME_PREFIX in d3_time_format:
# ``TimeFormatterRegistry.get`` strips the marker before building the
# formatter, so leaving it in would render it as literal text.
return False
```
Rejecting rather than stripping, for the reason behind the marker: it asks for the viewer's own time zone, and the server cannot reproduce that. Stripping it would format in UTC and quietly answer a different question, which is worse than leaving the column raw.
I used `in` rather than `startswith` to match the registry: only a prefix sets `useLocalTime`, but `replace` takes the first occurrence wherever it sits, so `%Y local!` loses it in the browser and would keep it here.
`local!%Y-%m-%d` is now one of the cases in `test_table_skips_a_time_format_strftime_cannot_express`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(reports): apply d3TimeFormat in table post-processing [superset]
Posted by "codeant-ai-for-open-source[bot] (via GitHub)" <gi...@apache.org>.
codeant-ai-for-open-source[bot] commented on code in PR #45410:
URL: https://github.com/apache/superset/pull/45410#discussion_r4322303230
##########
superset/charts/client_processing.py:
##########
@@ -1237,6 +1250,20 @@ def label_of(name: str) -> str:
# left untouched, matching the browser.
for column in df.columns:
config = config_by_label.get(column) or {}
+
+ # Mirror the Table plugin's first formatter branch
+ # (``if (isTime || config.d3TimeFormat)`` in ``transformProps.ts``):
+ # an explicit per-column ``d3TimeFormat`` formats a temporal column and
+ # takes precedence over number formatting. Scoped to datetime columns
+ # with an explicit format; the bare ``isTime`` fallback,
+ # ``tableTimestampFormat`` default, SMART_DATE granularity, and string
+ # temporal columns are intentionally left to a follow-up, since they
+ # need a full server-side d3-time-format port rather than ``strftime``.
+ time_format = config.get("d3TimeFormat")
+ if time_format and pd.api.types.is_datetime64_any_dtype(df[column]):
+ format_time_column(df, column, time_format)
Review Comment:
โ
**CodeAnt verified this suggestion was addressed in subsequent commits and marked this thread resolved** as of `dd634c3`.
The datetime-format branch now skips `SMART_DATE` and only calls `format_time_column` for formats accepted by `is_strftime_expressible`, so `smart_date` is not passed to `strftime`.
<sub>If that's not right, unresolve this thread and CodeAnt will leave it open.</sub>
<!-- codeant-auto-resolve-reply -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(reports): apply d3TimeFormat in table post-processing [superset]
Posted by "SEPURI-SAI-KRISHNA (via GitHub)" <gi...@apache.org>.
SEPURI-SAI-KRISHNA commented on code in PR #45410:
URL: https://github.com/apache/superset/pull/45410#discussion_r4325435817
##########
superset/charts/client_processing.py:
##########
@@ -73,6 +75,27 @@
# The Excel equivalent, applied as a cell format so the value stays numeric.
EXCEL_PERCENT_FORMAT = "0.0%"
+# "Adaptive formatting" in the time format controls, mirroring ``SMART_DATE_ID``
+# in ``superset-ui-core/time-format``. It is an alias the browser resolves to a
+# formatter that picks a representation per value, not a format string.
+SMART_DATE = "smart_date"
+
+# ``strftime`` copies a directive it does not recognize into its output
+# verbatim, so a d3-only one would reach the report as literal text: ``%L`` and
+# ``%q`` stay ``%L`` and ``%q`` rather than becoming milliseconds and a quarter.
+# These are the directives that carry the same meaning in d3-time-format and in
+# ``strftime``. Left out are the locale-composite ones (``%c``, ``%x``, ``%X``),
+# whose output follows the server's locale rather than the chart's, and the
+# platform-dependent or d3-only ones (``%e``, ``%g``, ``%L``, ``%q``, ``%Q``,
+# ``%s``, ``%Z``, ``%z``).
+SHARED_TIME_DIRECTIVES = frozenset("aAbBdfGHIjmMpSuUVwWyY%")
Review Comment:
You are right, and I had the wrong mental model of what those five directives
depend on. I ran the chart's own formatter to be sure, `timeFormatLocale` from
`d3-time-format` with the defaults and then with French names merged over them
the way `TimeFormatterRegistry.setD3Format` does, against
`2024-01-15 14:30 UTC`:
| format | d3 defaults | with `D3_TIME_FORMAT` names |
| --- | --- | --- |
| `%B %Y` | `January 2024` | `janvier 2024` |
| `%b %d` | `Jan 15` | `janv. 15` |
| `%A` | `Monday` | `lundi` |
| `%a` | `Mon` | `lun.` |
| `%I:%M %p` | `02:30 PM` | `02:30 p.m.` (with `periods`) |
| `%Y-%m-%d` | `2024-01-15` | `2024-01-15` |
`strftime` gives the left column on an English worker whatever is configured,
and the worker's `LC_TIME` on a localized one, so the two agree only by
coincidence. `D3TimeFormat` in `config.py` carries exactly the five tables
behind those directives: `days`, `shortDays`, `months`, `shortMonths` and
`periods`.
Took the first of your two options: the five are out of
`SHARED_TIME_DIRECTIVES`, so a format using one leaves the column raw, the same
answer `local!` and the composite directives get. Reasons for not applying the
configured names instead:
The worker can read `D3_TIME_FORMAT`, but only the operator's overrides. The
names for every key left unset live in `DEFAULT_D3_TIME_FORMAT` on the frontend,
so applying them means a second copy of those five lists in Python, and a report
that goes quietly wrong if the two ever drift. That is the same kind of mirror
this file already is, and a locale table is a worse thing to mirror than a
format string.
Leaving the column raw is also what the report did before this PR, so no column
gets worse, and none of the eight presets in `D3_TIME_FORMAT_OPTIONS` uses a
name directive, so the loss is limited to a hand-typed format.
`test_table_skips_a_time_format_naming_a_month_weekday_or_period` pins all five.
On the previous commit each one fails with the English name: `January 2024`,
`Jan 15`, `Monday`, `Mon`, `02:30 PM`.
Your comment also made me stop trusting how I built that set in the first
place, so I compared the two implementations directive by directive instead of
by name, running `utcFormat('%<c>')` and `Timestamp.strftime('%<c>')` over the
same instants. That turned up one more wrong entry, `%f`:
```
2024-01-15 14:30:00.123456 d3 "123000" strftime "123456"
```
d3 builds microseconds as `getUTCMilliseconds() * 1000`, so the browser's value
always ends in `000`, while a warehouse column with sub-millisecond precision
reaches `strftime` with the real digits. `%f` is out of the set too, and
`%H:%M:%S.%f` joins the parametrized skip test.
The same comparison is also why I did not catch your finding myself: on this
worker, English and with nothing configured, all five name directives match
`strftime` exactly. The mismatch only exists against a configuration, which is
the part I was not treating as an input.
One adjacent thing I found while checking this, not in this PR's scope but you
should know it is there: `superset/utils/number_format.py` has the same exposure
to `D3_FORMAT` and does not read it at all. With
`D3_FORMAT = {"decimal": ",", "thousands": "."}` the chart formats `1234.5` as
`1.234,50` under `,.2f` and the report formats it as `1,234.50`. Setting it
changes nothing server-side. Happy to open that separately if you want it
tracked.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(reports): apply d3TimeFormat in table post-processing [superset]
Posted by "codeant-ai-for-open-source[bot] (via GitHub)" <gi...@apache.org>.
codeant-ai-for-open-source[bot] commented on code in PR #45410:
URL: https://github.com/apache/superset/pull/45410#discussion_r4325444039
##########
superset/charts/client_processing.py:
##########
@@ -1237,6 +1250,20 @@ def label_of(name: str) -> str:
# left untouched, matching the browser.
for column in df.columns:
config = config_by_label.get(column) or {}
+
+ # Mirror the Table plugin's first formatter branch
+ # (``if (isTime || config.d3TimeFormat)`` in ``transformProps.ts``):
+ # an explicit per-column ``d3TimeFormat`` formats a temporal column and
+ # takes precedence over number formatting. Scoped to datetime columns
+ # with an explicit format; the bare ``isTime`` fallback,
+ # ``tableTimestampFormat`` default, SMART_DATE granularity, and string
+ # temporal columns are intentionally left to a follow-up, since they
+ # need a full server-side d3-time-format port rather than ``strftime``.
+ time_format = config.get("d3TimeFormat")
+ if time_format and pd.api.types.is_datetime64_any_dtype(df[column]):
+ format_time_column(df, column, time_format)
Review Comment:
โ
**CodeAnt verified this suggestion was addressed in subsequent commits and marked this thread resolved** as of `018470f`.
The datetime-format branch now skips `SMART_DATE` and formats only formats recognized as `strftime`-expressible, so `smart_date` is not passed to `format_time_column`.
<sub>If that's not right, unresolve this thread and CodeAnt will leave it open.</sub>
<!-- codeant-auto-resolve-reply -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(reports): apply d3TimeFormat in table post-processing [superset]
Posted by "SEPURI-SAI-KRISHNA (via GitHub)" <gi...@apache.org>.
SEPURI-SAI-KRISHNA commented on code in PR #45410:
URL: https://github.com/apache/superset/pull/45410#discussion_r4356562234
##########
superset/charts/client_processing.py:
##########
@@ -1304,6 +1391,44 @@ def _read_excel_for_client_processing(
return df.set_index(first_column)
+def payload_coltypes(
+ df: pd.DataFrame,
+ datasource: Optional[Union["BaseDatasource", "Query"]],
+) -> list[GenericDataType]:
+ """The payload's column types, reporting a text-rendered column as text.
+
+ ``extract_dataframe_dtypes`` lets the dataset's ``is_dttm`` flag decide the
+ type, so a temporal column a post-processor has rendered as text is still
+ reported as ``TEMPORAL``. ``chart_data_to_dataframe`` casts every
+ ``TEMPORAL`` column back to ``datetime64[ms]`` to build the notification
+ table, which would re-parse those strings and lose the configured format,
+ and would read ``02/01/2024`` month-first while reading ``15/01/2024``
+ day-first.
+
+ Only a column whose values really are strings is reported differently. The
+ XLSX path hands these types to ``apply_column_types`` to get date cells out
+ of a frame whose dates are still dates, and that frame is untouched here.
+
+ Nulls are dropped before the values are read. ``format_time_column`` leaves
+ them in place, and ``infer_dtype`` counts a ``NaT`` as a value of its own
+ type rather than one ``skipna`` drops, so it calls a formatted column
+ holding one ``"mixed"``. A column of nothing but nulls reads as neither
+ text nor mixed and keeps whatever type the dataset gives it.
+
+ The index is positional because ``extract_dataframe_dtypes`` walks the
+ frame the same way, so the two line up even where a label repeats.
+ """
+ coltypes = extract_dataframe_dtypes(df, datasource)
+ return [
+ GenericDataType.STRING
+ if coltype == GenericDataType.TEMPORAL
+ and pd.api.types.infer_dtype(df.iloc[:, index].dropna(), skipna=False)
Review Comment:
Confirmed. On a frame with two `ds` columns, one holding `"02/01/2024"` and one a real timestamp:
```
df.iloc[:, 0] -> infer_dtype "string"
df.iloc[:, 1] -> infer_dtype "datetime64"
df[df.columns[0]] -> DataFrame (1, 2), infer_dtype "string"
```
So the label form hands back both columns and types them off the first, and the real timestamp column loses `TEMPORAL`, which is what `chart_data_to_dataframe` reads to rebuild dates for the notification table.
Added `test_payload_coltypes_reads_repeated_labels_positionally`, calling `payload_coltypes` directly with two `ds` columns and asserting `[STRING, TEMPORAL]`. Switching the read to `df[df.columns[index]]` fails it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(reports): apply d3TimeFormat in table post-processing [superset]
Posted by "SEPURI-SAI-KRISHNA (via GitHub)" <gi...@apache.org>.
SEPURI-SAI-KRISHNA commented on code in PR #45410:
URL: https://github.com/apache/superset/pull/45410#discussion_r4385017259
##########
superset/charts/client_processing.py:
##########
@@ -835,6 +863,51 @@ def union_currency_context(
}
+def is_strftime_expressible(d3_time_format: str) -> bool:
+ """Whether ``strftime`` renders a d3 time format the way the browser does."""
+ if LOCAL_TIME_PREFIX in d3_time_format:
+ # ``TimeFormatterRegistry.get`` strips the marker before building the
+ # formatter, so leaving it in would render it as literal text.
+ return False
+ directives = re.findall(r"%(.)", d3_time_format)
+ if not directives:
+ # Nothing for ``strftime`` to substitute, so it hands the string back
+ # unchanged and every cell would read the format itself. This is also
+ # what rejects the aliases the time format controls offer, which the
+ # browser resolves to a formatter that picks a representation per value
+ # rather than to a format string: ``smart_date`` ("Adaptive
+ # formatting", ``SMART_DATE_ID`` in ``superset-ui-core/time-format``)
+ # among them.
+ return False
+ if "%" in re.sub(r"%.", "", d3_time_format):
+ # A ``%`` no directive consumed, as the trailing one in ``%Y%``. d3
+ # drops it (``2021``) and ``strftime`` copies it (``2021%``), so the
+ # two disagree even where every directive is shared. ``%%`` is not
+ # this case: both read it as one escaped ``%``.
+ return False
+ return all(directive in SHARED_TIME_DIRECTIVES for directive in directives)
+
+
+def format_time_column(df: pd.DataFrame, column: Any, d3_time_format: str) -> None:
+ """Format a temporal column in place with an explicit d3 time format.
+
+ The caller gates this on ``is_strftime_expressible``, so every directive in
+ ``d3_time_format`` means the same thing here as it does in the browser. Null
+ values are left as-is rather than rendered as the string ``"NaT"``.
Review Comment:
You are right, and the behaviour was never the problem. `pd.NaT.strftime("%Y")`
raises `ValueError: NaTType does not support strftime`, so without the
`pd.notna` guard one null fails the whole report rather than printing `"NaT"`.
The guard itself is already pinned: dropping it fails both
`test_table_time_format_keeps_nulls` and
`test_table_report_time_format_survives_a_null_in_the_column`. So this was the
wording alone, and the docstring and the test docstring now say what `strftime`
does with a `NaT`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org