You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@superset.apache.org by "seunggabi (via GitHub)" <gi...@apache.org> on 2026/09/21 12:12:41 UTC
[PR] feat(dataset): support epoch_us python_date_format [superset]
seunggabi opened a new pull request, #44500:
URL: https://github.com/apache/superset/pull/44500
### SUMMARY
Dataset columns can declare that an integer holds a Unix epoch via the *Datetime format* field, but only `epoch_s` and `epoch_ms` are understood. Microsecond epochs are just as common in modern warehouses — Debezium emits `MicroTimestamp` (µs) for `DATETIME(4-6)` sources under its default `adaptive*` precision modes, Avro `timestamp-micros`, Parquet `TIMESTAMP_MICROS` and Iceberg timestamps are all µs. For such a column the only options today are a calculated column (which turns the raw column comparison into an expression and loses pruning on the source column) or leaving the format empty, in which case the time range filter is rendered as a string literal and the warehouse rejects it:
```
TYPE_MISMATCH: Cannot apply operator: bigint <= varchar(26)
```
This PR adds `epoch_us` next to the existing formats, mirroring `epoch_ms` in every place it is handled:
- **Filter literals** (`ExploreMixin.dttm_sql_literal`): UTC seconds × 1 000 000.
- **Time grain expressions** (`BaseEngineSpec.get_timestamp_expr`): new `epoch_us_to_dttm()` classmethod, defaulting to `epoch_to_dttm()` applied to `({col}/1000000)` so every engine spec that implements `epoch_to_dttm` gets it for free; engines can override like they do for `epoch_ms_to_dttm`.
- **Result normalization** (`utils.core._process_datetime_column`, `_retry_temporal_join_values_at_wider_resolution`): pandas unit `us`.
- **Validation** (`datasets.schemas.validate_python_date_format`, `DatasetDAO.validate_python_date_format`): accept the value.
- **Dataset editor help text** mentions the new value.
The hardcoded `("epoch_s", "epoch_ms")` tuples scattered across these modules are replaced by a single `EPOCH_FORMATS` constant (`{"epoch_s": 1, "epoch_ms": 1_000, "epoch_us": 1_000_000}`) so the formats can't drift apart again. No behaviour change for `epoch_s` / `epoch_ms`.
Generated SQL for an Athena/Trino BIGINT µs column with a daily grain and a time range:
```sql
SELECT date_trunc('day', CAST(from_unixtime((created_at/1000000)) AS TIMESTAMP)) AS created_at, ...
FROM app_community_receipt
WHERE created_at >= 1789344000000000 AND created_at < 1789948800000000
```
### BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF
N/A (help text change only in the UI).
### TESTING INSTRUCTIONS
1. Create a physical dataset on a table with a BIGINT column storing microsecond epochs (e.g. a Debezium-fed Iceberg/Trino/Athena table, or `SELECT 1704067200000000 AS ts`).
2. Edit the column: check *Is temporal*, set *Datetime format* to `epoch_us`, save.
3. Build a time-series chart on that column with a time range filter and a daily grain — the query renders the filter as an integer literal and the grain via `from_unixtime(col/1000000)`; the x-axis shows the expected dates.
Unit tests added/extended:
- `tests/unit_tests/models/core_test.py` — `dttm_sql_literal` for `epoch_us`
- `tests/unit_tests/db_engine_specs/test_athena.py` — `get_timestamp_expr` for all three epoch formats, with and without grain
- `tests/unit_tests/utils/test_core.py` — `normalize_dttm_col` for microseconds
- `tests/unit_tests/datasets/schema_tests.py` — `epoch_us` accepted by the validator
### ADDITIONAL INFORMATION
- [ ] Has associated issue:
- [ ] Required feature flags:
- [x] Changes UI
- [ ] Includes DB Migration (follow approval process in [SIP-59](https://github.com/apache/superset/issues/13351))
- [ ] Migration is atomic, supports rollback & is backwards-compatible
- [ ] Confirm DB migration upgrade and downgrade tested
- [ ] Runtime estimates and downtime expectations provided
- [x] Introduces new feature or API
- [ ] Removes existing feature or API
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(dataset): support epoch_us python_date_format [superset]
Posted by "codeant-ai-for-open-source[bot] (via GitHub)" <gi...@apache.org>.
codeant-ai-for-open-source[bot] commented on code in PR #44500:
URL: https://github.com/apache/superset/pull/44500#discussion_r4062022655
##########
superset/connectors/sqla/models.py:
##########
@@ -1295,7 +1296,7 @@ def get_timestamp_expression( # noqa: C901
label = label or utils.DTTM_ALIAS
pdf = self.python_date_format
- is_epoch = pdf in ("epoch_s", "epoch_ms")
+ is_epoch = pdf in EPOCH_FORMATS
Review Comment:
**Suggestion:** A string column configured as `epoch_us` now enters epoch arithmetic, so PostgreSQL attempts numeric division on text and the generated query fails at runtime.
**Assessment:** 🟠 `Major` · 🔁 `Occurrence: Sometimes` · 🏷️ `Type error`
[](https://docs.codeant.ai/cli/resolve-pr-comments-skill) [](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=398c4d1bb13d4f6b9ad337cf967ec3f5&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=398c4d1bb13d4f6b9ad337cf967ec3f5&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset)
<details>
<summary><b>Prompt for AI Agent 🤖 </b></summary>
```mdx
This is a comment left during a code review.
**Path:** superset/connectors/sqla/models.py
**Line:** 1299:1299
**Comment:**
*Type Error: A string column configured as `epoch_us` now enters epoch arithmetic, so PostgreSQL attempts numeric division on text and the generated query fails at runtime.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F44500&comment_hash=b451128522c7357f5f949fdf6c9b9f08c58aa47b6849a5addcc0fd5b5ec3c60b&reaction=like'>👍</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F44500&comment_hash=b451128522c7357f5f949fdf6c9b9f08c58aa47b6849a5addcc0fd5b5ec3c60b&reaction=dislike'>👎</a>
##########
superset/db_engine_specs/base.py:
##########
@@ -1529,6 +1532,16 @@ def epoch_ms_to_dttm(cls) -> str:
"""
return cls.epoch_to_dttm().replace("{col}", "({col}/1000)")
+ @classmethod
+ def epoch_us_to_dttm(cls) -> str:
+ """
+ SQL expression that converts epoch (microseconds) to datetime that can be used
+ in a query.
+
+ :return: SQL Expression
+ """
+ return cls.epoch_to_dttm().replace("{col}", "({col}/1000000)")
Review Comment:
**Suggestion:** BigQuery inherits this implementation, producing `TIMESTAMP_SECONDS(col / 1000000)`, but BigQuery requires an integer argument and rejects the resulting floating-point expression.
**Assessment:** 🟠 `Major` · 🔁 `Occurrence: Sometimes` · 🏷️ `Api mismatch`
[](https://docs.codeant.ai/cli/resolve-pr-comments-skill) [](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=7af414a2f9924fe1adac23b9f031614f&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=7af414a2f9924fe1adac23b9f031614f&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset)
<details>
<summary><b>Prompt for AI Agent 🤖 </b></summary>
```mdx
This is a comment left during a code review.
**Path:** superset/db_engine_specs/base.py
**Line:** 1543:1543
**Comment:**
*Api Mismatch: BigQuery inherits this implementation, producing `TIMESTAMP_SECONDS(col / 1000000)`, but BigQuery requires an integer argument and rejects the resulting floating-point expression.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F44500&comment_hash=dc631bbf554dbb5716c80bfe18d78c0fe6939dab6282331d5e7aff73d1bd6a75&reaction=like'>👍</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F44500&comment_hash=dc631bbf554dbb5716c80bfe18d78c0fe6939dab6282331d5e7aff73d1bd6a75&reaction=dislike'>👎</a>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(dataset): support epoch_us python_date_format [superset]
Posted by "seunggabi (via GitHub)" <gi...@apache.org>.
seunggabi commented on code in PR #44500:
URL: https://github.com/apache/superset/pull/44500#discussion_r4067893175
##########
superset/db_engine_specs/base.py:
##########
@@ -1529,6 +1536,21 @@ def epoch_ms_to_dttm(cls) -> str:
"""
return cls.epoch_to_dttm().replace("{col}", "({col}/1000)")
+ @classmethod
+ def epoch_us_to_dttm(cls) -> str:
+ """
+ SQL expression that converts epoch (microseconds) to datetime that can be used
+ in a query.
+
+ The default routes through ``epoch_ms_to_dttm`` so engines that already
+ override the millisecond conversion keep their validated SQL; the result
+ has millisecond resolution. Engines with a native microsecond function
Review Comment:
Good catch, and thanks for verifying on SQLite. Reworded the docstring with your phrasing in fc33d58.
##########
tests/unit_tests/db_engine_specs/test_base.py:
##########
@@ -1652,3 +1652,79 @@ def test_base_spec_extended_aggregation_func_defaults_to_unsupported(
def test_base_spec_extended_aggregation_func_unknown_name_is_unsupported() -> None:
"""An aggregate name outside the known extended set is also just None."""
assert BaseEngineSpec.get_extended_aggregation_func("NOT_A_REAL_AGGREGATE") is None
+
+
+class _EpochSpec(BaseEngineSpec):
+ """Minimal spec implementing only ``epoch_to_dttm``, like most engines."""
+
+ engine = "epoch"
+ engine_name = "epoch"
+ _time_grain_expressions = {
+ None: "{col}",
+ "P1D": "DATE_TRUNC('day', {col})",
+ }
+
+ @classmethod
+ def epoch_to_dttm(cls) -> str:
+ return "from_unixtime({col})"
+
+
+@pytest.mark.parametrize(
+ "pdf,time_grain,expected",
+ [
+ ("epoch_s", None, "from_unixtime(ts)"),
+ ("epoch_ms", None, "from_unixtime((ts/1000))"),
+ ("epoch_us", None, "from_unixtime(((ts/1000)/1000))"),
+ (
+ "epoch_us",
+ "P1D",
+ "DATE_TRUNC('day', from_unixtime(((ts/1000)/1000)))",
+ ),
+ (None, None, "ts"),
+ ],
+)
+def test_get_timestamp_expr_epoch_formats(
+ pdf: str | None, time_grain: str | None, expected: str
+) -> None:
+ """
+ Every ``epoch_*`` python_date_format routes the raw column through the
+ matching ``epoch*_to_dttm`` template before the time grain is applied; the
+ default ``epoch_us_to_dttm`` reuses the millisecond template.
+ """
+ expr = _EpochSpec.get_timestamp_expr(column("ts"), pdf, time_grain)
+ assert str(expr.compile(compile_kwargs={"literal_binds": True})) == expected
+
+
+@pytest.mark.parametrize(
+ "spec_path,expected",
+ [
+ # engines relying on the default: their millisecond template is reused
+ ("athena.AthenaEngineSpec", "from_unixtime((({col}/1000)/1000))"),
+ ("crate.CrateEngineSpec", "({col}/1000)"),
+ ("druid.DruidEngineSpec", "MILLIS_TO_TIMESTAMP(({col}/1000))"),
+ ("couchbase.CouchbaseEngineSpec", "MILLIS_TO_STR(({col}/1000))"),
+ # engines with a native microsecond conversion
+ ("bigquery.BigQueryEngineSpec", "TIMESTAMP_MICROS({col})"),
+ ("snowflake.SnowflakeEngineSpec", "DATEADD(US, {col}, '1970-01-01')"),
+ ("kusto.KustoKqlEngineSpec", "unixtime_microseconds_todatetime({col})"),
+ (
+ "pinot.PinotEngineSpec",
+ "DATETIMECONVERT({col}, '1:MICROSECONDS:EPOCH', "
+ "'1:MICROSECONDS:EPOCH', '1:MICROSECONDS')",
+ ),
+ ],
+)
+def test_epoch_us_to_dttm(spec_path: str, expected: str) -> None:
+ """
+ ``epoch_us_to_dttm`` yields valid SQL for engines that override
+ ``epoch_ms_to_dttm`` (via the default) and for engines with a native
+ microsecond function (via their own override).
+ """
+ import importlib
Review Comment:
Agreed, there was no import-cost reason. Switched to direct class imports in fc33d58.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(dataset): support epoch_us python_date_format [superset]
Posted by "bito-code-review[bot] (via GitHub)" <gi...@apache.org>.
bito-code-review[bot] commented on PR #44500:
URL: https://github.com/apache/superset/pull/44500#issuecomment-5763111040
AI Code Review is in progress (usually takes 3 to 15 minutes unless it's a very large PR).
<details>
<summary>
Bito Usage Guide
</summary>
**Commands**
Type the following command in the pull request comment and save the comment.
- `/review` - Manually triggers an incremental AI Review.
- `/review full` - Manually triggers a full AI Review.
- `/pause` - Pauses automatic reviews on this pull request.
- `/resume` - Resumes automatic reviews.
- `/resolve` - Marks all Bito-posted review comments as resolved.
- `/abort` - Cancels all in-progress reviews.
Refer to the <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/available-commands" target="_blank" rel="noopener noreferrer">documentation</a> for additional commands.
**Configuration**
This repository uses `Superset` You can customize the agent settings <a href="https://alpha.bito.ai/home/ai-agents/code-review-agent" target="_blank" rel="noopener noreferrer">here</a> or contact your Bito workspace admin at evan@preset.io.
**Documentation & Help**
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/getting-started/install-run-using-bito-cloud/create-or-customize-an-agent-instance" target="_blank" rel="noopener noreferrer">Customize agent settings</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/implementing-custom-code-review-rules" target="_blank" rel="noopener noreferrer">Review rules</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent" target="_blank" rel="noopener noreferrer">General documentation</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/faqs" target="_blank" rel="noopener noreferrer">FAQ</a>
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(dataset): support epoch_us python_date_format [superset]
Posted by "netlify[bot] (via GitHub)" <gi...@apache.org>.
netlify[bot] commented on PR #44500:
URL: https://github.com/apache/superset/pull/44500#issuecomment-5770194254
### <span aria-hidden="true">✅</span> Deploy Preview for *superset-docs-preview* ready!
| Name | Link |
|:-:|------------------------|
|<span aria-hidden="true">🔨</span> Latest commit | 5dbc0166a8a43b33b1ad4dce1ccdb693647762bc |
|<span aria-hidden="true">🔍</span> Latest deploy log | https://app.netlify.com/projects/superset-docs-preview/deploys/6ab1e1df8d2d150008d74fe7 |
|<span aria-hidden="true">😎</span> Deploy Preview | [https://deploy-preview-44500--superset-docs-preview.netlify.app](https://deploy-preview-44500--superset-docs-preview.netlify.app) |
|<span aria-hidden="true">📱</span> Preview on mobile | <details><summary> Toggle QR Code... </summary><br /><br /><br /><br />_Use your smartphone camera to open QR code link._</details> |
|<span aria-hidden="true">🤖</span> Make changes | [Run an agent on this branch](https://app.netlify.com/projects/superset-docs-preview/agent-runs#q1YqKMrPLShRslJS0lFKKkrMS84AstNSE0v0UwvykzN0S4uBEimpBTn5lZ4pQCmzxCTDVMOUNIsUoxRDUwMDA4sUc5O0VHOlWgA) |
---
<!-- [superset-docs-preview Preview](https://deploy-preview-44500--superset-docs-preview.netlify.app) -->
_To edit notification comments on pull requests, go to your [Netlify project configuration](https://app.netlify.com/projects/superset-docs-preview/configuration/notifications#deploy-notifications)._
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(dataset): support epoch_us python_date_format [superset]
Posted by "rusackas (via GitHub)" <gi...@apache.org>.
rusackas merged PR #44500:
URL: https://github.com/apache/superset/pull/44500
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(dataset): support epoch_us python_date_format [superset]
Posted by "seunggabi (via GitHub)" <gi...@apache.org>.
seunggabi commented on code in PR #44500:
URL: https://github.com/apache/superset/pull/44500#discussion_r4067892530
##########
superset/utils/core.py:
##########
@@ -2046,7 +2047,7 @@ def _process_datetime_column(
col: DateColumn,
) -> None:
"""Process a single datetime column with format detection."""
- if col.timestamp_format in ("epoch_s", "epoch_ms"):
+ if col.timestamp_format in EPOCH_FORMATS:
Review Comment:
The pandas unit is already derived from the format on the lines right below this check: `unit = col.timestamp_format.replace("epoch_", "")`, so `epoch_us` resolves to `unit="us"`. That line was on `master` before this PR; only the membership test changed. `test_normalize_dttm_col_epoch_microseconds` in `tests/unit_tests/utils/test_core.py` asserts `1577836800000000` → `2020-01-01`.
##########
superset/connectors/sqla/models.py:
##########
@@ -1295,7 +1296,7 @@ def get_timestamp_expression( # noqa: C901
label = label or utils.DTTM_ALIAS
pdf = self.python_date_format
- is_epoch = pdf in ("epoch_s", "epoch_ms")
+ is_epoch = pdf in EPOCH_FORMATS
Review Comment:
This is the existing contract for `epoch_s` / `epoch_ms`: a column tagged with an epoch `python_date_format` is routed through epoch arithmetic regardless of its SQL type. `epoch_us` follows the same rule, so a text column mislabelled as `epoch_us` fails the same way a text column mislabelled as `epoch_ms` already does. That is a dataset configuration error rather than something introduced here.
##########
superset/db_engine_specs/base.py:
##########
@@ -1529,6 +1532,16 @@ def epoch_ms_to_dttm(cls) -> str:
"""
return cls.epoch_to_dttm().replace("{col}", "({col}/1000)")
+ @classmethod
+ def epoch_us_to_dttm(cls) -> str:
+ """
+ SQL expression that converts epoch (microseconds) to datetime that can be used
+ in a query.
+
+ :return: SQL Expression
+ """
+ return cls.epoch_to_dttm().replace("{col}", "({col}/1000000)")
Review Comment:
BigQuery does not inherit the default: `BigQueryEngineSpec.epoch_us_to_dttm` returns `TIMESTAMP_MICROS({col})` (see `superset/db_engine_specs/bigquery.py`), and `test_epoch_us_to_dttm` in `tests/unit_tests/db_engine_specs/test_base.py` asserts exactly that.
##########
superset/utils/core.py:
##########
@@ -2046,7 +2047,7 @@ def _process_datetime_column(
col: DateColumn,
) -> None:
"""Process a single datetime column with format detection."""
- if col.timestamp_format in ("epoch_s", "epoch_ms"):
+ if col.timestamp_format in EPOCH_FORMATS:
Review Comment:
Pre-existing behaviour shared by all epoch formats: the numeric branch is keyed on the pandas dtype, and non-numeric series go through `pd.Timestamp` for `epoch_s` and `epoch_ms` today as well. This PR keeps the three formats consistent; changing how string-typed epoch columns are handled would be a separate change.
##########
superset/models/helpers.py:
##########
@@ -4194,9 +4199,7 @@ def dttm_sql_literal(self, dttm: datetime, col: "TableColumn") -> str:
dttm_tz_aware = dttm_tz_aware.replace(tzinfo=timezone.utc)
seconds_since_epoch = int(dttm_tz_aware.timestamp())
- if tf == "epoch_s":
- return str(seconds_since_epoch)
- return str(seconds_since_epoch * 1000)
+ return str(seconds_since_epoch * EPOCH_FORMATS[tf])
Review Comment:
Intentional. `dttm_sql_literal` emits the filter literal in the same unit the column is stored in, so the time-range comparison is like-for-like: an `epoch_us` column holds microseconds, hence `seconds * 1_000_000`; `epoch_ms` still gets `* 1_000` and `epoch_s` `* 1`. Covered by `test_dttm_sql_literal` in `tests/unit_tests/models/core_test.py`.
##########
superset/db_engine_specs/base.py:
##########
@@ -1529,6 +1536,21 @@ def epoch_ms_to_dttm(cls) -> str:
"""
return cls.epoch_to_dttm().replace("{col}", "({col}/1000)")
+ @classmethod
+ def epoch_us_to_dttm(cls) -> str:
+ """
+ SQL expression that converts epoch (microseconds) to datetime that can be used
+ in a query.
+
+ The default routes through ``epoch_ms_to_dttm`` so engines that already
+ override the millisecond conversion keep their validated SQL; the result
+ has millisecond resolution. Engines with a native microsecond function
Review Comment:
Good catch, and thanks for verifying on SQLite. Reworded the docstring with your phrasing in 5dbc016.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(dataset): support epoch_us python_date_format [superset]
Posted by "bito-code-review[bot] (via GitHub)" <gi...@apache.org>.
bito-code-review[bot] commented on PR #44500:
URL: https://github.com/apache/superset/pull/44500#issuecomment-5760360885
<!-- Bito Reply -->
The flagged issue is correct. The current implementation of `get_timestamp_expr` in `superset/db_engine_specs/postgres.py` does not account for the `epoch_us` format when casting string columns to timestamps, leading to runtime errors when PostgreSQL attempts numeric division on text columns.
To resolve this, you should update the condition in `superset/db_engine_specs/postgres.py` to ensure that columns configured with `epoch_us` are handled correctly, similar to how `epoch_s` and `epoch_ms` are handled.
Would you like me to fetch all other comments on this PR to validate and implement fixes for them as well?
**superset/db_engine_specs/postgres.py**
```
if time_grain and isinstance(col_type, String) and pdf not in EPOCH_FORMATS:
expr = TimestampExpression(
expr.name.replace("{col}", "CAST({col} AS TIMESTAMP)"),
col,
)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(dataset): support epoch_us python_date_format [superset]
Posted by "gabotorresruiz (via GitHub)" <gi...@apache.org>.
gabotorresruiz commented on code in PR #44500:
URL: https://github.com/apache/superset/pull/44500#discussion_r4066561439
##########
superset/db_engine_specs/base.py:
##########
@@ -1529,6 +1536,21 @@ def epoch_ms_to_dttm(cls) -> str:
"""
return cls.epoch_to_dttm().replace("{col}", "({col}/1000)")
+ @classmethod
+ def epoch_us_to_dttm(cls) -> str:
+ """
+ SQL expression that converts epoch (microseconds) to datetime that can be used
+ in a query.
+
+ The default routes through ``epoch_ms_to_dttm`` so engines that already
+ override the millisecond conversion keep their validated SQL; the result
+ has millisecond resolution. Engines with a native microsecond function
Review Comment:
Just a small NIT on the docstring: "the result has millisecond resolution" holds for the engines that override `epoch_ms_to_dttm` (Druid, Crate, Couchbase, Ocient, Oracle, Drill, Parseable), but every other engine inherits the default `epoch_ms_to_dttm`, so the chain ends at `epoch_to_dttm()` fed `(({col}/1000)/1000)` and the resolution is seconds.
I confirmed it on this branch with a real SQLite dataset on `epoch_us`: the column renders as `datetime(((ts_us/1000)/1000), 'unixepoch')` and a stored `1672534800123456` comes back as `2023-01-01 01:00:00`, with the `123456` gone. `epoch_ms` truncates the same way there today, so nothing regresses, it is only the docstring that would set the wrong expectation for the next engine author.
Maybe something like "the result inherits whatever resolution that engine's `epoch_ms_to_dttm` has, which is seconds when the default is inherited".
##########
tests/unit_tests/db_engine_specs/test_base.py:
##########
@@ -1652,3 +1652,79 @@ def test_base_spec_extended_aggregation_func_defaults_to_unsupported(
def test_base_spec_extended_aggregation_func_unknown_name_is_unsupported() -> None:
"""An aggregate name outside the known extended set is also just None."""
assert BaseEngineSpec.get_extended_aggregation_func("NOT_A_REAL_AGGREGATE") is None
+
+
+class _EpochSpec(BaseEngineSpec):
+ """Minimal spec implementing only ``epoch_to_dttm``, like most engines."""
+
+ engine = "epoch"
+ engine_name = "epoch"
+ _time_grain_expressions = {
+ None: "{col}",
+ "P1D": "DATE_TRUNC('day', {col})",
+ }
+
+ @classmethod
+ def epoch_to_dttm(cls) -> str:
+ return "from_unixtime({col})"
+
+
+@pytest.mark.parametrize(
+ "pdf,time_grain,expected",
+ [
+ ("epoch_s", None, "from_unixtime(ts)"),
+ ("epoch_ms", None, "from_unixtime((ts/1000))"),
+ ("epoch_us", None, "from_unixtime(((ts/1000)/1000))"),
+ (
+ "epoch_us",
+ "P1D",
+ "DATE_TRUNC('day', from_unixtime(((ts/1000)/1000)))",
+ ),
+ (None, None, "ts"),
+ ],
+)
+def test_get_timestamp_expr_epoch_formats(
+ pdf: str | None, time_grain: str | None, expected: str
+) -> None:
+ """
+ Every ``epoch_*`` python_date_format routes the raw column through the
+ matching ``epoch*_to_dttm`` template before the time grain is applied; the
+ default ``epoch_us_to_dttm`` reuses the millisecond template.
+ """
+ expr = _EpochSpec.get_timestamp_expr(column("ts"), pdf, time_grain)
+ assert str(expr.compile(compile_kwargs={"literal_binds": True})) == expected
+
+
+@pytest.mark.parametrize(
+ "spec_path,expected",
+ [
+ # engines relying on the default: their millisecond template is reused
+ ("athena.AthenaEngineSpec", "from_unixtime((({col}/1000)/1000))"),
+ ("crate.CrateEngineSpec", "({col}/1000)"),
+ ("druid.DruidEngineSpec", "MILLIS_TO_TIMESTAMP(({col}/1000))"),
+ ("couchbase.CouchbaseEngineSpec", "MILLIS_TO_STR(({col}/1000))"),
+ # engines with a native microsecond conversion
+ ("bigquery.BigQueryEngineSpec", "TIMESTAMP_MICROS({col})"),
+ ("snowflake.SnowflakeEngineSpec", "DATEADD(US, {col}, '1970-01-01')"),
+ ("kusto.KustoKqlEngineSpec", "unixtime_microseconds_todatetime({col})"),
+ (
+ "pinot.PinotEngineSpec",
+ "DATETIMECONVERT({col}, '1:MICROSECONDS:EPOCH', "
+ "'1:MICROSECONDS:EPOCH', '1:MICROSECONDS')",
+ ),
+ ],
+)
+def test_epoch_us_to_dttm(spec_path: str, expected: str) -> None:
+ """
+ ``epoch_us_to_dttm`` yields valid SQL for engines that override
+ ``epoch_ms_to_dttm`` (via the default) and for engines with a native
+ microsecond function (via their own override).
+ """
+ import importlib
Review Comment:
Not a blocker, just a NIT: resolving the specs from dotted strings with `importlib` inside the test body turns a renamed class or a moved module into a runtime `AttributeError` / `ModuleNotFoundError` rather than an import failure at collection, and it is the only dynamic import in this file. Unless there was an import cost reason, the classes themselves read better:
```python
@pytest.mark.parametrize(
"spec,expected",
[
(AthenaEngineSpec, "from_unixtime((({col}/1000)/1000))"),
(CrateEngineSpec, "({col}/1000)"),
...
(BigQueryEngineSpec, "TIMESTAMP_MICROS({col})"),
],
)
def test_epoch_us_to_dttm(spec: type[BaseEngineSpec], expected: str) -> None:
assert spec.epoch_us_to_dttm() == expected
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(dataset): support epoch_us python_date_format [superset]
Posted by "codeant-ai-for-open-source[bot] (via GitHub)" <gi...@apache.org>.
codeant-ai-for-open-source[bot] commented on code in PR #44500:
URL: https://github.com/apache/superset/pull/44500#discussion_r4062024075
##########
superset/utils/core.py:
##########
@@ -2046,7 +2047,7 @@ def _process_datetime_column(
col: DateColumn,
) -> None:
"""Process a single datetime column with format detection."""
- if col.timestamp_format in ("epoch_s", "epoch_ms"):
+ if col.timestamp_format in EPOCH_FORMATS:
Review Comment:
**Suggestion:** String microsecond epochs enter the already-formatted path instead of being parsed with unit `us`, producing incorrect timestamps or leaving the original strings after conversion fails.
**Assessment:** 🟠 `Major` · 🔁 `Occurrence: Rarely` · 🏷️ `Type error`
[](https://docs.codeant.ai/cli/resolve-pr-comments-skill) [](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=f9b6418982b6478d8a7018bdd1d87980&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=f9b6418982b6478d8a7018bdd1d87980&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset)
<details>
<summary><b>Prompt for AI Agent 🤖 </b></summary>
```mdx
This is a comment left during a code review.
**Path:** superset/utils/core.py
**Line:** 2050:2050
**Comment:**
*Type Error: String microsecond epochs enter the already-formatted path instead of being parsed with unit `us`, producing incorrect timestamps or leaving the original strings after conversion fails.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F44500&comment_hash=c1d7999012cce34456157646ee36e2d031144f94a648570123d16924db585370&reaction=like'>👍</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F44500&comment_hash=c1d7999012cce34456157646ee36e2d031144f94a648570123d16924db585370&reaction=dislike'>👎</a>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(dataset): support epoch_us python_date_format [superset]
Posted by "seunggabi (via GitHub)" <gi...@apache.org>.
seunggabi commented on code in PR #44500:
URL: https://github.com/apache/superset/pull/44500#discussion_r4067893175
##########
superset/db_engine_specs/base.py:
##########
@@ -1529,6 +1536,21 @@ def epoch_ms_to_dttm(cls) -> str:
"""
return cls.epoch_to_dttm().replace("{col}", "({col}/1000)")
+ @classmethod
+ def epoch_us_to_dttm(cls) -> str:
+ """
+ SQL expression that converts epoch (microseconds) to datetime that can be used
+ in a query.
+
+ The default routes through ``epoch_ms_to_dttm`` so engines that already
+ override the millisecond conversion keep their validated SQL; the result
+ has millisecond resolution. Engines with a native microsecond function
Review Comment:
Good catch, and thanks for verifying on SQLite. Reworded the docstring with your phrasing in d671a0a.
##########
tests/unit_tests/db_engine_specs/test_base.py:
##########
@@ -1652,3 +1652,79 @@ def test_base_spec_extended_aggregation_func_defaults_to_unsupported(
def test_base_spec_extended_aggregation_func_unknown_name_is_unsupported() -> None:
"""An aggregate name outside the known extended set is also just None."""
assert BaseEngineSpec.get_extended_aggregation_func("NOT_A_REAL_AGGREGATE") is None
+
+
+class _EpochSpec(BaseEngineSpec):
+ """Minimal spec implementing only ``epoch_to_dttm``, like most engines."""
+
+ engine = "epoch"
+ engine_name = "epoch"
+ _time_grain_expressions = {
+ None: "{col}",
+ "P1D": "DATE_TRUNC('day', {col})",
+ }
+
+ @classmethod
+ def epoch_to_dttm(cls) -> str:
+ return "from_unixtime({col})"
+
+
+@pytest.mark.parametrize(
+ "pdf,time_grain,expected",
+ [
+ ("epoch_s", None, "from_unixtime(ts)"),
+ ("epoch_ms", None, "from_unixtime((ts/1000))"),
+ ("epoch_us", None, "from_unixtime(((ts/1000)/1000))"),
+ (
+ "epoch_us",
+ "P1D",
+ "DATE_TRUNC('day', from_unixtime(((ts/1000)/1000)))",
+ ),
+ (None, None, "ts"),
+ ],
+)
+def test_get_timestamp_expr_epoch_formats(
+ pdf: str | None, time_grain: str | None, expected: str
+) -> None:
+ """
+ Every ``epoch_*`` python_date_format routes the raw column through the
+ matching ``epoch*_to_dttm`` template before the time grain is applied; the
+ default ``epoch_us_to_dttm`` reuses the millisecond template.
+ """
+ expr = _EpochSpec.get_timestamp_expr(column("ts"), pdf, time_grain)
+ assert str(expr.compile(compile_kwargs={"literal_binds": True})) == expected
+
+
+@pytest.mark.parametrize(
+ "spec_path,expected",
+ [
+ # engines relying on the default: their millisecond template is reused
+ ("athena.AthenaEngineSpec", "from_unixtime((({col}/1000)/1000))"),
+ ("crate.CrateEngineSpec", "({col}/1000)"),
+ ("druid.DruidEngineSpec", "MILLIS_TO_TIMESTAMP(({col}/1000))"),
+ ("couchbase.CouchbaseEngineSpec", "MILLIS_TO_STR(({col}/1000))"),
+ # engines with a native microsecond conversion
+ ("bigquery.BigQueryEngineSpec", "TIMESTAMP_MICROS({col})"),
+ ("snowflake.SnowflakeEngineSpec", "DATEADD(US, {col}, '1970-01-01')"),
+ ("kusto.KustoKqlEngineSpec", "unixtime_microseconds_todatetime({col})"),
+ (
+ "pinot.PinotEngineSpec",
+ "DATETIMECONVERT({col}, '1:MICROSECONDS:EPOCH', "
+ "'1:MICROSECONDS:EPOCH', '1:MICROSECONDS')",
+ ),
+ ],
+)
+def test_epoch_us_to_dttm(spec_path: str, expected: str) -> None:
+ """
+ ``epoch_us_to_dttm`` yields valid SQL for engines that override
+ ``epoch_ms_to_dttm`` (via the default) and for engines with a native
+ microsecond function (via their own override).
+ """
+ import importlib
Review Comment:
Agreed, there was no import-cost reason. Switched to direct class imports in d671a0a.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(dataset): support epoch_us python_date_format [superset]
Posted by "bito-code-review[bot] (via GitHub)" <gi...@apache.org>.
bito-code-review[bot] commented on PR #44500:
URL: https://github.com/apache/superset/pull/44500#issuecomment-5771480357
AI Code Review is in progress (usually takes 3 to 15 minutes unless it's a very large PR).
<details>
<summary>
Bito Usage Guide
</summary>
**Commands**
Type the following command in the pull request comment and save the comment.
- `/review` - Manually triggers an incremental AI Review.
- `/review full` - Manually triggers a full AI Review.
- `/pause` - Pauses automatic reviews on this pull request.
- `/resume` - Resumes automatic reviews.
- `/resolve` - Marks all Bito-posted review comments as resolved.
- `/abort` - Cancels all in-progress reviews.
Refer to the <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/available-commands" target="_blank" rel="noopener noreferrer">documentation</a> for additional commands.
**Configuration**
This repository uses `Superset` You can customize the agent settings <a href="https://alpha.bito.ai/home/ai-agents/code-review-agent" target="_blank" rel="noopener noreferrer">here</a> or contact your Bito workspace admin at evan@preset.io.
**Documentation & Help**
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/getting-started/install-run-using-bito-cloud/create-or-customize-an-agent-instance" target="_blank" rel="noopener noreferrer">Customize agent settings</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/implementing-custom-code-review-rules" target="_blank" rel="noopener noreferrer">Review rules</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent" target="_blank" rel="noopener noreferrer">General documentation</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/faqs" target="_blank" rel="noopener noreferrer">FAQ</a>
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(dataset): support epoch_us python_date_format [superset]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #44500:
URL: https://github.com/apache/superset/pull/44500#discussion_r4064811962
##########
superset/utils/core.py:
##########
@@ -2046,7 +2047,7 @@ def _process_datetime_column(
col: DateColumn,
) -> None:
"""Process a single datetime column with format detection."""
- if col.timestamp_format in ("epoch_s", "epoch_ms"):
+ if col.timestamp_format in EPOCH_FORMATS:
Review Comment:
Adding `epoch_us` to this branch is not sufficient for result normalization: the conversion logic below still selects seconds for every format other than `epoch_ms`. Consequently, values such as `1577836800000000` are interpreted as seconds instead of microseconds and will overflow or produce invalid datetimes. Derive the pandas unit from the epoch format, including `us`, before calling `pd.to_datetime`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(dataset): support epoch_us python_date_format [superset]
Posted by "bito-code-review[bot] (via GitHub)" <gi...@apache.org>.
bito-code-review[bot] commented on code in PR #44500:
URL: https://github.com/apache/superset/pull/44500#discussion_r4063828235
##########
superset/models/helpers.py:
##########
@@ -4194,9 +4199,7 @@ def dttm_sql_literal(self, dttm: datetime, col: "TableColumn") -> str:
dttm_tz_aware = dttm_tz_aware.replace(tzinfo=timezone.utc)
seconds_since_epoch = int(dttm_tz_aware.timestamp())
- if tf == "epoch_s":
- return str(seconds_since_epoch)
- return str(seconds_since_epoch * 1000)
+ return str(seconds_since_epoch * EPOCH_FORMATS[tf])
Review Comment:
<div>
<div id="suggestion">
<div id="issue"><b>Wrong epoch unit for epoch_us</b></div>
<div id="fix">
`EPOCH_FORMATS` now includes `epoch_us`, so `dttm_sql_literal` renders `seconds_since_epoch * 1_000_000` for it — microseconds, while `epoch_ms` consumers expect milliseconds. `validate_python_date_format` accepts `epoch_us` (superset/datasets/schemas.py), so a dataset column can reach this line. Guard or reject `epoch_us` here; previously only s/ms were supported.
</div>
</div>
<small><i>Code Review Run #300647</i></small>
</div>
---
Should Bito avoid suggestions like this for future reviews? (<a href=https://alpha.bito.ai/home/ai-agents/review-rules>Manage Rules</a>)
- [ ] Yes, avoid them
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(dataset): support epoch_us python_date_format [superset]
Posted by "seunggabi (via GitHub)" <gi...@apache.org>.
seunggabi commented on PR #44500:
URL: https://github.com/apache/superset/pull/44500#issuecomment-5775069335
Note on the red `lint-frontend` / `sharded-jest-tests (1)`: both fail on `master` as well (`getClientErrorObject` is used in `src/dashboard/actions/dashboardState.ts` without an import since #44238), and `babel-extract` was red on `master` from a stale `messages.pot`. #44513 fixes both; every other job on this PR is green. I will rebase once it lands.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(dataset): support epoch_us python_date_format [superset]
Posted by "seunggabi (via GitHub)" <gi...@apache.org>.
seunggabi commented on code in PR #44500:
URL: https://github.com/apache/superset/pull/44500#discussion_r4067893175
##########
superset/db_engine_specs/base.py:
##########
@@ -1529,6 +1536,21 @@ def epoch_ms_to_dttm(cls) -> str:
"""
return cls.epoch_to_dttm().replace("{col}", "({col}/1000)")
+ @classmethod
+ def epoch_us_to_dttm(cls) -> str:
+ """
+ SQL expression that converts epoch (microseconds) to datetime that can be used
+ in a query.
+
+ The default routes through ``epoch_ms_to_dttm`` so engines that already
+ override the millisecond conversion keep their validated SQL; the result
+ has millisecond resolution. Engines with a native microsecond function
Review Comment:
Good catch, and thanks for verifying on SQLite. Reworded the docstring with your phrasing in 4ba5eb3.
##########
tests/unit_tests/db_engine_specs/test_base.py:
##########
@@ -1652,3 +1652,79 @@ def test_base_spec_extended_aggregation_func_defaults_to_unsupported(
def test_base_spec_extended_aggregation_func_unknown_name_is_unsupported() -> None:
"""An aggregate name outside the known extended set is also just None."""
assert BaseEngineSpec.get_extended_aggregation_func("NOT_A_REAL_AGGREGATE") is None
+
+
+class _EpochSpec(BaseEngineSpec):
+ """Minimal spec implementing only ``epoch_to_dttm``, like most engines."""
+
+ engine = "epoch"
+ engine_name = "epoch"
+ _time_grain_expressions = {
+ None: "{col}",
+ "P1D": "DATE_TRUNC('day', {col})",
+ }
+
+ @classmethod
+ def epoch_to_dttm(cls) -> str:
+ return "from_unixtime({col})"
+
+
+@pytest.mark.parametrize(
+ "pdf,time_grain,expected",
+ [
+ ("epoch_s", None, "from_unixtime(ts)"),
+ ("epoch_ms", None, "from_unixtime((ts/1000))"),
+ ("epoch_us", None, "from_unixtime(((ts/1000)/1000))"),
+ (
+ "epoch_us",
+ "P1D",
+ "DATE_TRUNC('day', from_unixtime(((ts/1000)/1000)))",
+ ),
+ (None, None, "ts"),
+ ],
+)
+def test_get_timestamp_expr_epoch_formats(
+ pdf: str | None, time_grain: str | None, expected: str
+) -> None:
+ """
+ Every ``epoch_*`` python_date_format routes the raw column through the
+ matching ``epoch*_to_dttm`` template before the time grain is applied; the
+ default ``epoch_us_to_dttm`` reuses the millisecond template.
+ """
+ expr = _EpochSpec.get_timestamp_expr(column("ts"), pdf, time_grain)
+ assert str(expr.compile(compile_kwargs={"literal_binds": True})) == expected
+
+
+@pytest.mark.parametrize(
+ "spec_path,expected",
+ [
+ # engines relying on the default: their millisecond template is reused
+ ("athena.AthenaEngineSpec", "from_unixtime((({col}/1000)/1000))"),
+ ("crate.CrateEngineSpec", "({col}/1000)"),
+ ("druid.DruidEngineSpec", "MILLIS_TO_TIMESTAMP(({col}/1000))"),
+ ("couchbase.CouchbaseEngineSpec", "MILLIS_TO_STR(({col}/1000))"),
+ # engines with a native microsecond conversion
+ ("bigquery.BigQueryEngineSpec", "TIMESTAMP_MICROS({col})"),
+ ("snowflake.SnowflakeEngineSpec", "DATEADD(US, {col}, '1970-01-01')"),
+ ("kusto.KustoKqlEngineSpec", "unixtime_microseconds_todatetime({col})"),
+ (
+ "pinot.PinotEngineSpec",
+ "DATETIMECONVERT({col}, '1:MICROSECONDS:EPOCH', "
+ "'1:MICROSECONDS:EPOCH', '1:MICROSECONDS')",
+ ),
+ ],
+)
+def test_epoch_us_to_dttm(spec_path: str, expected: str) -> None:
+ """
+ ``epoch_us_to_dttm`` yields valid SQL for engines that override
+ ``epoch_ms_to_dttm`` (via the default) and for engines with a native
+ microsecond function (via their own override).
+ """
+ import importlib
Review Comment:
Agreed, there was no import-cost reason. Switched to direct class imports in 4ba5eb3.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(dataset): support epoch_us python_date_format [superset]
Posted by "bito-code-review[bot] (via GitHub)" <gi...@apache.org>.
bito-code-review[bot] commented on code in PR #44500:
URL: https://github.com/apache/superset/pull/44500#discussion_r4068775474
##########
tests/unit_tests/models/core_test.py:
##########
@@ -217,6 +217,12 @@ class OldDBEngineSpec(BaseEngineSpec):
Database(),
"1672536225000",
),
+ (
+ datetime(2023, 1, 1, 1, 23, 45, 600000),
+ TableColumn(python_date_format="epoch_us"),
+ Database(),
+ "1672536225000000",
Review Comment:
<div>
<div id="suggestion">
<div id="issue"><b>Epoch us truncation</b></div>
<div id="fix">
Expected epoch_us literal drops the 600000 µs component: datetime(2023,1,1,1,23,45,600000) UTC is 1672536225600000 µs, not 1672536225000000. The value comes from `dttm_sql_literal` doing `int(dttm.timestamp()) * EPOCH_FORMATS[tf]`, truncating to whole seconds before scaling. This test locks in the truncation instead of catching it.
</div>
</div>
<small><i>Code Review Run #947069</i></small>
</div>
---
Should Bito avoid suggestions like this for future reviews? (<a href=https://alpha.bito.ai/home/ai-agents/review-rules>Manage Rules</a>)
- [ ] Yes, avoid them
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(dataset): support epoch_us python_date_format [superset]
Posted by "bito-code-review[bot] (via GitHub)" <gi...@apache.org>.
bito-code-review[bot] commented on PR #44500:
URL: https://github.com/apache/superset/pull/44500#issuecomment-5789051618
AI Code Review is in progress (usually takes 3 to 15 minutes unless it's a very large PR).
<details>
<summary>
Bito Usage Guide
</summary>
**Commands**
Type the following command in the pull request comment and save the comment.
- `/review` - Manually triggers an incremental AI Review.
- `/review full` - Manually triggers a full AI Review.
- `/pause` - Pauses automatic reviews on this pull request.
- `/resume` - Resumes automatic reviews.
- `/resolve` - Marks all Bito-posted review comments as resolved.
- `/abort` - Cancels all in-progress reviews.
Refer to the <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/available-commands" target="_blank" rel="noopener noreferrer">documentation</a> for additional commands.
**Configuration**
This repository uses `Superset` You can customize the agent settings <a href="https://alpha.bito.ai/home/ai-agents/code-review-agent" target="_blank" rel="noopener noreferrer">here</a> or contact your Bito workspace admin at evan@preset.io.
**Documentation & Help**
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/getting-started/install-run-using-bito-cloud/create-or-customize-an-agent-instance" target="_blank" rel="noopener noreferrer">Customize agent settings</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/implementing-custom-code-review-rules" target="_blank" rel="noopener noreferrer">Review rules</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent" target="_blank" rel="noopener noreferrer">General documentation</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/faqs" target="_blank" rel="noopener noreferrer">FAQ</a>
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(dataset): support epoch_us python_date_format [superset]
Posted by "bito-code-review[bot] (via GitHub)" <gi...@apache.org>.
bito-code-review[bot] commented on PR #44500:
URL: https://github.com/apache/superset/pull/44500#issuecomment-5760288035
AI Code Review is in progress (usually takes 3 to 15 minutes unless it's a very large PR).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(dataset): support epoch_us python_date_format [superset]
Posted by "seunggabi (via GitHub)" <gi...@apache.org>.
seunggabi commented on code in PR #44500:
URL: https://github.com/apache/superset/pull/44500#discussion_r4067893261
##########
tests/unit_tests/db_engine_specs/test_base.py:
##########
@@ -1652,3 +1652,79 @@ def test_base_spec_extended_aggregation_func_defaults_to_unsupported(
def test_base_spec_extended_aggregation_func_unknown_name_is_unsupported() -> None:
"""An aggregate name outside the known extended set is also just None."""
assert BaseEngineSpec.get_extended_aggregation_func("NOT_A_REAL_AGGREGATE") is None
+
+
+class _EpochSpec(BaseEngineSpec):
+ """Minimal spec implementing only ``epoch_to_dttm``, like most engines."""
+
+ engine = "epoch"
+ engine_name = "epoch"
+ _time_grain_expressions = {
+ None: "{col}",
+ "P1D": "DATE_TRUNC('day', {col})",
+ }
+
+ @classmethod
+ def epoch_to_dttm(cls) -> str:
+ return "from_unixtime({col})"
+
+
+@pytest.mark.parametrize(
+ "pdf,time_grain,expected",
+ [
+ ("epoch_s", None, "from_unixtime(ts)"),
+ ("epoch_ms", None, "from_unixtime((ts/1000))"),
+ ("epoch_us", None, "from_unixtime(((ts/1000)/1000))"),
+ (
+ "epoch_us",
+ "P1D",
+ "DATE_TRUNC('day', from_unixtime(((ts/1000)/1000)))",
+ ),
+ (None, None, "ts"),
+ ],
+)
+def test_get_timestamp_expr_epoch_formats(
+ pdf: str | None, time_grain: str | None, expected: str
+) -> None:
+ """
+ Every ``epoch_*`` python_date_format routes the raw column through the
+ matching ``epoch*_to_dttm`` template before the time grain is applied; the
+ default ``epoch_us_to_dttm`` reuses the millisecond template.
+ """
+ expr = _EpochSpec.get_timestamp_expr(column("ts"), pdf, time_grain)
+ assert str(expr.compile(compile_kwargs={"literal_binds": True})) == expected
+
+
+@pytest.mark.parametrize(
+ "spec_path,expected",
+ [
+ # engines relying on the default: their millisecond template is reused
+ ("athena.AthenaEngineSpec", "from_unixtime((({col}/1000)/1000))"),
+ ("crate.CrateEngineSpec", "({col}/1000)"),
+ ("druid.DruidEngineSpec", "MILLIS_TO_TIMESTAMP(({col}/1000))"),
+ ("couchbase.CouchbaseEngineSpec", "MILLIS_TO_STR(({col}/1000))"),
+ # engines with a native microsecond conversion
+ ("bigquery.BigQueryEngineSpec", "TIMESTAMP_MICROS({col})"),
+ ("snowflake.SnowflakeEngineSpec", "DATEADD(US, {col}, '1970-01-01')"),
+ ("kusto.KustoKqlEngineSpec", "unixtime_microseconds_todatetime({col})"),
+ (
+ "pinot.PinotEngineSpec",
+ "DATETIMECONVERT({col}, '1:MICROSECONDS:EPOCH', "
+ "'1:MICROSECONDS:EPOCH', '1:MICROSECONDS')",
+ ),
+ ],
+)
+def test_epoch_us_to_dttm(spec_path: str, expected: str) -> None:
+ """
+ ``epoch_us_to_dttm`` yields valid SQL for engines that override
+ ``epoch_ms_to_dttm`` (via the default) and for engines with a native
+ microsecond function (via their own override).
+ """
+ import importlib
Review Comment:
Agreed, there was no import-cost reason. Switched to direct class imports in 5dbc016.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(dataset): support epoch_us python_date_format [superset]
Posted by "bito-code-review[bot] (via GitHub)" <gi...@apache.org>.
bito-code-review[bot] commented on code in PR #44500:
URL: https://github.com/apache/superset/pull/44500#discussion_r4067893757
##########
superset/models/helpers.py:
##########
@@ -4194,9 +4199,7 @@ def dttm_sql_literal(self, dttm: datetime, col: "TableColumn") -> str:
dttm_tz_aware = dttm_tz_aware.replace(tzinfo=timezone.utc)
seconds_since_epoch = int(dttm_tz_aware.timestamp())
- if tf == "epoch_s":
- return str(seconds_since_epoch)
- return str(seconds_since_epoch * 1000)
+ return str(seconds_since_epoch * EPOCH_FORMATS[tf])
Review Comment:
<!-- Bito Reply -->
The suggestion is appropriate because it addresses a potential mismatch in epoch units. By introducing a guard or rejecting `epoch_us` in `dttm_sql_literal`, you ensure that the function does not incorrectly apply a multiplier intended for milliseconds to a microsecond-based column, which would lead to incorrect data representation.
**superset/models/helpers.py**
```
seconds_since_epoch = int(dttm_tz_aware.timestamp())
return str(seconds_since_epoch * EPOCH_FORMATS[tf])
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(dataset): support epoch_us python_date_format [superset]
Posted by "codeant-ai-for-open-source[bot] (via GitHub)" <gi...@apache.org>.
codeant-ai-for-open-source[bot] commented on code in PR #44500:
URL: https://github.com/apache/superset/pull/44500#discussion_r4067894402
##########
superset/connectors/sqla/models.py:
##########
@@ -1295,7 +1296,7 @@ def get_timestamp_expression( # noqa: C901
label = label or utils.DTTM_ALIAS
pdf = self.python_date_format
- is_epoch = pdf in ("epoch_s", "epoch_ms")
+ is_epoch = pdf in EPOCH_FORMATS
Review Comment:
✅ **Customized review instruction saved!**
**Instruction:**
> Do not flag epoch arithmetic based on the configured epoch date format as a type error; SQL type mismatches are dataset configuration errors under the existing contract.
**Applied to:**
- `superset/connectors/sqla/models.py`
---
💡 *To manage or update this instruction, visit: [CodeAnt AI Settings](https://app.codeant.ai/org/settings/learnings)*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(dataset): support epoch_us python_date_format [superset]
Posted by "codeant-ai-for-open-source[bot] (via GitHub)" <gi...@apache.org>.
codeant-ai-for-open-source[bot] commented on code in PR #44500:
URL: https://github.com/apache/superset/pull/44500#discussion_r4067896400
##########
superset/utils/core.py:
##########
@@ -2046,7 +2047,7 @@ def _process_datetime_column(
col: DateColumn,
) -> None:
"""Process a single datetime column with format detection."""
- if col.timestamp_format in ("epoch_s", "epoch_ms"):
+ if col.timestamp_format in EPOCH_FORMATS:
Review Comment:
✅ **Customized review instruction saved!**
**Instruction:**
> Do not flag string-typed epoch handling in _process_datetime_column; preserve the existing pandas-dtype-based behavior consistently across epoch formats.
**Applied to:**
- `superset/utils/core.py`
---
💡 *To manage or update this instruction, visit: [CodeAnt AI Settings](https://app.codeant.ai/org/settings/learnings)*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(dataset): support epoch_us python_date_format [superset]
Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #44500:
URL: https://github.com/apache/superset/pull/44500#issuecomment-5763031764
## [Codecov](https://app.codecov.io/gh/apache/superset/pull/44500?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `48.48485%` with `17 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 57.32%. Comparing base ([`fc01645`](https://app.codecov.io/gh/apache/superset/commit/fc016454fb0874b3a242253d9f6d9c04355c7a43?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`f912435`](https://app.codecov.io/gh/apache/superset/commit/f9124358ddcfdcabfca96b9e2662b86a8ccbf22d?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 2 commits behind head on master.
| [Files with missing lines](https://app.codecov.io/gh/apache/superset/pull/44500?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [superset/db\_engine\_specs/base.py](https://app.codecov.io/gh/apache/superset/pull/44500?src=pr&el=tree&filepath=superset%2Fdb_engine_specs%2Fbase.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c3VwZXJzZXQvZGJfZW5naW5lX3NwZWNzL2Jhc2UucHk=) | 42.85% | [4 Missing :warning: ](https://app.codecov.io/gh/apache/superset/pull/44500?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [superset/models/helpers.py](https://app.codecov.io/gh/apache/superset/pull/44500?src=pr&el=tree&filepath=superset%2Fmodels%2Fhelpers.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c3VwZXJzZXQvbW9kZWxzL2hlbHBlcnMucHk=) | 0.00% | [4 Missing :warning: ](https://app.codecov.io/gh/apache/superset/pull/44500?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [superset/connectors/sqla/models.py](https://app.codecov.io/gh/apache/superset/pull/44500?src=pr&el=tree&filepath=superset%2Fconnectors%2Fsqla%2Fmodels.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c3VwZXJzZXQvY29ubmVjdG9ycy9zcWxhL21vZGVscy5weQ==) | 50.00% | [1 Missing :warning: ](https://app.codecov.io/gh/apache/superset/pull/44500?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [superset/daos/dataset.py](https://app.codecov.io/gh/apache/superset/pull/44500?src=pr&el=tree&filepath=superset%2Fdaos%2Fdataset.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c3VwZXJzZXQvZGFvcy9kYXRhc2V0LnB5) | 50.00% | [1 Missing :warning: ](https://app.codecov.io/gh/apache/superset/pull/44500?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [superset/datasets/schemas.py](https://app.codecov.io/gh/apache/superset/pull/44500?src=pr&el=tree&filepath=superset%2Fdatasets%2Fschemas.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c3VwZXJzZXQvZGF0YXNldHMvc2NoZW1hcy5weQ==) | 50.00% | [1 Missing :warning: ](https://app.codecov.io/gh/apache/superset/pull/44500?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [superset/db\_engine\_specs/bigquery.py](https://app.codecov.io/gh/apache/superset/pull/44500?src=pr&el=tree&filepath=superset%2Fdb_engine_specs%2Fbigquery.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c3VwZXJzZXQvZGJfZW5naW5lX3NwZWNzL2JpZ3F1ZXJ5LnB5) | 66.66% | [1 Missing :warning: ](https://app.codecov.io/gh/apache/superset/pull/44500?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [superset/db\_engine\_specs/kusto.py](https://app.codecov.io/gh/apache/superset/pull/44500?src=pr&el=tree&filepath=superset%2Fdb_engine_specs%2Fkusto.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c3VwZXJzZXQvZGJfZW5naW5lX3NwZWNzL2t1c3RvLnB5) | 66.66% | [1 Missing :warning: ](https://app.codecov.io/gh/apache/superset/pull/44500?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [superset/db\_engine\_specs/pinot.py](https://app.codecov.io/gh/apache/superset/pull/44500?src=pr&el=tree&filepath=superset%2Fdb_engine_specs%2Fpinot.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c3VwZXJzZXQvZGJfZW5naW5lX3NwZWNzL3Bpbm90LnB5) | 66.66% | [1 Missing :warning: ](https://app.codecov.io/gh/apache/superset/pull/44500?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [superset/db\_engine\_specs/postgres.py](https://app.codecov.io/gh/apache/superset/pull/44500?src=pr&el=tree&filepath=superset%2Fdb_engine_specs%2Fpostgres.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c3VwZXJzZXQvZGJfZW5naW5lX3NwZWNzL3Bvc3RncmVzLnB5) | 50.00% | [1 Missing :warning: ](https://app.codecov.io/gh/apache/superset/pull/44500?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [superset/db\_engine\_specs/snowflake.py](https://app.codecov.io/gh/apache/superset/pull/44500?src=pr&el=tree&filepath=superset%2Fdb_engine_specs%2Fsnowflake.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c3VwZXJzZXQvZGJfZW5naW5lX3NwZWNzL3Nub3dmbGFrZS5weQ==) | 66.66% | [1 Missing :warning: ](https://app.codecov.io/gh/apache/superset/pull/44500?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| ... and [1 more](https://app.codecov.io/gh/apache/superset/pull/44500?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | |
> :exclamation: There is a different number of reports uploaded between BASE (fc01645) and HEAD (f912435). Click for more details.
>
> <details><summary>HEAD has 59 uploads less than BASE</summary>
>
>| Flag | BASE (fc01645) | HEAD (f912435) |
>|------|------|------|
>|python|52|1|
>|presto|9|1|
></details>
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #44500 +/- ##
===========================================
- Coverage 80.65% 57.32% -23.34%
===========================================
Files 2942 2942
Lines 175691 175707 +16
Branches 40788 40786 -2
===========================================
- Hits 141704 100718 -40986
- Misses 31325 74029 +42704
+ Partials 2662 960 -1702
```
| [Flag](https://app.codecov.io/gh/apache/superset/pull/44500/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [hive](https://app.codecov.io/gh/apache/superset/pull/44500/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [mysql](https://app.codecov.io/gh/apache/superset/pull/44500/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [postgres](https://app.codecov.io/gh/apache/superset/pull/44500/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [presto](https://app.codecov.io/gh/apache/superset/pull/44500/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `39.04% <48.48%> (+<0.01%)` | :arrow_up: |
| [python](https://app.codecov.io/gh/apache/superset/pull/44500/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `39.04% <48.48%> (-45.92%)` | :arrow_down: |
| [sqlite](https://app.codecov.io/gh/apache/superset/pull/44500/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [unit](https://app.codecov.io/gh/apache/superset/pull/44500/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
</details>
[:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/superset/pull/44500?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(dataset): support epoch_us python_date_format [superset]
Posted by "bito-code-review[bot] (via GitHub)" <gi...@apache.org>.
bito-code-review[bot] commented on code in PR #44500:
URL: https://github.com/apache/superset/pull/44500#discussion_r4070451660
##########
tests/unit_tests/models/core_test.py:
##########
@@ -217,6 +217,12 @@ class OldDBEngineSpec(BaseEngineSpec):
Database(),
"1672536225000",
),
+ (
+ datetime(2023, 1, 1, 1, 23, 45, 600000),
+ TableColumn(python_date_format="epoch_us"),
+ Database(),
+ "1672536225000000",
Review Comment:
<!-- Bito Reply -->
The suggestion from the reviewer highlights that the current implementation truncates the microsecond component of the datetime object before scaling, which leads to a loss of precision. While the existing test pins this behavior to maintain consistency across formats, the reviewer is pointing out that this approach effectively locks in a loss of data rather than accurately representing the full timestamp. Whether to apply this suggestion depends on whether the project's requirement for consistency across `epoch_ms` and `epoch_us` outweighs the need for microsecond-level accuracy in the `epoch_us` format.
**tests/unit_tests/models/core_test.py**
```
+ (
+ datetime(2023, 1, 1, 1, 23, 45, 600000),
+ TableColumn(python_date_format="epoch_us"),
+ Database(),
+ "1672536225000000",
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(dataset): support epoch_us python_date_format [superset]
Posted by "seunggabi (via GitHub)" <gi...@apache.org>.
seunggabi commented on code in PR #44500:
URL: https://github.com/apache/superset/pull/44500#discussion_r4070450279
##########
tests/unit_tests/models/core_test.py:
##########
@@ -217,6 +217,12 @@ class OldDBEngineSpec(BaseEngineSpec):
Database(),
"1672536225000",
),
+ (
+ datetime(2023, 1, 1, 1, 23, 45, 600000),
+ TableColumn(python_date_format="epoch_us"),
+ Database(),
+ "1672536225000000",
Review Comment:
Intentional and pre-existing: `dttm_sql_literal` has always truncated to whole seconds before scaling for `epoch_ms` (`int(dttm.timestamp()) * 1000`), and `epoch_us` follows the same rule so the three formats stay consistent. Time-range boundaries are whole seconds in practice, and keeping `int()` on the seconds avoids the float precision loss that multiplying `timestamp()` directly would introduce (see the approving review above). The test pins that shared contract for all three formats.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org