You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@superset.apache.org by "Vitor-Avila (via GitHub)" <gi...@apache.org> on 2025/12/16 17:22:29 UTC
[PR] fix: Support datetime_format during import [superset]
Vitor-Avila opened a new pull request, #36679:
URL: https://github.com/apache/superset/pull/36679
### SUMMARY
PR https://github.com/apache/superset/pull/36150 added a new `datetime_format` column to `table_columns`. This new column was getting exported but wasn't accepted by the import schema, causing errors:
<img width="762" height="814" alt="image" src="https://github.com/user-attachments/assets/508ecac7-25a3-41fe-b78e-4b8746e89044" />
This PR fixes the issue.
I also moved some local imports from the test file to the top.
### BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF
Added screenshot of the error before. The PR fixes it.
### TESTING INSTRUCTIONS
Unit tests added. For manual testing:
1. Export a dataset.
2. Import it.
### ADDITIONAL INFORMATION
<!--- Check any relevant boxes with "x" -->
<!--- HINT: Include "Fixes #nnn" if you are fixing an existing issue -->
- [ ] Has associated issue:
- [ ] Required feature flags:
- [ ] Changes UI
- [ ] Includes DB Migration (follow approval process in [SIP-59](https://github.com/apache/superset/issues/13351))
- [ ] Migration is atomic, supports rollback & is backwards-compatible
- [ ] Confirm DB migration upgrade and downgrade tested
- [ ] Runtime estimates and downtime expectations provided
- [ ] Introduces new feature or API
- [ ] Removes existing feature or API
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix: Support datetime_format during import [superset]
Posted by "codeant-ai-for-open-source[bot] (via GitHub)" <gi...@apache.org>.
codeant-ai-for-open-source[bot] commented on PR #36679:
URL: https://github.com/apache/superset/pull/36679#issuecomment-3661626851
CodeAnt AI finished reviewing your PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix: Support datetime_format during import [superset]
Posted by "codeant-ai-for-open-source[bot] (via GitHub)" <gi...@apache.org>.
codeant-ai-for-open-source[bot] commented on PR #36679:
URL: https://github.com/apache/superset/pull/36679#issuecomment-3661626612
## Nitpicks 🔍
<table>
<tr><td>🔒 <strong>No security issues identified</strong></td></tr>
<tr><td>⚡ <strong>Recommended areas for review</strong><br><br>
- [ ] <a href='https://github.com/apache/superset/pull/36679/files#diff-b6a186993d7414cb29c7015ca427727a7191ad89ade1714cf1c659a38eae81ebR239-R260'><strong>Empty-string Normalization</strong></a><br>Exported datasets sometimes contain empty strings for optional fields. The `fix_extra` pre-load hook normalizes `extra`, but `datetime_format` (and `python_date_format`) may be empty strings and should be coerced to `None` during import to avoid surprising downstream logic.<br>
- [ ] <a href='https://github.com/apache/superset/pull/36679/files#diff-b6a186993d7414cb29c7015ca427727a7191ad89ade1714cf1c659a38eae81ebR259-R260'><strong>Missing Validation</strong></a><br>The new `datetime_format` field on the import column schema has no validators. The `DatasetColumnsPutSchema` uses `Length` and `validate_python_date_format`, so the import schema is now accepting values that would be rejected later or cause runtime parsing errors. Add the same validation to keep import behavior consistent and to reject invalid formats early.<br>
- [ ] <a href='https://github.com/apache/superset/pull/36679/files#diff-6661a283bfedcc486b1b2caacb933f10c9ad5441238d7bf5110df7cc1270c996R32-R39'><strong>Import-time side effects</strong></a><br>Several heavy application imports (e.g., `import_dataset`, `SqlaTable`, `Database`, security manager) were moved from local imports inside test functions to module-level imports. That can cause expensive import-time work or side effects during pytest collection (DB/model registration, app context requirements) and may make tests slower or flaky in different environments.<br>
- [ ] <a href='https://github.com/apache/superset/pull/36679/files#diff-6661a283bfedcc486b1b2caacb933f10c9ad5441238d7bf5110df7cc1270c996R46-R48'><strong>Fixture coupling</strong></a><br>The tests now import `dataset_config` from integration tests as `dataset_fixture`. These integration fixtures may introduce coupling to slower resources or assumptions about fixture shape. Confirm the fixture is self-contained for unit tests and not mutated across tests (deepcopy is used in some cases, but verify all usages).<br>
- [ ] <a href='https://github.com/apache/superset/pull/36679/files#diff-6661a283bfedcc486b1b2caacb933f10c9ad5441238d7bf5110df7cc1270c996R674-R702'><strong>Schema validation risk</strong></a><br>The new test `test_import_dataset_column_datetime_format` exercises that `datetime_format` is preserved after import. Ensure the import schema and code changes that accept `datetime_format` exist and perform decoding before calling `import_dataset`. If the schema change is missing, this test will fail. Also consider asserting the schema load step preserved the field to isolate failures.<br>
</td></tr>
</table>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix: Support datetime_format during import [superset]
Posted by "Vitor-Avila (via GitHub)" <gi...@apache.org>.
Vitor-Avila merged PR #36679:
URL: https://github.com/apache/superset/pull/36679
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix: Support datetime_format during import [superset]
Posted by "codeant-ai-for-open-source[bot] (via GitHub)" <gi...@apache.org>.
codeant-ai-for-open-source[bot] commented on PR #36679:
URL: https://github.com/apache/superset/pull/36679#issuecomment-3661612217
CodeAnt AI is reviewing your PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix: Support datetime_format during import [superset]
Posted by "bito-code-review[bot] (via GitHub)" <gi...@apache.org>.
bito-code-review[bot] commented on PR #36679:
URL: https://github.com/apache/superset/pull/36679#issuecomment-3661612562
AI Code Review is in progress (usually takes 3 to 15 minutes unless it's a very large PR).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix: Support datetime_format during import [superset]
Posted by "Vitor-Avila (via GitHub)" <gi...@apache.org>.
Vitor-Avila commented on code in PR #36679:
URL: https://github.com/apache/superset/pull/36679#discussion_r2624160634
##########
superset/datasets/schemas.py:
##########
@@ -257,6 +257,7 @@ def fix_extra(self, data: dict[str, Any], **kwargs: Any) -> dict[str, Any]:
expression = fields.String(allow_none=True)
description = fields.String(allow_none=True)
python_date_format = fields.String(allow_none=True)
+ datetime_format = fields.String(allow_none=True)
Review Comment:
This is technically correct, but it's the current pattern for import operations. We should have a follow up dedicated primarily on fixing this across all import schemas
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix: Support datetime_format during import [superset]
Posted by "codeant-ai-for-open-source[bot] (via GitHub)" <gi...@apache.org>.
codeant-ai-for-open-source[bot] commented on code in PR #36679:
URL: https://github.com/apache/superset/pull/36679#discussion_r2624135008
##########
superset/datasets/schemas.py:
##########
@@ -257,6 +257,7 @@ def fix_extra(self, data: dict[str, Any], **kwargs: Any) -> dict[str, Any]:
expression = fields.String(allow_none=True)
description = fields.String(allow_none=True)
python_date_format = fields.String(allow_none=True)
+ datetime_format = fields.String(allow_none=True)
Review Comment:
**Suggestion:** The import schema for columns now accepts any string for `datetime_format` without validating its length or format, which can lead to values longer than the 100-character database column or formats that later fail validation in the update API, causing runtime errors or inconsistent behavior between import and edit flows. [logic error]
**Severity Level:** Minor ⚠️
```suggestion
datetime_format = fields.String(
allow_none=True, validate=[Length(1, 100), validate_python_date_format]
)
```
<details>
<summary><b>Why it matters? ⭐ </b></summary>
The import schema currently allows any string for datetime_format while elsewhere (DatasetColumnsPutSchema) the field is validated with Length(1, 100) and validate_python_date_format. That mismatch can let overly long or invalid formats enter the system during import and later fail when the same dataset is edited or persisted, causing runtime errors or data inconsistency. Adding the same validators restores consistency and prevents invalid data from being imported. This fixes a real data-validation issue, not just style.
</details>
<details>
<summary><b>Prompt for AI Agent 🤖 </b></summary>
```mdx
This is a comment left during a code review.
**Path:** superset/datasets/schemas.py
**Line:** 260:260
**Comment:**
*Logic Error: The import schema for columns now accepts any string for `datetime_format` without validating its length or format, which can lead to values longer than the 100-character database column or formats that later fail validation in the update API, causing runtime errors or inconsistent behavior between import and edit flows.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
```
</details>
##########
tests/unit_tests/datasets/commands/importers/v1/import_test.py:
##########
@@ -702,19 +671,43 @@ def test_import_dataset_managed_externally(
assert sqla_table.external_url == "https://example.org/my_table"
+def test_import_dataset_column_datetime_format(
+ mocker: MockerFixture,
+ session: Session,
+) -> None:
+ """
+ Test importing a dataset with a column including a datetime format.
+ """
+ mocker.patch.object(security_manager, "can_access", return_value=True)
+
+ engine = db.session.get_bind()
+ SqlaTable.metadata.create_all(engine) # pylint: disable=no-member
+
+ database = Database(database_name="my_database", sqlalchemy_uri="sqlite://")
+ db.session.add(database)
+ db.session.flush()
+
+ config = copy.deepcopy(dataset_fixture)
+ for column in config["columns"]:
+ column["datetime_format"] = "%Y-%m-%d"
+
+ schema = ImportV1DatasetSchema()
+ dataset_config = schema.load(config)
+
+ dataset_config["database_id"] = database.id
+
+ sqla_table = import_dataset(dataset_config)
+ for column in sqla_table.columns:
+ assert column.datetime_format == "%Y-%m-%d"
Review Comment:
**Suggestion:** The new test checks `column.datetime_format`, but the SQLAlchemy model exposes the date format as `python_date_format`, so this assertion will raise an AttributeError or validate the wrong field instead of confirming that the imported `datetime_format` config is correctly mapped. [logic error]
**Severity Level:** Minor ⚠️
```suggestion
assert column.python_date_format == "%Y-%m-%d"
```
<details>
<summary><b>Why it matters? ⭐ </b></summary>
The PR adds a test that checks `column.datetime_format`, but elsewhere in this test module (and in the project's models) the column-level date format is surfaced as `python_date_format` (see other assertions in this file that use `column.python_date_format`). As written, the assertion will raise AttributeError or incorrectly pass/fail because `datetime_format` is not the model attribute. Changing the assertion to `column.python_date_format` correctly verifies that the imported `datetime_format` configuration was mapped to the model attribute. This fixes a real test bug rather than a cosmetic change.
</details>
<details>
<summary><b>Prompt for AI Agent 🤖 </b></summary>
```mdx
This is a comment left during a code review.
**Path:** tests/unit_tests/datasets/commands/importers/v1/import_test.py
**Line:** 701:701
**Comment:**
*Logic Error: The new test checks `column.datetime_format`, but the SQLAlchemy model exposes the date format as `python_date_format`, so this assertion will raise an AttributeError or validate the wrong field instead of confirming that the imported `datetime_format` config is correctly mapped.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org