You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@superset.apache.org by "mapledan (via GitHub)" <gi...@apache.org> on 2023/10/04 10:03:18 UTC
[PR] fix: change the validation logic for python_date_format [superset]
mapledan opened a new pull request, #25510:
URL: https://github.com/apache/superset/pull/25510
### SUMMARY
The `PUT /api/v1/dataset/{id}` endpoint includes the `python_date_format` validation.
A PR #24113 optimizes the validation, but there's still an issue with the '%Y%m%d' format.
According to the `DATETIME FORMAT` description, "If the timestamp format does not adhere to the ISO 8601 standard, you will need to define an expression and type for transforming the string into a date or timestamp.". Therefore, the validation should be more flexible.
In this PR, I have modified the logic to use the `strftime` function for validating the format string.
### BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF
<!--- Skip this if not applicable -->
### TESTING INSTRUCTIONS
<!--- Required! What steps can be taken to manually verify the changes? -->
### ADDITIONAL INFORMATION
<!--- Check any relevant boxes with "x" -->
<!--- HINT: Include "Fixes #nnn" if you are fixing an existing issue -->
- [ ] Has associated issue:
- [ ] Required feature flags:
- [ ] Changes UI
- [ ] Includes DB Migration (follow approval process in [SIP-59](https://github.com/apache/superset/issues/13351))
- [ ] Migration is atomic, supports rollback & is backwards-compatible
- [ ] Confirm DB migration upgrade and downgrade tested
- [ ] Runtime estimates and downtime expectations provided
- [ ] Introduces new feature or API
- [ ] Removes existing feature or API
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix: change the validation logic for python_date_format [superset]
Posted by "john-bodley (via GitHub)" <gi...@apache.org>.
john-bodley commented on code in PR #25510:
URL: https://github.com/apache/superset/pull/25510#discussion_r1397621278
##########
superset/datasets/schemas.py:
##########
@@ -43,18 +43,12 @@
def validate_python_date_format(value: str) -> None:
- regex = re.compile(
- r"""
- ^(
- epoch_s|epoch_ms|
- (?P<date>%Y([-/]%m([-/]%d)?)?)([\sT](?P<time>%H(:%M(:%S(\.%f)?)?)?))?
- )$
- """,
- re.VERBOSE,
- )
- match = regex.match(value or "")
- if not match:
- raise ValidationError([_("Invalid date/timestamp format")])
+ if value in ("epoch_s", "epoch_ms"):
+ return
+ try:
+ datetime.now().strftime(value or "")
Review Comment:
Thanks @mapledan for flagging the other PR. Per the tooltip (see attached) it clearly states that the format should be ISO 8601, though the placeholder text uses `/`. Granted the `/` (in addition to `-`) does adhere to lexicographical ordering, but we should strictly adhere to the ISO 8601 standard for historical reasons.
<img width="434" alt="Screenshot 2023-11-17 at 8 48 07 AM" src="https://github.com/apache/superset/assets/4567245/877b3ced-93fe-46e4-a52b-a4d682c0f094">
@jfrag1 this PR reverts the logic you defined in https://github.com/apache/superset/pull/24113. I've also authored https://github.com/apache/superset/pull/26017 which updates the placeholder text which likely lead people astray.
##########
superset/datasets/schemas.py:
##########
@@ -43,18 +43,12 @@
def validate_python_date_format(value: str) -> None:
- regex = re.compile(
- r"""
- ^(
- epoch_s|epoch_ms|
- (?P<date>%Y([-/]%m([-/]%d)?)?)([\sT](?P<time>%H(:%M(:%S(\.%f)?)?)?))?
- )$
- """,
- re.VERBOSE,
- )
- match = regex.match(value or "")
- if not match:
- raise ValidationError([_("Invalid date/timestamp format")])
+ if value in ("epoch_s", "epoch_ms"):
+ return
+ try:
+ datetime.now().strftime(value or "")
Review Comment:
Thanks @mapledan for flagging the other PR. Per the tooltip (see attached) it clearly states that the format should be ISO 8601, though the placeholder text uses `/`. Granted the `/` (in addition to `-`) does adhere to lexicographical ordering, but we should strictly adhere to the ISO 8601 standard for historical reasons.
<img width="434" alt="Screenshot 2023-11-17 at 8 48 07 AM" src="https://github.com/apache/superset/assets/4567245/877b3ced-93fe-46e4-a52b-a4d682c0f094">
@jfrag1 this PR reverts the logic you defined in https://github.com/apache/superset/pull/24113. I've also authored https://github.com/apache/superset/pull/26017 which updates the placeholder text which likely lead people astray.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix: change the validation logic for python_date_format [superset]
Posted by "mapledan (via GitHub)" <gi...@apache.org>.
mapledan commented on code in PR #25510:
URL: https://github.com/apache/superset/pull/25510#discussion_r1396767049
##########
superset/datasets/schemas.py:
##########
@@ -43,18 +43,12 @@
def validate_python_date_format(value: str) -> None:
- regex = re.compile(
- r"""
- ^(
- epoch_s|epoch_ms|
- (?P<date>%Y([-/]%m([-/]%d)?)?)([\sT](?P<time>%H(:%M(:%S(\.%f)?)?)?))?
- )$
- """,
- re.VERBOSE,
- )
- match = regex.match(value or "")
- if not match:
- raise ValidationError([_("Invalid date/timestamp format")])
+ if value in ("epoch_s", "epoch_ms"):
+ return
+ try:
+ datetime.now().strftime(value or "")
Review Comment:
Sorry, I missed the preceding sentence: 'which needs to adhere to the ISO 8601 standard to ensure that the lexicographical ordering coincides with the chronological ordering.'
In my test case, the `dateutil.parser.isoparse` strictly follows the ISO 8601 standard, so it does not allow YYYY/MM/DD format.
This will impact PR (#24113) as it allows the slash format.
And I traced the python_date_format.
It is using by [pandas.to_datetime](https://pandas.pydata.org/docs/reference/api/pandas.to_datetime.html) in `superset.utils.core.normalize_dttm_col`.
Which should we use as the validation, the`pandas.to_datetime` method, regex like `(?P<date>%Y([-/]?%m([-/]?%d)?)?)([\sT](?P<time>%H(:%M(:%S(\.%f)?)?)?))?`, or the `dateutil.parser.isoparse` method?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix: change the validation logic for python_date_format [superset]
Posted by "michael-s-molina (via GitHub)" <gi...@apache.org>.
michael-s-molina commented on PR #25510:
URL: https://github.com/apache/superset/pull/25510#issuecomment-1900207080
> I sense this PR should be resurrected and merged as part of the 4.0 breaking window. I realize that the proposal window has closed, but this is actually a (breaking) fix and thus I gather it's exempt from those rules.
This missed the 4.0 breaking window and was not submitted for lazy consensus. Someone might have disagreed with the proposed change if it was submitted for lazy consensus. I think the correct process here is to add a card to [Punted to 5.0 column](https://github.com/orgs/apache/projects/292/views/1) so we don't miss this again in the next breaking window.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix: change the validation logic for python_date_format [superset]
Posted by "john-bodley (via GitHub)" <gi...@apache.org>.
john-bodley commented on code in PR #25510:
URL: https://github.com/apache/superset/pull/25510#discussion_r1396618759
##########
superset/datasets/schemas.py:
##########
@@ -43,18 +43,12 @@
def validate_python_date_format(value: str) -> None:
- regex = re.compile(
- r"""
- ^(
- epoch_s|epoch_ms|
- (?P<date>%Y([-/]%m([-/]%d)?)?)([\sT](?P<time>%H(:%M(:%S(\.%f)?)?)?))?
- )$
- """,
- re.VERBOSE,
- )
- match = regex.match(value or "")
- if not match:
- raise ValidationError([_("Invalid date/timestamp format")])
+ if value in ("epoch_s", "epoch_ms"):
+ return
+ try:
+ datetime.now().strftime(value or "")
Review Comment:
@mapledan it should _only_ support ISO 8601—in addition to `epoch_s` and `epoch_ms`. The reason being is ISO 8601 guarantees lexigraphical ordering when comparing strings.
This is [mentioned](https://github.com/search?q=repo%3Aapache%2Fsuperset+%22ISO+8601%22&type=code) throughout the code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix: change the validation logic for python_date_format [superset]
Posted by "john-bodley (via GitHub)" <gi...@apache.org>.
john-bodley commented on code in PR #25510:
URL: https://github.com/apache/superset/pull/25510#discussion_r1397737566
##########
superset/datasets/schemas.py:
##########
@@ -43,18 +43,12 @@
def validate_python_date_format(value: str) -> None:
- regex = re.compile(
- r"""
- ^(
- epoch_s|epoch_ms|
- (?P<date>%Y([-/]%m([-/]%d)?)?)([\sT](?P<time>%H(:%M(:%S(\.%f)?)?)?))?
- )$
- """,
- re.VERBOSE,
- )
- match = regex.match(value or "")
- if not match:
- raise ValidationError([_("Invalid date/timestamp format")])
+ if value in ("epoch_s", "epoch_ms"):
+ return
+ try:
+ datetime.now().strftime(value or "")
Review Comment:
@jfrag1 the column in question isn't in the metadata database it's in the underlying analytical database(s), i.e., Trino, Hive, etc. where the temporal format is defined. It's difficult to estimate the impact of non-strict ISO 8601 validation, though since https://github.com/apache/superset/pull/24113 it only impacts newly registered datasets where the format is `%Y/%m/%d[...]`. Organizations likely have many registered temporal columns where the format is complete nonsense pre-https://github.com/apache/superset/pull/23678.
I think there's two option:
1. Relax requiring the ISO 8601 standard to include `/`, i.e., what is currently implemented (for right or wrong)*. This is potentially risky as the ISO 8601 standard _guarantees_ lexicographical ordering** whereas custom formatters don't.
2. Wait until 4.0 and mention in `UPDATING.md` the (re)restriction that only ISO 8601 formats are acceptable (enforced by way of validation at the API and database level) and that dataset owners will need to use a SQL expression instead to convert their string columns of the form `%Y/%m/%d` etc. to a `DATE`, `DATETIME`, etc. type.
\* Note due to lack of validation prior to https://github.com/apache/superset/pull/23678 (when we switched to the new RESTful API) there are likely organizations which have datasets which have a slew of formats in `table_columns.python_datetime_format` column which don't adhere to the current format and thus wont work as expected. I was able to confirm this in Airbnb's Superset metadata database. As part of 4.0 we should add a database level CHECK constraint for the `table_columns.python_datetime_format` column which will force admins and dataset owners to correct any violating temporal columns.
\** Lexicographical ordering is essential when we're filtering temporal string columns via the `>`, `>=`, `<`, and `<=` operators.
This is a classic "shift left" problem where the underlying validation should also reside in the database layer. This is mentioned in [[SIP-99C] Proposal for model and business validation](https://github.com/apache/superset/issues/25828) and something that has been somewhat of an achilles heal for community over the years.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix: change the validation logic for python_date_format [superset]
Posted by "mapledan (via GitHub)" <gi...@apache.org>.
mapledan commented on code in PR #25510:
URL: https://github.com/apache/superset/pull/25510#discussion_r1396600251
##########
superset/datasets/schemas.py:
##########
@@ -43,18 +43,12 @@
def validate_python_date_format(value: str) -> None:
- regex = re.compile(
- r"""
- ^(
- epoch_s|epoch_ms|
- (?P<date>%Y([-/]%m([-/]%d)?)?)([\sT](?P<time>%H(:%M(:%S(\.%f)?)?)?))?
- )$
- """,
- re.VERBOSE,
- )
- match = regex.match(value or "")
- if not match:
- raise ValidationError([_("Invalid date/timestamp format")])
+ if value in ("epoch_s", "epoch_ms"):
+ return
+ try:
+ datetime.now().strftime(value or "")
Review Comment:
Sure, I can use the `dateutil.parser.isoparse` method to make it support a more accurate ISO 8601 format than the original regular expression validation.
However, let me confirm again, does python_date_format only support the ISO 8601 format?
Because based on the description I read, it is written as follows: "If the timestamp format does not adhere to the ISO 8601 standard, you will need to define an expression and type for transforming the string into a date or timestamp."
That's why I chose to use `strftime` for validation to handle the user's defined expression.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix: change the validation logic for python_date_format [superset]
Posted by "mapledan (via GitHub)" <gi...@apache.org>.
mapledan commented on code in PR #25510:
URL: https://github.com/apache/superset/pull/25510#discussion_r1396767049
##########
superset/datasets/schemas.py:
##########
@@ -43,18 +43,12 @@
def validate_python_date_format(value: str) -> None:
- regex = re.compile(
- r"""
- ^(
- epoch_s|epoch_ms|
- (?P<date>%Y([-/]%m([-/]%d)?)?)([\sT](?P<time>%H(:%M(:%S(\.%f)?)?)?))?
- )$
- """,
- re.VERBOSE,
- )
- match = regex.match(value or "")
- if not match:
- raise ValidationError([_("Invalid date/timestamp format")])
+ if value in ("epoch_s", "epoch_ms"):
+ return
+ try:
+ datetime.now().strftime(value or "")
Review Comment:
Thank you for explaining it to me. I missed the preceding sentence: 'which needs to adhere to the ISO 8601 standard to ensure that the lexicographical ordering coincides with the chronological ordering.'
In my test case, the `dateutil.parser.isoparse` strictly follows the ISO 8601 standard, so it does not allow YYYY/MM/DD format.
This will impact PR (#24113) as it allows the slash format.
And I traced the python_date_format.
It is using by [pandas.to_datetime](https://pandas.pydata.org/docs/reference/api/pandas.to_datetime.html) in `superset.utils.core.normalize_dttm_col`.
Which should we use as the validation, the`pandas.to_datetime` method, regex like `(?P<date>%Y([-/]?%m([-/]?%d)?)?)([\sT](?P<time>%H(:%M(:%S(\.%f)?)?)?))?`, or the `dateutil.parser.isoparse` method?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix: change the validation logic for python_date_format [superset]
Posted by "john-bodley (via GitHub)" <gi...@apache.org>.
john-bodley commented on PR #25510:
URL: https://github.com/apache/superset/pull/25510#issuecomment-1903255836
@michael-s-molina having looked over this PR again, I don’t believe it’s breaking per se, i.e., assuming we don’t add a `CHECK` constraint then any existing non-ISO 8601 values will still be “valid”. This change merely ensures any newly defined values are ISO-8601 compliant which is consistent with the documentation, tooltips, etc.
My thinking was that I add the logic outlined in https://github.com/apache/superset/pull/26017 as a commit to this PR—alongside a line item in `UPDATING.md`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix: change the validation logic for python_date_format [superset]
Posted by "mapledan (via GitHub)" <gi...@apache.org>.
mapledan commented on code in PR #25510:
URL: https://github.com/apache/superset/pull/25510#discussion_r1463055166
##########
superset/daos/dataset.py:
##########
@@ -150,6 +152,17 @@ def validate_metrics_uniqueness(dataset_id: int, metrics_names: list[str]) -> bo
).all()
return len(dataset_query) == 0
+ @staticmethod
+ def validate_column_pdf_is_iso8601(dt_format: str) -> bool:
+ if dt_format in ("epoch_s", "epoch_ms", None):
Review Comment:
The TableColumn insert/update using bulk_insert_mappings or bulk_update_mappings does not trigger the ORM's `@validates`.
I refactored the duplicated code to perform validation exclusively at the DAO level.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix: change the validation logic for python_date_format [superset]
Posted by "john-bodley (via GitHub)" <gi...@apache.org>.
john-bodley commented on code in PR #25510:
URL: https://github.com/apache/superset/pull/25510#discussion_r1462618302
##########
UPDATING.md:
##########
@@ -40,6 +40,7 @@ assists people when migrating to a new version.
- [26636](https://github.com/apache/superset/issues/26636): Sets the `DASHBOARD_VIRTUALIZATION` feature flag to `True` by default. This feature was introduced by [21438](https://github.com/apache/superset/pull/21438) and will enable virtualization when rendering a dashboard's charts in an attempt to reduce the number of elements (DOM nodes) rendered at once. This is especially useful for large dashboards.
- [26637](https://github.com/apache/superset/issues/26637): Sets the `DRILL_BY` feature flag to `True` by default given that the feature has been tested for a while and reached a stable state.
- [26462](https://github.com/apache/superset/issues/26462): Removes the Profile feature given that it's not actively maintained and not widely used.
+- [25510](https://github.com/apache/superset/pull/25510): (re)Restriction that only ISO 8601 formats are acceptable (enforced by way of validation at the API and database level) and that dataset owners will need to use a SQL expression instead to convert their string columns of the form %Y/%m/%d etc. to a DATE, DATETIME, etc. type.
Review Comment:
```suggestion
- [25510](https://github.com/apache/superset/pull/25510): Reenforces that any newly defined Python data format (other than epoch) must adhere to the ISO 8601 standard (enforced by way of validation at the API and database level) after a previous relaxation to include slashes in addition to dashes. From now on when specifying new columns, dataset owners will need to use a SQL expression instead to convert their string columns of the form %Y/%m/%d etc. to a `DATE`, `DATETIME`, etc. type.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix: change the validation logic for python_date_format [superset]
Posted by "mapledan (via GitHub)" <gi...@apache.org>.
mapledan commented on code in PR #25510:
URL: https://github.com/apache/superset/pull/25510#discussion_r1396600251
##########
superset/datasets/schemas.py:
##########
@@ -43,18 +43,12 @@
def validate_python_date_format(value: str) -> None:
- regex = re.compile(
- r"""
- ^(
- epoch_s|epoch_ms|
- (?P<date>%Y([-/]%m([-/]%d)?)?)([\sT](?P<time>%H(:%M(:%S(\.%f)?)?)?))?
- )$
- """,
- re.VERBOSE,
- )
- match = regex.match(value or "")
- if not match:
- raise ValidationError([_("Invalid date/timestamp format")])
+ if value in ("epoch_s", "epoch_ms"):
+ return
+ try:
+ datetime.now().strftime(value or "")
Review Comment:
Sure, I can use the `dateutil.parser.isoparse` method to make it support a more flexiable ISO 8601 format than the original regular expression validation.
However, let me confirm again, does python_date_format only support the ISO 8601 format?
Because based on the description I read, it is written as follows: "If the timestamp format does not adhere to the ISO 8601 standard, you will need to define an expression and type for transforming the string into a date or timestamp."
That's why I chose to use `strftime` for validation to handle the user's defined expression.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix: change the validation logic for python_date_format [superset]
Posted by "john-bodley (via GitHub)" <gi...@apache.org>.
john-bodley commented on PR #25510:
URL: https://github.com/apache/superset/pull/25510#issuecomment-1899385655
@mapledan and @michael-s-molina I sense this PR should be resurrected and merged as part of the 4.0 breaking window. I realize that the proposal window has closed, but this is actually a (breaking) fix and thus I gather it's exempt from those rules.
@mapledan in addition to the current logic you've outlined here I sense the PR should be updated per (2) in https://github.com/apache/superset/pull/25510#discussion_r1397737566 which would require:
- Updating the `UPDATING.md`.
- Adding a `CHECK` constraint to the `table_columns.python_datetime_format` column
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix: change the validation logic for python_date_format [superset]
Posted by "sfirke (via GitHub)" <gi...@apache.org>.
sfirke commented on PR #25510:
URL: https://github.com/apache/superset/pull/25510#issuecomment-1747096426
Thanks for submitting this! ping @Antonio-RiveroMartnez and @jfrag1 for review, they worked on that PR you linked.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix: change the validation logic for python_date_format [superset]
Posted by "john-bodley (via GitHub)" <gi...@apache.org>.
john-bodley commented on code in PR #25510:
URL: https://github.com/apache/superset/pull/25510#discussion_r1397675319
##########
superset/datasets/schemas.py:
##########
@@ -43,18 +43,12 @@
def validate_python_date_format(value: str) -> None:
- regex = re.compile(
- r"""
- ^(
- epoch_s|epoch_ms|
- (?P<date>%Y([-/]%m([-/]%d)?)?)([\sT](?P<time>%H(:%M(:%S(\.%f)?)?)?))?
- )$
- """,
- re.VERBOSE,
- )
- match = regex.match(value or "")
- if not match:
- raise ValidationError([_("Invalid date/timestamp format")])
+ if value in ("epoch_s", "epoch_ms"):
+ return
+ try:
+ datetime.now().strftime(value or "")
Review Comment:
@jfrag1 a migration isn't possible as the string format is specified by the underly column in the dataset. Granted this change has the potential to churn users who were exposed to the relaxation in https://github.com/apache/superset/pull/24113, i.e., since 3.0 and thus they would need to update there dataset definition to use a SQL expression instead.
@michael-s-molina this is likely another great example of a breaking change and thus we likely need to wait until 4.0 to re-restrict the eligible formats.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix: change the validation logic for python_date_format [superset]
Posted by "john-bodley (via GitHub)" <gi...@apache.org>.
john-bodley commented on PR #25510:
URL: https://github.com/apache/superset/pull/25510#issuecomment-1905115628
@mapledan would you also mind incorporating y changes from https://github.com/apache/superset/pull/26017 into this PR? Ideally it would be best—especially given that there's Now a line item in `UPDATING.md` to bundle everything into one change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix: change the validation logic for python_date_format [superset]
Posted by "sfirke (via GitHub)" <gi...@apache.org>.
sfirke commented on PR #25510:
URL: https://github.com/apache/superset/pull/25510#issuecomment-1900714202
Maybe we could discuss at the townhall today? I feel like a bug fix should not be subject to as rigorous a community approval as the changes that were passed via lazy consensus and think perhaps we should be able to merge breaking bug fixes during this window.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix: change the validation logic for python_date_format [superset]
Posted by "john-bodley (via GitHub)" <gi...@apache.org>.
john-bodley merged PR #25510:
URL: https://github.com/apache/superset/pull/25510
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix: change the validation logic for python_date_format [superset]
Posted by "john-bodley (via GitHub)" <gi...@apache.org>.
john-bodley commented on code in PR #25510:
URL: https://github.com/apache/superset/pull/25510#discussion_r1462636105
##########
superset/connectors/sqla/models.py:
##########
@@ -830,6 +831,17 @@ def init_on_load(self) -> None:
def __repr__(self) -> str:
return str(self.column_name)
+ @validates("python_date_format")
+ def validate_pdf_is_iso8601(self, _: str, dt_format: str) -> str:
Review Comment:
The function name is somewhat misleading as the Python date format doesn't strictly need to be ISO 8601 seeing as we support the `epoch_s` and `epoch_ms` values.
##########
superset/commands/dataset/update.py:
##########
@@ -139,6 +140,13 @@ def _validate_columns(
):
exceptions.append(DatasetColumnsExistsValidationError())
+ # validate python_date_format is ISO8601 format
+ for col in columns:
+ if not DatasetDAO.validate_column_pdf_is_iso8601(
+ col.get("python_date_format", None)
Review Comment:
I'm surprised Mypy doesn't complain here given you potentially passing `None`. Personally I think it's best not to accept `None` as a valid input if you then simply return. This case should be handled separately.
##########
superset/daos/dataset.py:
##########
@@ -150,6 +152,17 @@ def validate_metrics_uniqueness(dataset_id: int, metrics_names: list[str]) -> bo
).all()
return len(dataset_query) == 0
+ @staticmethod
+ def validate_column_pdf_is_iso8601(dt_format: str) -> bool:
+ if dt_format in ("epoch_s", "epoch_ms", None):
Review Comment:
See previous comment.
##########
superset/connectors/sqla/models.py:
##########
@@ -830,6 +831,17 @@ def init_on_load(self) -> None:
def __repr__(self) -> str:
return str(self.column_name)
+ @validates("python_date_format")
+ def validate_pdf_is_iso8601(self, _: str, dt_format: str) -> str:
+ if dt_format in ("epoch_s", "epoch_ms", None):
+ return dt_format
Review Comment:
Mypy should be borking on the return type here as well given `dt_format` could be `None`.
##########
superset/commands/dataset/update.py:
##########
@@ -139,6 +140,13 @@ def _validate_columns(
):
exceptions.append(DatasetColumnsExistsValidationError())
+ # validate python_date_format is ISO8601 format
Review Comment:
```suggestion
# validate python_date_format is ISO 8601 format
```
##########
superset/connectors/sqla/models.py:
##########
@@ -830,6 +831,17 @@ def init_on_load(self) -> None:
def __repr__(self) -> str:
return str(self.column_name)
+ @validates("python_date_format")
+ def validate_pdf_is_iso8601(self, _: str, dt_format: str) -> str:
+ if dt_format in ("epoch_s", "epoch_ms", None):
Review Comment:
I don't think `None` is required given the type hint for `dt_formart` is `str`.
##########
superset/daos/dataset.py:
##########
@@ -150,6 +152,17 @@ def validate_metrics_uniqueness(dataset_id: int, metrics_names: list[str]) -> bo
).all()
return len(dataset_query) == 0
+ @staticmethod
+ def validate_column_pdf_is_iso8601(dt_format: str) -> bool:
+ if dt_format in ("epoch_s", "epoch_ms", None):
Review Comment:
Any reason this logic is somewhat duplicated in three places? Can this be refactored into a helper method if neither the model nor the DAO seems appropriate for both cases?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix: change the validation logic for python_date_format [superset]
Posted by "john-bodley (via GitHub)" <gi...@apache.org>.
john-bodley commented on PR #25510:
URL: https://github.com/apache/superset/pull/25510#issuecomment-1747346031
Thanks @mapledan for the PR. Would you mind completing the PR description to include testing instructions? Additionally adding unit tests would be highly beneficial.
cc @betodealmeida @jfrag1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix: change the validation logic for python_date_format [superset]
Posted by "jfrag1 (via GitHub)" <gi...@apache.org>.
jfrag1 commented on code in PR #25510:
URL: https://github.com/apache/superset/pull/25510#discussion_r1397684990
##########
superset/datasets/schemas.py:
##########
@@ -43,18 +43,12 @@
def validate_python_date_format(value: str) -> None:
- regex = re.compile(
- r"""
- ^(
- epoch_s|epoch_ms|
- (?P<date>%Y([-/]%m([-/]%d)?)?)([\sT](?P<time>%H(:%M(:%S(\.%f)?)?)?))?
- )$
- """,
- re.VERBOSE,
- )
- match = regex.match(value or "")
- if not match:
- raise ValidationError([_("Invalid date/timestamp format")])
+ if value in ("epoch_s", "epoch_ms"):
+ return
+ try:
+ datetime.now().strftime(value or "")
Review Comment:
Also I'm not sure what you mean by "a migration isn't possible", can't we have a migration that updates the table that stores column definitions?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix: change the validation logic for python_date_format [superset]
Posted by "john-bodley (via GitHub)" <gi...@apache.org>.
john-bodley commented on code in PR #25510:
URL: https://github.com/apache/superset/pull/25510#discussion_r1399919952
##########
superset/datasets/schemas.py:
##########
@@ -42,26 +43,25 @@
}
-def validate_python_date_format(value: str) -> None:
- regex = re.compile(
- r"""
- ^(
- epoch_s|epoch_ms|
- (?P<date>%Y([-/]%m([-/]%d)?)?)([\sT](?P<time>%H(:%M(:%S(\.%f)?)?)?))?
- )$
- """,
- re.VERBOSE,
- )
- match = regex.match(value or "")
- if not match:
- raise ValidationError([_("Invalid date/timestamp format")])
+def validate_python_date_format(value: str) -> bool:
+ if value in ("epoch_s", "epoch_ms", None):
+ return True
+ try:
+ dt_str = datetime.now().strftime(value)
+ isoparse(dt_str)
Review Comment:
Thanks @mapledan for the explanation. I forgot that `value` was a `strptime` format as opposed to a serialized datetime.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix: change the validation logic for python_date_format [superset]
Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #25510:
URL: https://github.com/apache/superset/pull/25510#issuecomment-1817421812
## [Codecov](https://app.codecov.io/gh/apache/superset/pull/25510?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
All modified and coverable lines are covered by tests :white_check_mark:
> Comparison is base [(`db7f5fe`)](https://app.codecov.io/gh/apache/superset/commit/db7f5fed31ea8ec4166e45422036e47544ed33a8?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) 67.02% compared to head [(`baae852`)](https://app.codecov.io/gh/apache/superset/pull/25510?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) 58.75%.
> Report is 190 commits behind head on master.
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #25510 +/- ##
==========================================
- Coverage 67.02% 58.75% -8.27%
==========================================
Files 1909 1911 +2
Lines 74918 75252 +334
Branches 8297 8297
==========================================
- Hits 50212 44213 -5999
- Misses 22564 28897 +6333
Partials 2142 2142
```
| [Flag](https://app.codecov.io/gh/apache/superset/pull/25510/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [mysql](https://app.codecov.io/gh/apache/superset/pull/25510/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [postgres](https://app.codecov.io/gh/apache/superset/pull/25510/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [presto](https://app.codecov.io/gh/apache/superset/pull/25510/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `53.62% <41.66%> (?)` | |
| [python](https://app.codecov.io/gh/apache/superset/pull/25510/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `61.48% <100.00%> (-17.17%)` | :arrow_down: |
| [sqlite](https://app.codecov.io/gh/apache/superset/pull/25510/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [unit](https://app.codecov.io/gh/apache/superset/pull/25510/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `55.71% <100.00%> (?)` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
</details>
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/superset/pull/25510?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix: change the validation logic for python_date_format [superset]
Posted by "john-bodley (via GitHub)" <gi...@apache.org>.
john-bodley commented on code in PR #25510:
URL: https://github.com/apache/superset/pull/25510#discussion_r1396618759
##########
superset/datasets/schemas.py:
##########
@@ -43,18 +43,12 @@
def validate_python_date_format(value: str) -> None:
- regex = re.compile(
- r"""
- ^(
- epoch_s|epoch_ms|
- (?P<date>%Y([-/]%m([-/]%d)?)?)([\sT](?P<time>%H(:%M(:%S(\.%f)?)?)?))?
- )$
- """,
- re.VERBOSE,
- )
- match = regex.match(value or "")
- if not match:
- raise ValidationError([_("Invalid date/timestamp format")])
+ if value in ("epoch_s", "epoch_ms"):
+ return
+ try:
+ datetime.now().strftime(value or "")
Review Comment:
@mapledan it should _only_ support ISO 8601—in addition to `epoch_s` and `epoch_ms`. The reason being is ISO 8601 guarantees lexigraphical ordering when comparing strings.
This is [mentioned]{https://github.com/search?q=repo%3Aapache%2Fsuperset+%22ISO+8601%22&type=code) throughout the code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix: change the validation logic for python_date_format [superset]
Posted by "jfrag1 (via GitHub)" <gi...@apache.org>.
jfrag1 commented on code in PR #25510:
URL: https://github.com/apache/superset/pull/25510#discussion_r1397663895
##########
superset/datasets/schemas.py:
##########
@@ -43,18 +43,12 @@
def validate_python_date_format(value: str) -> None:
- regex = re.compile(
- r"""
- ^(
- epoch_s|epoch_ms|
- (?P<date>%Y([-/]%m([-/]%d)?)?)([\sT](?P<time>%H(:%M(:%S(\.%f)?)?)?))?
- )$
- """,
- re.VERBOSE,
- )
- match = regex.match(value or "")
- if not match:
- raise ValidationError([_("Invalid date/timestamp format")])
+ if value in ("epoch_s", "epoch_ms"):
+ return
+ try:
+ datetime.now().strftime(value or "")
Review Comment:
I think if we want to move towards strict ISO, we definitely need a migration to update existing `python_date_format`'s that currently use `/`. Without a migration, any existing datasets with columns that use `/` will not be able to be updated without changing the `python_date_format` since they'll fail the validation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix: change the validation logic for python_date_format [superset]
Posted by "jfrag1 (via GitHub)" <gi...@apache.org>.
jfrag1 commented on code in PR #25510:
URL: https://github.com/apache/superset/pull/25510#discussion_r1397682880
##########
superset/datasets/schemas.py:
##########
@@ -43,18 +43,12 @@
def validate_python_date_format(value: str) -> None:
- regex = re.compile(
- r"""
- ^(
- epoch_s|epoch_ms|
- (?P<date>%Y([-/]%m([-/]%d)?)?)([\sT](?P<time>%H(:%M(:%S(\.%f)?)?)?))?
- )$
- """,
- re.VERBOSE,
- )
- match = regex.match(value or "")
- if not match:
- raise ValidationError([_("Invalid date/timestamp format")])
+ if value in ("epoch_s", "epoch_ms"):
+ return
+ try:
+ datetime.now().strftime(value or "")
Review Comment:
@john-bodley #24113 didn't expose a new relaxation to users; that PR was a follow-up to https://github.com/apache/superset/pull/23678, which changed the endpoint used to update dataset definitions. The old endpoint did no validation at all for `python_date_format`, so there were people using slashes before that too (since the placeholder indicated to do so).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix: change the validation logic for python_date_format [superset]
Posted by "john-bodley (via GitHub)" <gi...@apache.org>.
john-bodley commented on code in PR #25510:
URL: https://github.com/apache/superset/pull/25510#discussion_r1397737566
##########
superset/datasets/schemas.py:
##########
@@ -43,18 +43,12 @@
def validate_python_date_format(value: str) -> None:
- regex = re.compile(
- r"""
- ^(
- epoch_s|epoch_ms|
- (?P<date>%Y([-/]%m([-/]%d)?)?)([\sT](?P<time>%H(:%M(:%S(\.%f)?)?)?))?
- )$
- """,
- re.VERBOSE,
- )
- match = regex.match(value or "")
- if not match:
- raise ValidationError([_("Invalid date/timestamp format")])
+ if value in ("epoch_s", "epoch_ms"):
+ return
+ try:
+ datetime.now().strftime(value or "")
Review Comment:
@jfrag1 the column in question isn't in the metadata database it's in the underlying analytical database(s), i.e., Trino, Hive, etc. where the temporal format is defined and thus Superset merely reflects how the data is defined. It's difficult to estimate the impact of non-strict ISO 8601 validation, though since https://github.com/apache/superset/pull/24113 it only impacts newly registered datasets where the format is `%Y/%m/%d[...]`. Organizations likely have many registered temporal columns where the format is complete nonsense pre-https://github.com/apache/superset/pull/23678.
I think there's two option:
1. Relax requiring the ISO 8601 standard to include `/`, i.e., what is currently implemented (for right or wrong)*. This is potentially risky as the ISO 8601 standard _guarantees_ lexicographical ordering** whereas custom formatters don't.
2. Wait until 4.0 and mention in `UPDATING.md` the (re)restriction that only ISO 8601 formats are acceptable (enforced by way of validation at the API and database level) and that dataset owners will need to use a SQL expression instead to convert their string columns of the form `%Y/%m/%d` etc. to a `DATE`, `DATETIME`, etc. type.
\* Note due to lack of validation prior to https://github.com/apache/superset/pull/23678 (when we switched to the new RESTful API) there are likely organizations which have datasets which have a slew of formats in `table_columns.python_datetime_format` column which don't adhere to the current format and thus wont work as expected. I was able to confirm this in Airbnb's Superset metadata database. As part of 4.0 we should add a database level CHECK constraint for the `table_columns.python_datetime_format` column which will force admins and dataset owners to correct any violating temporal columns.
\** Lexicographical ordering is essential when we're filtering temporal string columns via the `>`, `>=`, `<`, and `<=` operators.
This is a classic "shift left" problem where the underlying validation should also reside in the database layer. This is mentioned in [[SIP-99C] Proposal for model and business validation](https://github.com/apache/superset/issues/25828) and something that has been somewhat of an achilles heal for community over the years.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix: change the validation logic for python_date_format [superset]
Posted by "john-bodley (via GitHub)" <gi...@apache.org>.
john-bodley commented on code in PR #25510:
URL: https://github.com/apache/superset/pull/25510#discussion_r1397621278
##########
superset/datasets/schemas.py:
##########
@@ -43,18 +43,12 @@
def validate_python_date_format(value: str) -> None:
- regex = re.compile(
- r"""
- ^(
- epoch_s|epoch_ms|
- (?P<date>%Y([-/]%m([-/]%d)?)?)([\sT](?P<time>%H(:%M(:%S(\.%f)?)?)?))?
- )$
- """,
- re.VERBOSE,
- )
- match = regex.match(value or "")
- if not match:
- raise ValidationError([_("Invalid date/timestamp format")])
+ if value in ("epoch_s", "epoch_ms"):
+ return
+ try:
+ datetime.now().strftime(value or "")
Review Comment:
Thanks @mapledan for flagging the other PR. Per the tooltip (see attached) it clearly states that the format should be ISO 8601, though the placeholder text uses `/`. Granted the `/` (in addition to `-`) does adhere to lexicographical ordering, but we should strictly adhere to the ISO 8601 standard for historical reasons.
<img width="434" alt="Screenshot 2023-11-17 at 8 48 07 AM" src="https://github.com/apache/superset/assets/4567245/877b3ced-93fe-46e4-a52b-a4d682c0f094">
@jfrag1 this PR reverts the logic you defined in https://github.com/apache/superset/pull/24113. I've also authored https://github.com/apache/superset/pull/26017 which updates the placeholder text which likely lead people astray.
@mapledan feel free to merge the logic from https://github.com/apache/superset/pull/26017 into your PR if you think it makes sense to have both changes combined.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix: change the validation logic for python_date_format [superset]
Posted by "john-bodley (via GitHub)" <gi...@apache.org>.
john-bodley commented on code in PR #25510:
URL: https://github.com/apache/superset/pull/25510#discussion_r1397741013
##########
superset/datasets/schemas.py:
##########
@@ -42,26 +43,25 @@
}
-def validate_python_date_format(value: str) -> None:
- regex = re.compile(
- r"""
- ^(
- epoch_s|epoch_ms|
- (?P<date>%Y([-/]%m([-/]%d)?)?)([\sT](?P<time>%H(:%M(:%S(\.%f)?)?)?))?
- )$
- """,
- re.VERBOSE,
- )
- match = regex.match(value or "")
- if not match:
- raise ValidationError([_("Invalid date/timestamp format")])
+def validate_python_date_format(value: str) -> bool:
+ if value in ("epoch_s", "epoch_ms", None):
+ return True
+ try:
+ dt_str = datetime.now().strftime(value)
Review Comment:
```suggestion
```
##########
superset/datasets/schemas.py:
##########
@@ -42,26 +43,25 @@
}
-def validate_python_date_format(value: str) -> None:
- regex = re.compile(
- r"""
- ^(
- epoch_s|epoch_ms|
- (?P<date>%Y([-/]%m([-/]%d)?)?)([\sT](?P<time>%H(:%M(:%S(\.%f)?)?)?))?
- )$
- """,
- re.VERBOSE,
- )
- match = regex.match(value or "")
- if not match:
- raise ValidationError([_("Invalid date/timestamp format")])
+def validate_python_date_format(value: str) -> bool:
+ if value in ("epoch_s", "epoch_ms", None):
+ return True
+ try:
+ dt_str = datetime.now().strftime(value)
+ isoparse(dt_str)
Review Comment:
```suggestion
isoparse(value)
```
##########
superset/datasets/schemas.py:
##########
@@ -42,26 +43,25 @@
}
-def validate_python_date_format(value: str) -> None:
- regex = re.compile(
- r"""
- ^(
- epoch_s|epoch_ms|
- (?P<date>%Y([-/]%m([-/]%d)?)?)([\sT](?P<time>%H(:%M(:%S(\.%f)?)?)?))?
- )$
- """,
- re.VERBOSE,
- )
- match = regex.match(value or "")
- if not match:
- raise ValidationError([_("Invalid date/timestamp format")])
+def validate_python_date_format(value: str) -> bool:
+ if value in ("epoch_s", "epoch_ms", None):
Review Comment:
```suggestion
if value in ("epoch_s", "epoch_ms"):
```
Per the type hint `value` cannot be `None`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix: change the validation logic for python_date_format [superset]
Posted by "mapledan (via GitHub)" <gi...@apache.org>.
mapledan commented on PR #25510:
URL: https://github.com/apache/superset/pull/25510#issuecomment-1748727855
Sure, I have added the testing instructions and included unit tests as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix: change the validation logic for python_date_format [superset]
Posted by "john-bodley (via GitHub)" <gi...@apache.org>.
john-bodley commented on code in PR #25510:
URL: https://github.com/apache/superset/pull/25510#discussion_r1397625390
##########
tests/unit_tests/datasets/schema_tests.py:
##########
@@ -0,0 +1,49 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# pylint: disable=import-outside-toplevel, invalid-name, unused-argument, redefined-outer-name
+import pytest
+from marshmallow import ValidationError
+
+from superset.datasets.schemas import validate_python_date_format
+
+
+# pylint: disable=too-few-public-methods
+@pytest.mark.parametrize(
+ "payload",
+ [
+ None,
+ "epoch_ms",
+ "epoch_s",
+ "%Y/%m/%dT%H:%M:%S.%f",
Review Comment:
@mapledan could you move this test case to `test_validate_python_date_format_raises`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix: change the validation logic for python_date_format [superset]
Posted by "john-bodley (via GitHub)" <gi...@apache.org>.
john-bodley commented on code in PR #25510:
URL: https://github.com/apache/superset/pull/25510#discussion_r1396303474
##########
superset/datasets/schemas.py:
##########
@@ -43,18 +43,12 @@
def validate_python_date_format(value: str) -> None:
- regex = re.compile(
- r"""
- ^(
- epoch_s|epoch_ms|
- (?P<date>%Y([-/]%m([-/]%d)?)?)([\sT](?P<time>%H(:%M(:%S(\.%f)?)?)?))?
- )$
- """,
- re.VERBOSE,
- )
- match = regex.match(value or "")
- if not match:
- raise ValidationError([_("Invalid date/timestamp format")])
+ if value in ("epoch_s", "epoch_ms"):
+ return
+ try:
+ datetime.now().strftime(value or "")
Review Comment:
@mapledan would you mind using the [dateutil.parser.isoparse](https://dateutil.readthedocs.io/en/stable/parser.html#dateutil.parser.isoparse) method instead?
##########
tests/unit_tests/datasets/schema_tests.py:
##########
@@ -0,0 +1,77 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# pylint: disable=import-outside-toplevel, invalid-name, unused-argument, redefined-outer-name
+import pytest
+from marshmallow import ValidationError
+
+
+# pylint: disable=too-few-public-methods
+@pytest.mark.parametrize(
+ "payload",
+ [
+ (
+ {
+ "column_name": "insert_time",
+ "filterable": True,
+ "groupby": True,
+ "python_date_format": None,
+ }
+ ),
+ (
+ {
+ "column_name": "insert_time",
+ "filterable": True,
+ "groupby": True,
+ "python_date_format": "epoch_ms",
+ }
+ ),
+ (
+ {
+ "column_name": "insert_time",
+ "filterable": True,
+ "groupby": True,
+ "python_date_format": "epoch_s",
+ }
+ ),
+ (
+ {
+ "column_name": "insert_time",
+ "filterable": True,
+ "groupby": True,
+ "python_date_format": "%Y/%m/%dT%H:%M:%S.%f",
+ }
+ ),
+ (
+ {
+ "column_name": "insert_time",
+ "filterable": True,
+ "groupby": True,
+ "python_date_format": "%Y%m%d",
+ }
+ ),
+ ],
+)
+def test_datasets_schema_update_column_datetime_format(payload) -> None:
+ from superset.datasets.schemas import DatasetColumnsPutSchema
+
+ schema = DatasetColumnsPutSchema()
+
+ try:
+ schema.load(payload)
+ except ValidationError as err:
+ assert False, err.messages
Review Comment:
Thanks for adding the tests. I'm a little perplexed as how this works or is intended to work. Ideally these tests should be split into two: i) those where the schema are valid and those which the schema is valid. The logic would be of the form:
```python
def test_datasets_schema_update_column_datetime_format(payload) -> None:
DatasetColumnsPutSchema().load(payload)
```
and
```python
def test_datasets_schema_update_column_datetime_format_raises(payload) -> None:
with pytest.raises(ValidationError):
DatasetColumnsPutSchema().load(payload)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix: change the validation logic for python_date_format [superset]
Posted by "john-bodley (via GitHub)" <gi...@apache.org>.
john-bodley commented on code in PR #25510:
URL: https://github.com/apache/superset/pull/25510#discussion_r1396618759
##########
superset/datasets/schemas.py:
##########
@@ -43,18 +43,12 @@
def validate_python_date_format(value: str) -> None:
- regex = re.compile(
- r"""
- ^(
- epoch_s|epoch_ms|
- (?P<date>%Y([-/]%m([-/]%d)?)?)([\sT](?P<time>%H(:%M(:%S(\.%f)?)?)?))?
- )$
- """,
- re.VERBOSE,
- )
- match = regex.match(value or "")
- if not match:
- raise ValidationError([_("Invalid date/timestamp format")])
+ if value in ("epoch_s", "epoch_ms"):
+ return
+ try:
+ datetime.now().strftime(value or "")
Review Comment:
@mapledan it should _only_ support ISO 8601—in addition to `epoch_s` and `epoch_ms`. The reason being is ISO 8601 guarantees lexigraphical ordering when comparing strings.
##########
superset/datasets/schemas.py:
##########
@@ -43,18 +43,12 @@
def validate_python_date_format(value: str) -> None:
- regex = re.compile(
- r"""
- ^(
- epoch_s|epoch_ms|
- (?P<date>%Y([-/]%m([-/]%d)?)?)([\sT](?P<time>%H(:%M(:%S(\.%f)?)?)?))?
- )$
- """,
- re.VERBOSE,
- )
- match = regex.match(value or "")
- if not match:
- raise ValidationError([_("Invalid date/timestamp format")])
+ if value in ("epoch_s", "epoch_ms"):
+ return
+ try:
+ datetime.now().strftime(value or "")
Review Comment:
@mapledan it should _only_ support ISO 8601—in addition to `epoch_s` and `epoch_ms`. The reason being is ISO 8601 guarantees lexigraphical ordering when comparing strings.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix: change the validation logic for python_date_format [superset]
Posted by "mapledan (via GitHub)" <gi...@apache.org>.
mapledan commented on code in PR #25510:
URL: https://github.com/apache/superset/pull/25510#discussion_r1396767049
##########
superset/datasets/schemas.py:
##########
@@ -43,18 +43,12 @@
def validate_python_date_format(value: str) -> None:
- regex = re.compile(
- r"""
- ^(
- epoch_s|epoch_ms|
- (?P<date>%Y([-/]%m([-/]%d)?)?)([\sT](?P<time>%H(:%M(:%S(\.%f)?)?)?))?
- )$
- """,
- re.VERBOSE,
- )
- match = regex.match(value or "")
- if not match:
- raise ValidationError([_("Invalid date/timestamp format")])
+ if value in ("epoch_s", "epoch_ms"):
+ return
+ try:
+ datetime.now().strftime(value or "")
Review Comment:
Thanks for the explanation, the lexicographical ordering is crucial in data analysis.
In my test case, the `dateutil.parser.isoparse` strictly follows the ISO 8601 standard, so it does not allow YYYY/MM/DD format.
This will impact PR (#24113) as it allows the slash format.
And I traced the python_date_format.
It is using by [pandas.to_datetime](https://pandas.pydata.org/docs/reference/api/pandas.to_datetime.html) in `superset.utils.core.normalize_dttm_col`.
Which should we use as the validation, the`pandas.to_datetime` method, regex like `(?P<date>%Y([-/]?%m([-/]?%d)?)?)([\sT](?P<time>%H(:%M(:%S(\.%f)?)?)?))?`, or the `dateutil.parser.isoparse` method?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix: change the validation logic for python_date_format [superset]
Posted by "john-bodley (via GitHub)" <gi...@apache.org>.
john-bodley commented on code in PR #25510:
URL: https://github.com/apache/superset/pull/25510#discussion_r1462618302
##########
UPDATING.md:
##########
@@ -40,6 +40,7 @@ assists people when migrating to a new version.
- [26636](https://github.com/apache/superset/issues/26636): Sets the `DASHBOARD_VIRTUALIZATION` feature flag to `True` by default. This feature was introduced by [21438](https://github.com/apache/superset/pull/21438) and will enable virtualization when rendering a dashboard's charts in an attempt to reduce the number of elements (DOM nodes) rendered at once. This is especially useful for large dashboards.
- [26637](https://github.com/apache/superset/issues/26637): Sets the `DRILL_BY` feature flag to `True` by default given that the feature has been tested for a while and reached a stable state.
- [26462](https://github.com/apache/superset/issues/26462): Removes the Profile feature given that it's not actively maintained and not widely used.
+- [25510](https://github.com/apache/superset/pull/25510): (re)Restriction that only ISO 8601 formats are acceptable (enforced by way of validation at the API and database level) and that dataset owners will need to use a SQL expression instead to convert their string columns of the form %Y/%m/%d etc. to a DATE, DATETIME, etc. type.
Review Comment:
```suggestion
- [25510](https://github.com/apache/superset/pull/25510): Reenforces that any newly defined Python data format (other than epoch) must adhere to the ISO 8601 standard (enforced by way of validation at the API and database level) after a previous relaxation to include slashes in addition to dashes. From now on when specifying new columns, dataset owners will need to use a SQL expression instead to convert their string columns of the form %Y/%m/%d etc. to a `DATE`, `DATETIME`, etc. type.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix: change the validation logic for python_date_format [superset]
Posted by "michael-s-molina (via GitHub)" <gi...@apache.org>.
michael-s-molina commented on code in PR #25510:
URL: https://github.com/apache/superset/pull/25510#discussion_r1462089145
##########
UPDATING.md:
##########
@@ -40,6 +40,7 @@ assists people when migrating to a new version.
- [26636](https://github.com/apache/superset/issues/26636): Sets the `DASHBOARD_VIRTUALIZATION` feature flag to `True` by default. This feature was introduced by [21438](https://github.com/apache/superset/pull/21438) and will enable virtualization when rendering a dashboard's charts in an attempt to reduce the number of elements (DOM nodes) rendered at once. This is especially useful for large dashboards.
- [26637](https://github.com/apache/superset/issues/26637): Sets the `DRILL_BY` feature flag to `True` by default given that the feature has been tested for a while and reached a stable state.
- [26462](https://github.com/apache/superset/issues/26462): Removes the Profile feature given that it's not actively maintained and not widely used.
+- [25510](https://github.com/apache/superset/pull/25510): (re)Restriction that only ISO 8601 formats are acceptable (enforced by way of validation at the API and database level) and that dataset owners will need to use a SQL expression instead to convert their string columns of the form %Y/%m/%d etc. to a DATE, DATETIME, etc. type.
Review Comment:
As I understand by @john-bodley's comment, this won't be a breaking change. If that's the case, this should be moved to the Next section and the text should be changed given that:
> any existing non-ISO 8601 values will still be “valid”.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix: change the validation logic for python_date_format [superset]
Posted by "mapledan (via GitHub)" <gi...@apache.org>.
mapledan commented on code in PR #25510:
URL: https://github.com/apache/superset/pull/25510#discussion_r1463055166
##########
superset/daos/dataset.py:
##########
@@ -150,6 +152,17 @@ def validate_metrics_uniqueness(dataset_id: int, metrics_names: list[str]) -> bo
).all()
return len(dataset_query) == 0
+ @staticmethod
+ def validate_column_pdf_is_iso8601(dt_format: str) -> bool:
+ if dt_format in ("epoch_s", "epoch_ms", None):
Review Comment:
The TableColumn insert/update using bulk_insert_mappings or bulk_update_mappings does not trigger the ORM's @validates.
I refactored the duplicated code to perform validation exclusively at the DAO level.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org