You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@superset.apache.org by GitBox <gi...@apache.org> on 2020/02/14 01:19:35 UTC
[GitHub] [incubator-superset] fx19880617 opened a new pull request #9142:
Support human readable datetime type for PinotDB
fx19880617 opened a new pull request #9142: Support human readable datetime type for PinotDB
URL: https://github.com/apache/incubator-superset/pull/9142
### CATEGORY
Choose one
- [ ] Bug Fix
- [x] Enhancement (new features, refinement)
- [ ] Refactor
- [ ] Add tests
- [ ] Build / Development Environment
- [ ] Documentation
### SUMMARY
Support human readable datetime type for PinotDB
Now we can support datetime like '%Y-%m-%d' for time Series.
### BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF
<!--- Skip this if not applicable -->
### TEST PLAN
<!--- What steps should be taken to verify the changes -->
### ADDITIONAL INFORMATION
<!--- Check any relevant boxes with "x" -->
<!--- HINT: Include "Fixes #nnn" if you are fixing an existing issue -->
- [ ] Has associated issue:
- [ ] Changes UI
- [ ] Requires DB Migration.
- [ ] Confirm DB Migration upgrade and downgrade tested.
- [ ] Introduces new feature or API
- [ ] Removes existing feature or API
### REVIEWERS
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [incubator-superset] etr2460 commented on a change in pull request
#9142: Support human readable datetime type for PinotDB
Posted by GitBox <gi...@apache.org>.
etr2460 commented on a change in pull request #9142: Support human readable datetime type for PinotDB
URL: https://github.com/apache/incubator-superset/pull/9142#discussion_r379528730
##########
File path: superset/db_engine_specs/pinot.py
##########
@@ -39,18 +40,41 @@ class PinotEngineSpec(BaseEngineSpec): # pylint: disable=abstract-method
"P1Y": "1:YEARS",
}
+ _python_to_java_time_patterns = {
Review comment:
Let's type this as a `Dict[str, str]`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [incubator-superset] fx19880617 commented on a change in pull
request #9142: Support human readable datetime type for PinotDB
Posted by GitBox <gi...@apache.org>.
fx19880617 commented on a change in pull request #9142: Support human readable datetime type for PinotDB
URL: https://github.com/apache/incubator-superset/pull/9142#discussion_r379297153
##########
File path: superset/db_engine_specs/pinot.py
##########
@@ -44,13 +45,32 @@ def get_timestamp_expr(
cls, col: ColumnClause, pdf: Optional[str], time_grain: Optional[str]
) -> TimestampExpression:
is_epoch = pdf in ("epoch_s", "epoch_ms")
- if not is_epoch:
- raise NotImplementedError("Pinot currently only supports epochs")
+
# The DATETIMECONVERT pinot udf is documented at
# Per https://github.com/apache/incubator-pinot/wiki/dateTimeConvert-UDF
# We are not really converting any time units, just bucketing them.
- seconds_or_ms = "MILLISECONDS" if pdf == "epoch_ms" else "SECONDS"
- tf = f"1:{seconds_or_ms}:EPOCH"
+ tf = ""
+ if not is_epoch:
+ try:
+ today = datetime.datetime.today()
+ today.strftime(str(pdf))
+ except ValueError:
+ raise NotImplementedError(
+ "Pinot currently doesn't support date formate: " + str(pdf)
Review comment:
You are correct, now we support datetime format, so will raise a ValueError here.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [incubator-superset] villebro commented on a change in pull request
#9142: Support human readable datetime type for PinotDB
Posted by GitBox <gi...@apache.org>.
villebro commented on a change in pull request #9142: Support human readable datetime type for PinotDB
URL: https://github.com/apache/incubator-superset/pull/9142#discussion_r379497892
##########
File path: superset/db_engine_specs/pinot.py
##########
@@ -39,18 +40,41 @@ class PinotEngineSpec(BaseEngineSpec): # pylint: disable=abstract-method
"P1Y": "1:YEARS",
}
+ _python_to_java_time_patterns = {
+ "%Y": "yyyy",
+ "%m": "MM",
+ "%d": "dd",
+ "%H": "HH",
+ "%M": "mm",
+ "%S": "ss",
+ }
+
@classmethod
def get_timestamp_expr(
cls, col: ColumnClause, pdf: Optional[str], time_grain: Optional[str]
) -> TimestampExpression:
is_epoch = pdf in ("epoch_s", "epoch_ms")
- if not is_epoch:
- raise NotImplementedError("Pinot currently only supports epochs")
+
# The DATETIMECONVERT pinot udf is documented at
# Per https://github.com/apache/incubator-pinot/wiki/dateTimeConvert-UDF
# We are not really converting any time units, just bucketing them.
- seconds_or_ms = "MILLISECONDS" if pdf == "epoch_ms" else "SECONDS"
- tf = f"1:{seconds_or_ms}:EPOCH"
+ tf = ""
+ if not is_epoch:
+ try:
+ today = datetime.datetime.today()
+ today.strftime(str(pdf))
+ except ValueError:
+ raise ValueError(f"Invalid column datetime format:{str(pdf)}")
+ java_pdf = str(pdf)
Review comment:
very nitty: the p should not be here, perhaps call write full name, i.e. `java_date_format` or similar
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [incubator-superset] codecov-io commented on issue #9142: Support
human readable datetime type for PinotDB
Posted by GitBox <gi...@apache.org>.
codecov-io commented on issue #9142: Support human readable datetime type for PinotDB
URL: https://github.com/apache/incubator-superset/pull/9142#issuecomment-586053520
# [Codecov](https://codecov.io/gh/apache/incubator-superset/pull/9142?src=pr&el=h1) Report
> Merging [#9142](https://codecov.io/gh/apache/incubator-superset/pull/9142?src=pr&el=desc) into [master](https://codecov.io/gh/apache/incubator-superset/commit/a005e4568f14f32be23c317dc484b904ddc71dd9?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/incubator-superset/pull/9142?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #9142 +/- ##
======================================
Coverage 59.1% 59.1%
======================================
Files 372 372
Lines 11922 11922
Branches 2919 2919
======================================
Hits 7046 7046
Misses 4694 4694
Partials 182 182
```
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-superset/pull/9142?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/incubator-superset/pull/9142?src=pr&el=footer). Last update [a005e45...466384f](https://codecov.io/gh/apache/incubator-superset/pull/9142?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [incubator-superset] fx19880617 commented on a change in pull
request #9142: Support human readable datetime type for PinotDB
Posted by GitBox <gi...@apache.org>.
fx19880617 commented on a change in pull request #9142: Support human readable datetime type for PinotDB
URL: https://github.com/apache/incubator-superset/pull/9142#discussion_r379297196
##########
File path: superset/db_engine_specs/pinot.py
##########
@@ -44,13 +45,32 @@ def get_timestamp_expr(
cls, col: ColumnClause, pdf: Optional[str], time_grain: Optional[str]
) -> TimestampExpression:
is_epoch = pdf in ("epoch_s", "epoch_ms")
- if not is_epoch:
- raise NotImplementedError("Pinot currently only supports epochs")
+
# The DATETIMECONVERT pinot udf is documented at
# Per https://github.com/apache/incubator-pinot/wiki/dateTimeConvert-UDF
# We are not really converting any time units, just bucketing them.
- seconds_or_ms = "MILLISECONDS" if pdf == "epoch_ms" else "SECONDS"
- tf = f"1:{seconds_or_ms}:EPOCH"
+ tf = ""
+ if not is_epoch:
+ try:
+ today = datetime.datetime.today()
+ today.strftime(str(pdf))
+ except ValueError:
+ raise NotImplementedError(
+ "Pinot currently doesn't support date formate: " + str(pdf)
+ )
+ java_pdf = (
+ str(pdf)
+ .replace("%Y", "yyyy")
+ .replace("%m", "MM")
+ .replace("%d", "dd")
+ .replace("%H", "HH")
+ .replace("%M", "mm")
+ .replace("%S", "ss")
+ )
Review comment:
done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [incubator-superset] codecov-io edited a comment on issue #9142:
Support human readable datetime type for PinotDB
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #9142: Support human readable datetime type for PinotDB
URL: https://github.com/apache/incubator-superset/pull/9142#issuecomment-586053520
# [Codecov](https://codecov.io/gh/apache/incubator-superset/pull/9142?src=pr&el=h1) Report
> Merging [#9142](https://codecov.io/gh/apache/incubator-superset/pull/9142?src=pr&el=desc) into [master](https://codecov.io/gh/apache/incubator-superset/commit/a005e4568f14f32be23c317dc484b904ddc71dd9?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/incubator-superset/pull/9142?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #9142 +/- ##
======================================
Coverage 59.1% 59.1%
======================================
Files 372 372
Lines 11922 11922
Branches 2919 2919
======================================
Hits 7046 7046
Misses 4694 4694
Partials 182 182
```
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-superset/pull/9142?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/incubator-superset/pull/9142?src=pr&el=footer). Last update [a005e45...eb76d96](https://codecov.io/gh/apache/incubator-superset/pull/9142?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [incubator-superset] villebro commented on a change in pull request
#9142: Support human readable datetime type for PinotDB
Posted by GitBox <gi...@apache.org>.
villebro commented on a change in pull request #9142: Support human readable datetime type for PinotDB
URL: https://github.com/apache/incubator-superset/pull/9142#discussion_r379275786
##########
File path: superset/db_engine_specs/pinot.py
##########
@@ -44,13 +45,32 @@ def get_timestamp_expr(
cls, col: ColumnClause, pdf: Optional[str], time_grain: Optional[str]
) -> TimestampExpression:
is_epoch = pdf in ("epoch_s", "epoch_ms")
- if not is_epoch:
- raise NotImplementedError("Pinot currently only supports epochs")
+
# The DATETIMECONVERT pinot udf is documented at
# Per https://github.com/apache/incubator-pinot/wiki/dateTimeConvert-UDF
# We are not really converting any time units, just bucketing them.
- seconds_or_ms = "MILLISECONDS" if pdf == "epoch_ms" else "SECONDS"
- tf = f"1:{seconds_or_ms}:EPOCH"
+ tf = ""
+ if not is_epoch:
+ try:
+ today = datetime.datetime.today()
+ today.strftime(str(pdf))
+ except ValueError:
+ raise NotImplementedError(
+ "Pinot currently doesn't support date formate: " + str(pdf)
+ )
+ java_pdf = (
+ str(pdf)
+ .replace("%Y", "yyyy")
+ .replace("%m", "MM")
+ .replace("%d", "dd")
+ .replace("%H", "HH")
+ .replace("%M", "mm")
+ .replace("%S", "ss")
+ )
Review comment:
I think this would be cleaner as a dict mapping python format to java format, and doing a for loop with the `.replace(python_format, java_format)` on that.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [incubator-superset] codecov-io edited a comment on issue #9142:
Support human readable datetime type for PinotDB
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #9142: Support human readable datetime type for PinotDB
URL: https://github.com/apache/incubator-superset/pull/9142#issuecomment-586053520
# [Codecov](https://codecov.io/gh/apache/incubator-superset/pull/9142?src=pr&el=h1) Report
> Merging [#9142](https://codecov.io/gh/apache/incubator-superset/pull/9142?src=pr&el=desc) into [master](https://codecov.io/gh/apache/incubator-superset/commit/a005e4568f14f32be23c317dc484b904ddc71dd9?src=pr&el=desc) will **decrease** coverage by `0.03%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/incubator-superset/pull/9142?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #9142 +/- ##
==========================================
- Coverage 59.1% 59.06% -0.04%
==========================================
Files 372 372
Lines 11922 11922
Branches 2919 2919
==========================================
- Hits 7046 7042 -4
- Misses 4694 4698 +4
Partials 182 182
```
| [Impacted Files](https://codecov.io/gh/apache/incubator-superset/pull/9142?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [superset-frontend/src/components/Loading.jsx](https://codecov.io/gh/apache/incubator-superset/pull/9142/diff?src=pr&el=tree#diff-c3VwZXJzZXQtZnJvbnRlbmQvc3JjL2NvbXBvbmVudHMvTG9hZGluZy5qc3g=) | `55.55% <0%> (-44.45%)` | :arrow_down: |
| [...frontend/src/dashboard/components/MissingChart.jsx](https://codecov.io/gh/apache/incubator-superset/pull/9142/diff?src=pr&el=tree#diff-c3VwZXJzZXQtZnJvbnRlbmQvc3JjL2Rhc2hib2FyZC9jb21wb25lbnRzL01pc3NpbmdDaGFydC5qc3g=) | `100% <0%> (ø)` | :arrow_up: |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-superset/pull/9142?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/incubator-superset/pull/9142?src=pr&el=footer). Last update [a005e45...5ad52a2](https://codecov.io/gh/apache/incubator-superset/pull/9142?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [incubator-superset] villebro commented on a change in pull request
#9142: Support human readable datetime type for PinotDB
Posted by GitBox <gi...@apache.org>.
villebro commented on a change in pull request #9142: Support human readable datetime type for PinotDB
URL: https://github.com/apache/incubator-superset/pull/9142#discussion_r379276374
##########
File path: superset/db_engine_specs/pinot.py
##########
@@ -44,13 +45,32 @@ def get_timestamp_expr(
cls, col: ColumnClause, pdf: Optional[str], time_grain: Optional[str]
) -> TimestampExpression:
is_epoch = pdf in ("epoch_s", "epoch_ms")
- if not is_epoch:
- raise NotImplementedError("Pinot currently only supports epochs")
+
# The DATETIMECONVERT pinot udf is documented at
# Per https://github.com/apache/incubator-pinot/wiki/dateTimeConvert-UDF
# We are not really converting any time units, just bucketing them.
- seconds_or_ms = "MILLISECONDS" if pdf == "epoch_ms" else "SECONDS"
- tf = f"1:{seconds_or_ms}:EPOCH"
+ tf = ""
+ if not is_epoch:
+ try:
+ today = datetime.datetime.today()
+ today.strftime(str(pdf))
+ except ValueError:
+ raise NotImplementedError(
+ "Pinot currently doesn't support date formate: " + str(pdf)
Review comment:
Typo (formate -> format) + please use f-string. Also, the error message indicates that the error is specific to Pinot; it is my understanding that `ValueError` in this case will be thrown if it is not a valid python date format.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [incubator-superset] villebro merged pull request #9142: Support
human readable datetime type for PinotDB
Posted by GitBox <gi...@apache.org>.
villebro merged pull request #9142: Support human readable datetime type for PinotDB
URL: https://github.com/apache/incubator-superset/pull/9142
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org