You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@superset.apache.org by GitBox <gi...@apache.org> on 2022/10/18 09:39:24 UTC
[GitHub] [superset] GerbenvdHuizen opened a new pull request, #21850: feat(db_engine_specs): Allow columns of dtype datetime to be stored as TIMESTAMP in the Hive schema
GerbenvdHuizen opened a new pull request, #21850:
URL: https://github.com/apache/superset/pull/21850
### SUMMARY
he views for uploading data files (csv, excel etc.) to a database have form options to pass and/or infer `datetime` columns. However, the hive `df_to_sql` method does not support storing `datetime` columns as TIMESTAMPs in the Hive schema. It would be a nice for the `_get_hive_type` functionality to at least support the basic `datetime64[ns]` dtype.
### TESTING INSTRUCTIONS
The following steps require you to be able to upload files to a schema on Superset (`allow_file_upload`) + use Hive as DB engine.
Steps:
1. Upload a csv with an inferable timestamp:
```
test_column,timestamp
something,2022-01-01T01:01:01.000000
```
2. Make sure `parse_dates` is set to the timestamp column in your csv file and `infer_datetime_format` is set to true within the form.
3. Submit the CSV and confirm that the new table has a column with type TIMESTAMP.
### ADDITIONAL INFORMATION
- [ ] Has associated issue:
- [ ] Required feature flags:
- [ ] Changes UI
- [ ] Includes DB Migration (follow approval process in [SIP-59](https://github.com/apache/superset/issues/13351))
- [ ] Migration is atomic, supports rollback & is backwards-compatible
- [ ] Confirm DB migration upgrade and downgrade tested
- [ ] Runtime estimates and downtime expectations provided
- [ ] Introduces new feature or API
- [ ] Removes existing feature or API
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] codecov[bot] commented on pull request #21850: feat(db_engine_specs): Allow uploaded columns of dtype datetime to be stored as TIMESTAMP in the Hive schema
Posted by GitBox <gi...@apache.org>.
codecov[bot] commented on PR #21850:
URL: https://github.com/apache/superset/pull/21850#issuecomment-1294532499
# [Codecov](https://codecov.io/gh/apache/superset/pull/21850?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#21850](https://codecov.io/gh/apache/superset/pull/21850?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (40a5454) into [master](https://codecov.io/gh/apache/superset/commit/8f4415bc831d234965001a476acc0c123448aee9?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (8f4415b) will **decrease** coverage by `10.62%`.
> The diff coverage is `n/a`.
```diff
@@ Coverage Diff @@
## master #21850 +/- ##
===========================================
- Coverage 66.18% 55.56% -10.63%
===========================================
Files 1805 1805
Lines 69066 69066
Branches 7369 7369
===========================================
- Hits 45712 38375 -7337
- Misses 21448 28785 +7337
Partials 1906 1906
```
| Flag | Coverage Δ | |
|---|---|---|
| hive | `52.92% <ø> (ø)` | |
| presto | `52.82% <ø> (ø)` | |
| python | `57.94% <ø> (-21.99%)` | :arrow_down: |
| sqlite | `?` | |
| unit | `51.06% <ø> (ø)` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/superset/pull/21850?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [superset/db\_engine\_specs/hive.py](https://codecov.io/gh/apache/superset/pull/21850/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvZGJfZW5naW5lX3NwZWNzL2hpdmUucHk=) | `64.82% <ø> (-22.53%)` | :arrow_down: |
| [superset/utils/dashboard\_import\_export.py](https://codecov.io/gh/apache/superset/pull/21850/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvdXRpbHMvZGFzaGJvYXJkX2ltcG9ydF9leHBvcnQucHk=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [superset/tags/core.py](https://codecov.io/gh/apache/superset/pull/21850/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvdGFncy9jb3JlLnB5) | `4.54% <0.00%> (-95.46%)` | :arrow_down: |
| [superset/key\_value/commands/update.py](https://codecov.io/gh/apache/superset/pull/21850/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQva2V5X3ZhbHVlL2NvbW1hbmRzL3VwZGF0ZS5weQ==) | `0.00% <0.00%> (-90.91%)` | :arrow_down: |
| [superset/key\_value/commands/delete.py](https://codecov.io/gh/apache/superset/pull/21850/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQva2V5X3ZhbHVlL2NvbW1hbmRzL2RlbGV0ZS5weQ==) | `0.00% <0.00%> (-87.88%)` | :arrow_down: |
| [superset/key\_value/commands/delete\_expired.py](https://codecov.io/gh/apache/superset/pull/21850/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQva2V5X3ZhbHVlL2NvbW1hbmRzL2RlbGV0ZV9leHBpcmVkLnB5) | `0.00% <0.00%> (-84.00%)` | :arrow_down: |
| [superset/dashboards/commands/importers/v0.py](https://codecov.io/gh/apache/superset/pull/21850/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvZGFzaGJvYXJkcy9jb21tYW5kcy9pbXBvcnRlcnMvdjAucHk=) | `15.62% <0.00%> (-76.25%)` | :arrow_down: |
| [superset/reports/commands/execute.py](https://codecov.io/gh/apache/superset/pull/21850/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvcmVwb3J0cy9jb21tYW5kcy9leGVjdXRlLnB5) | `24.04% <0.00%> (-67.95%)` | :arrow_down: |
| [superset/views/datasource/utils.py](https://codecov.io/gh/apache/superset/pull/21850/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvdmlld3MvZGF0YXNvdXJjZS91dGlscy5weQ==) | `26.66% <0.00%> (-66.67%)` | :arrow_down: |
| [superset/datasets/commands/importers/v0.py](https://codecov.io/gh/apache/superset/pull/21850/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvZGF0YXNldHMvY29tbWFuZHMvaW1wb3J0ZXJzL3YwLnB5) | `24.03% <0.00%> (-66.67%)` | :arrow_down: |
| ... and [270 more](https://codecov.io/gh/apache/superset/pull/21850/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
:mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(db_engine_specs): Allow uploaded columns of dtype datetime to be stored as TIMESTAMP in the Hive schema [superset]
Posted by "rusackas (via GitHub)" <gi...@apache.org>.
rusackas closed pull request #21850: feat(db_engine_specs): Allow uploaded columns of dtype datetime to be stored as TIMESTAMP in the Hive schema
URL: https://github.com/apache/superset/pull/21850
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] john-bodley commented on pull request #21850: feat(db_engine_specs): Allow uploaded columns of dtype datetime to be stored as TIMESTAMP in the Hive schema
Posted by GitBox <gi...@apache.org>.
john-bodley commented on PR #21850:
URL: https://github.com/apache/superset/pull/21850#issuecomment-1282793031
Thanks @GerbenvdHuizen for the PR. Would you mind formalizing your testing strategy as either a unit or integration test?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(db_engine_specs): Allow uploaded columns of dtype datetime to be stored as TIMESTAMP in the Hive schema [superset]
Posted by "rusackas (via GitHub)" <gi...@apache.org>.
rusackas commented on PR #21850:
URL: https://github.com/apache/superset/pull/21850#issuecomment-1930373455
@john-bodley @bkyryliuk could one of you give this a look now that it has the requested unit tests? I'll close/reopen to kick-start CI.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] GerbenvdHuizen commented on pull request #21850: feat(db_engine_specs): Allow uploaded columns of dtype datetime to be stored as TIMESTAMP in the Hive schema
Posted by GitBox <gi...@apache.org>.
GerbenvdHuizen commented on PR #21850:
URL: https://github.com/apache/superset/pull/21850#issuecomment-1283981383
@john-bodley Added an integration test for all the schema definition types, since there didn't seem to be any yet.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org