You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@superset.apache.org by "hhhonzik (via GitHub)" <gi...@apache.org> on 2023/08/10 15:02:13 UTC
[GitHub] [superset] hhhonzik opened a new pull request, #24949: fix(opensearch): fix opensearch time conversion
hhhonzik opened a new pull request, #24949:
URL: https://github.com/apache/superset/pull/24949
### SUMMARY
Fixes dashboards with time range and opensearch
### BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF
Before we generated datetime string as `2023-08-03T00:00:00`. Opensearch (same as Elastic) accepts `2023-08-03 00:00:00` - separator being ` `.
### TESTING INSTRUCTIONS
1. Connect Opensearch with superset
2. Create time range filter on a dashboard with Opensearch dataset
### ADDITIONAL INFORMATION
- [ ] Has associated issue:
- [ ] Required feature flags:
- [ ] Changes UI
- [ ] Includes DB Migration (follow approval process in [SIP-59](https://github.com/apache/superset/issues/13351))
- [ ] Migration is atomic, supports rollback & is backwards-compatible
- [ ] Confirm DB migration upgrade and downgrade tested
- [ ] Runtime estimates and downtime expectations provided
- [ ] Introduces new feature or API
- [ ] Removes existing feature or API
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(opensearch): fix opensearch time conversion [superset]
Posted by "rusackas (via GitHub)" <gi...@apache.org>.
rusackas closed pull request #24949: fix(opensearch): fix opensearch time conversion
URL: https://github.com/apache/superset/pull/24949
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(opensearch): fix opensearch time conversion [superset]
Posted by "rusackas (via GitHub)" <gi...@apache.org>.
rusackas commented on PR #24949:
URL: https://github.com/apache/superset/pull/24949#issuecomment-3643353409
This PR hasn't been updated in a long time and the technical questions haven't been addressed. We'll mark it as draft, but please set it as 'ready for review' when it's been brought up to speed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(opensearch): fix opensearch time conversion [superset]
Posted by "rusackas (via GitHub)" <gi...@apache.org>.
rusackas commented on PR #24949:
URL: https://github.com/apache/superset/pull/24949#issuecomment-4792547299
Rebased this for good measure, but the old question about whether this affects the Elasticsearch integration never got answered. @hhhonzik are you still around to confirm, or shall I adopt it and verify on my end?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(opensearch): fix opensearch time conversion [superset]
Posted by "rusackas (via GitHub)" <gi...@apache.org>.
rusackas commented on PR #24949:
URL: https://github.com/apache/superset/pull/24949#issuecomment-5028849635
Closing since the OpenSearch/Elasticsearch divergence question never got answered and this has sat untouched for years. Happy to reopen if someone comes back with a clear answer on whether OpenSearch needs its own engine spec.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] john-bodley commented on a diff in pull request #24949: fix(opensearch): fix opensearch time conversion
Posted by "john-bodley (via GitHub)" <gi...@apache.org>.
john-bodley commented on code in PR #24949:
URL: https://github.com/apache/superset/pull/24949#discussion_r1290432021
##########
superset/db_engine_specs/elasticsearch.py:
##########
@@ -123,7 +123,7 @@ def convert_dttm(
sqla_type = cls.get_sqla_column_type(target_type)
if isinstance(sqla_type, types.DateTime):
- return f"""'{dttm.isoformat(timespec="seconds")}'"""
+ return f"""'{dttm.isoformat(sep=" ", timespec="seconds")}'"""
Review Comment:
@hhhonzik would you mind adding some unit tests for this? You can find examples of these with the other DB engine specs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(opensearch): fix opensearch time conversion [superset]
Posted by "rusackas (via GitHub)" <gi...@apache.org>.
rusackas commented on PR #24949:
URL: https://github.com/apache/superset/pull/24949#issuecomment-2819896615
We haven't heard back on the last question here in quite a while. Any interest/bandwidth to follow through on this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(opensearch): fix opensearch time conversion [superset]
Posted by "rusackas (via GitHub)" <gi...@apache.org>.
rusackas commented on PR #24949:
URL: https://github.com/apache/superset/pull/24949#issuecomment-2297535872
This looks OK to me, but I'm no ElasticSearch expert. Curious if @villebro @mistercrunch @betodealmeida have any idea if this would affect any downstream parts of the workflow, or present any other surprise incompatibilities. If not, this has been open for quite a while and still looks easily mergeable :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(opensearch): fix opensearch time conversion [superset]
Posted by "mistercrunch (via GitHub)" <gi...@apache.org>.
mistercrunch commented on PR #24949:
URL: https://github.com/apache/superset/pull/24949#issuecomment-2297667326
Looks like opensearch is a fork of elasticsearch, are we sure this doesn't break the elasticsearch integration? Also the PR will need to be rebased for it to pass the new required checks on the main branch. I'm guessing if you use opensearch you may not have the ability to test against elasticsearch (?) I don't know of any of the core maintainers using elasticsearch, though @dpgaspar might know of people in the community that use the driver (?)
If the fork is diverging from elasticsearch, should we create a separate db_engine_spec/dialect for opensearch? That db_engine_spec could inherit from elasticsearch but overrides the methods where it deviates.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(opensearch): fix opensearch time conversion [superset]
Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #24949:
URL: https://github.com/apache/superset/pull/24949#issuecomment-4255964616
## [Codecov](https://app.codecov.io/gh/apache/superset/pull/24949?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `0%` with `1 line` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 63.73%. Comparing base ([`76a2096`](https://app.codecov.io/gh/apache/superset/commit/76a209663d5a164991206a5b9901fcd74d8dc788?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`ea6849a`](https://app.codecov.io/gh/apache/superset/commit/ea6849a401c223a948af29136cebf04bde25be25?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
| [Files with missing lines](https://app.codecov.io/gh/apache/superset/pull/24949?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [superset/db\_engine\_specs/elasticsearch.py](https://app.codecov.io/gh/apache/superset/pull/24949?src=pr&el=tree&filepath=superset%2Fdb_engine_specs%2Felasticsearch.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c3VwZXJzZXQvZGJfZW5naW5lX3NwZWNzL2VsYXN0aWNzZWFyY2gucHk=) | 0.00% | [1 Missing :warning: ](https://app.codecov.io/gh/apache/superset/pull/24949?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
:x: Your project check has failed because the head coverage (99.79%) is below the target coverage (100.00%). You can increase the head coverage or adjust the [target](https://docs.codecov.com/docs/commit-status#target) coverage.
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #24949 +/- ##
==========================================
+ Coverage 55.47% 63.73% +8.26%
==========================================
Files 2557 2557
Lines 132981 132981
Branches 30888 30888
==========================================
+ Hits 73770 84759 +10989
+ Misses 58559 46713 -11846
- Partials 652 1509 +857
```
| [Flag](https://app.codecov.io/gh/apache/superset/pull/24949/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [hive](https://app.codecov.io/gh/apache/superset/pull/24949/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `39.95% <0.00%> (ø)` | |
| [presto](https://app.codecov.io/gh/apache/superset/pull/24949/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `41.74% <0.00%> (ø)` | |
| [python](https://app.codecov.io/gh/apache/superset/pull/24949/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `60.45% <0.00%> (+18.63%)` | :arrow_up: |
| [sqlite](https://app.codecov.io/gh/apache/superset/pull/24949/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `60.32% <0.00%> (?)` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
</details>
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/superset/pull/24949?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] hhhonzik commented on a diff in pull request #24949: fix(opensearch): fix opensearch time conversion
Posted by "hhhonzik (via GitHub)" <gi...@apache.org>.
hhhonzik commented on code in PR #24949:
URL: https://github.com/apache/superset/pull/24949#discussion_r1291172369
##########
superset/db_engine_specs/elasticsearch.py:
##########
@@ -123,7 +123,7 @@ def convert_dttm(
sqla_type = cls.get_sqla_column_type(target_type)
if isinstance(sqla_type, types.DateTime):
- return f"""'{dttm.isoformat(timespec="seconds")}'"""
+ return f"""'{dttm.isoformat(sep=" ", timespec="seconds")}'"""
Review Comment:
@john-bodley Sorry i ignored them in the first place. Turns out there is a test for this and i didn't update it.
##########
superset/db_engine_specs/elasticsearch.py:
##########
@@ -123,7 +123,7 @@ def convert_dttm(
sqla_type = cls.get_sqla_column_type(target_type)
if isinstance(sqla_type, types.DateTime):
- return f"""'{dttm.isoformat(timespec="seconds")}'"""
+ return f"""'{dttm.isoformat(sep=" ", timespec="seconds")}'"""
Review Comment:
@john-bodley Sorry i ignored them in the first place. Turns out there is a test for this and i didn't update it. All fixed now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(opensearch): fix opensearch time conversion [superset]
Posted by "rusackas (via GitHub)" <gi...@apache.org>.
rusackas commented on PR #24949:
URL: https://github.com/apache/superset/pull/24949#issuecomment-4255889052
@hhhonzik let us know if you want to complete this, otherwise we may close it due to inactivity.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(opensearch): fix opensearch time conversion [superset]
Posted by "rusackas (via GitHub)" <gi...@apache.org>.
rusackas commented on PR #24949:
URL: https://github.com/apache/superset/pull/24949#issuecomment-4724861300
Rebased for good measure, but may eventually be closed due to inactivity, patched up if I/we have time, or if someone wants to adopt this, say the word!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org