You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@superset.apache.org by GitBox <gi...@apache.org> on 2019/12/29 11:38:00 UTC
[GitHub] [incubator-superset] villebro opened a new pull request #8901:
[WIP] Add sql query to cache_key
villebro opened a new pull request #8901: [WIP] Add sql query to cache_key
URL: https://github.com/apache/incubator-superset/pull/8901
### CATEGORY
Choose one
- [x] Bug Fix
- [ ] Enhancement (new features, refinement)
- [ ] Refactor
- [ ] Add tests
- [ ] Build / Development Environment
- [ ] Documentation
### SUMMARY
When changing the SQL query in a datasource, the cache key doesn't reflect this change, as only the datasource id is added to the cache key. This PR adds the `sql` property of the `BaseDatasource` to the dict on which `cache_key` is based.
While this fixes the problem for SQL "views", this doesn't address other potential changes in the datasource, e.g. changes in expressions. An alternative approach would be to add the `changed_on` value from the datasource, which would change the cache key every time the datasource is updated. By doing so we would not need to add all properties that need change checking from the datasource to the cache key; however, it might result in unnecessary cache misses if minor changes to the datasource are done regularly.
### TEST PLAN
Tested locally by replicating the problem in the issue, and ensuring that the bug goes away with this diff. Unit tests to be added once we decide how to handle change tracking of datasource properties.
### ADDITIONAL INFORMATION
<!--- Check any relevant boxes with "x" -->
<!--- HINT: Include "Fixes #nnn" if you are fixing an existing issue -->
- [ ] Has associated issue: closes #8898
- [ ] Changes UI
- [ ] Requires DB Migration.
- [ ] Confirm DB Migration upgrade and downgrade tested.
- [ ] Introduces new feature or API
- [ ] Removes existing feature or API
### REVIEWERS
@durchgedreht
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [incubator-superset] codecov-io commented on issue #8901: [WIP] Add
sql query to cache_key
Posted by GitBox <gi...@apache.org>.
codecov-io commented on issue #8901: [WIP] Add sql query to cache_key
URL: https://github.com/apache/incubator-superset/pull/8901#issuecomment-569497895
# [Codecov](https://codecov.io/gh/apache/incubator-superset/pull/8901?src=pr&el=h1) Report
> Merging [#8901](https://codecov.io/gh/apache/incubator-superset/pull/8901?src=pr&el=desc) into [master](https://codecov.io/gh/apache/incubator-superset/commit/191aca1fb01b797dae21657021d8fcd1e836c40a?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/incubator-superset/pull/8901?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #8901 +/- ##
=======================================
Coverage 58.97% 58.97%
=======================================
Files 359 359
Lines 11333 11333
Branches 2787 2787
=======================================
Hits 6684 6684
Misses 4471 4471
Partials 178 178
```
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-superset/pull/8901?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/incubator-superset/pull/8901?src=pr&el=footer). Last update [191aca1...7713e08](https://codecov.io/gh/apache/incubator-superset/pull/8901?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [incubator-superset] villebro commented on issue #8901: [WIP] Add
sql query to cache_key
Posted by GitBox <gi...@apache.org>.
villebro commented on issue #8901: [WIP] Add sql query to cache_key
URL: https://github.com/apache/incubator-superset/pull/8901#issuecomment-569875453
Thanks for chiming in @willbarrett ; personally I also started leaning towards using `changed_on` instead of all potentially interesting attributes. I'll update the PR to reflect this. It would be valuable to hear if large deployments of Superset are using large scale automated metadata refreshes of existing datasources, and if this might cause unnecessary cache misses. @john-bodley @etr2460 @betodealmeida any feedback would be valuable to understand if this would be a blocker in your deployments?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [incubator-superset] villebro commented on a change in pull request
#8901: Add datasource.changed_on to cache_key
Posted by GitBox <gi...@apache.org>.
villebro commented on a change in pull request #8901: Add datasource.changed_on to cache_key
URL: https://github.com/apache/incubator-superset/pull/8901#discussion_r365982640
##########
File path: superset/common/query_object.py
##########
@@ -83,10 +83,7 @@ def __init__(
# Temporal solution for backward compatability issue
# due the new format of non-ad-hoc metric.
- self.metrics = [
- metric if "expressionType" in metric else metric["label"] # type: ignore
- for metric in metrics
- ]
+ self.metrics = [utils.get_metric_name(metric) for metric in metrics]
Review comment:
During testing I noticed that the existing logic was incomplete; `utils.get_metric_name` on the other hand is used elsewhere and handles all metric types correctly (legacy and ad-hoc).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [incubator-superset] villebro commented on issue #8901: fix: add
datasource.changed_on to cache_key
Posted by GitBox <gi...@apache.org>.
villebro commented on issue #8901: fix: add datasource.changed_on to cache_key
URL: https://github.com/apache/incubator-superset/pull/8901#issuecomment-573836461
@john-bodley oh absolutely, good idea. Will add a note in UPDATING.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [incubator-superset] villebro commented on a change in pull request
#8901: Add datasource.changed_on to cache_key
Posted by GitBox <gi...@apache.org>.
villebro commented on a change in pull request #8901: Add datasource.changed_on to cache_key
URL: https://github.com/apache/incubator-superset/pull/8901#discussion_r365982027
##########
File path: superset/common/query_context.py
##########
@@ -157,20 +157,25 @@ def cache_timeout(self) -> int:
return self.datasource.database.cache_timeout
return config["CACHE_DEFAULT_TIMEOUT"]
- def get_df_payload( # pylint: disable=too-many-locals,too-many-statements
- self, query_obj: QueryObject, **kwargs
- ) -> Dict[str, Any]:
- """Handles caching around the df payload retrieval"""
+ def cache_key(self, query_obj: QueryObject, **kwargs) -> Optional[str]:
extra_cache_keys = self.datasource.get_extra_cache_keys(query_obj.to_dict())
cache_key = (
query_obj.cache_key(
datasource=self.datasource.uid,
extra_cache_keys=extra_cache_keys,
+ changed_on=self.datasource.changed_on,
**kwargs
)
if query_obj
else None
)
+ return cache_key
Review comment:
Cache key calculation logic was broken into a method of its own to enable easier unit testing.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [incubator-superset] villebro commented on a change in pull request
#8901: fix: add datasource.changed_on to cache_key
Posted by GitBox <gi...@apache.org>.
villebro commented on a change in pull request #8901: fix: add datasource.changed_on to cache_key
URL: https://github.com/apache/incubator-superset/pull/8901#discussion_r372583519
##########
File path: superset/common/query_context.py
##########
@@ -157,20 +157,25 @@ def cache_timeout(self) -> int:
return self.datasource.database.cache_timeout
return config["CACHE_DEFAULT_TIMEOUT"]
- def get_df_payload( # pylint: disable=too-many-locals,too-many-statements
- self, query_obj: QueryObject, **kwargs
- ) -> Dict[str, Any]:
- """Handles caching around the df payload retrieval"""
+ def cache_key(self, query_obj: QueryObject, **kwargs) -> Optional[str]:
extra_cache_keys = self.datasource.get_extra_cache_keys(query_obj.to_dict())
cache_key = (
query_obj.cache_key(
datasource=self.datasource.uid,
extra_cache_keys=extra_cache_keys,
+ changed_on=self.datasource.changed_on,
Review comment:
It is my understanding that `changed_on` is updated every time any change is applied, i.e. does not check if only relevant metrics/expressions have changed. While this can cause unnecessary cache misses, I felt the added complexity of checking only for relevant changes was not warranted unless the ultimately proposed simpler solution was found to be too generic (I tried to convey this in the unit test which only changed the description). If this does cause unacceptable amounts of cache misses I think we need to revisit this logic; until then I personally think this is a good compromise. However, I'm happy to open up the discussion again if there are opinions to the contrary.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [incubator-superset] villebro commented on a change in pull request
#8901: Add datasource.changed_on to cache_key
Posted by GitBox <gi...@apache.org>.
villebro commented on a change in pull request #8901: Add datasource.changed_on to cache_key
URL: https://github.com/apache/incubator-superset/pull/8901#discussion_r365984496
##########
File path: tests/core_tests.py
##########
@@ -110,30 +128,40 @@ def test_cache_key(self):
qobj["groupby"] = []
self.assertNotEqual(cache_key, viz.cache_key(qobj))
+ def test_cache_key_changes_when_datasource_is_updated(self):
+ qc_dict = self._get_query_context_dict()
+
+ # construct baseline cache_key
+ query_context = QueryContext(**qc_dict)
+ query_object = query_context.queries[0]
+ cache_key_original = query_context.cache_key(query_object)
+
+ # make temporary change and revert it to refresh the changed_on property
+ datasource = ConnectorRegistry.get_datasource(
+ datasource_type=qc_dict["datasource"]["type"],
+ datasource_id=qc_dict["datasource"]["id"],
+ session=db.session,
+ )
+ description_original = datasource.description
+ datasource.description = "temporary description"
+ db.session.commit()
+ datasource.description = description_original
+ db.session.commit()
+
+ # create new QueryContext with unchanged attributes and extract new cache_key
+ query_context = QueryContext(**qc_dict)
+ query_object = query_context.queries[0]
+ cache_key_new = query_context.cache_key(query_object)
+
+ # the new cache_key should be different due to updated datasource
+ self.assertNotEqual(cache_key_original, cache_key_new)
+
def test_api_v1_query_endpoint(self):
self.login(username="admin")
- slc = self.get_slice("Girl Name Cloud", db.session)
- form_data = slc.form_data
- data = json.dumps(
- {
- "datasource": {"id": slc.datasource_id, "type": slc.datasource_type},
- "queries": [
- {
- "granularity": "ds",
- "groupby": ["name"],
- "metrics": ["sum__num"],
- "filters": [],
- "time_range": "{} : {}".format(
- form_data.get("since"), form_data.get("until")
- ),
- "limit": 100,
- }
- ],
- }
- )
- # TODO: update once get_data is implemented for QueryObject
- with self.assertRaises(Exception):
- self.get_resp("/api/v1/query/", {"query_context": data})
+ qc_dict = self._get_query_context_dict()
+ data = json.dumps(qc_dict)
+ resp = json.loads(self.get_resp("/api/v1/query/", {"query_context": data}))
+ self.assertEqual(resp[0]["rowcount"], 100)
Review comment:
The old unit test seemed to be incomplete, so fixed a few bugs in the body (`limit` -> `row_limit` and removed `time_range`) to make it work properly.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [incubator-superset] villebro commented on issue #8901: [WIP] Add
sql query to cache_key
Posted by GitBox <gi...@apache.org>.
villebro commented on issue #8901: [WIP] Add sql query to cache_key
URL: https://github.com/apache/incubator-superset/pull/8901#issuecomment-572222051
As there doesn't seem to be any objections to adding `changed_on` to `cache_key`, I'll try to finalize this in the coming days.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [incubator-superset] villebro commented on issue #8901: Add
datasource.changed_on to cache_key
Posted by GitBox <gi...@apache.org>.
villebro commented on issue #8901: Add datasource.changed_on to cache_key
URL: https://github.com/apache/incubator-superset/pull/8901#issuecomment-573829094
This is ready for review.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [incubator-superset] john-bodley commented on issue #8901: fix: add
datasource.changed_on to cache_key
Posted by GitBox <gi...@apache.org>.
john-bodley commented on issue #8901: fix: add datasource.changed_on to cache_key
URL: https://github.com/apache/incubator-superset/pull/8901#issuecomment-573834733
@villebro do you think there is merit in adding a line item to UPDATING.md to mention that this PR will invalidate the entire cache given that the hashing logic has changed?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [incubator-superset] john-bodley edited a comment on issue #8901:
fix: add datasource.changed_on to cache_key
Posted by GitBox <gi...@apache.org>.
john-bodley edited a comment on issue #8901: fix: add datasource.changed_on to cache_key
URL: https://github.com/apache/incubator-superset/pull/8901#issuecomment-573834733
@villebro do you think there is merit in adding a line item to UPDATING.md to mention that this PR will invalidate the entire cache given that the keys used for hashing have changed?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [incubator-superset] mistercrunch merged pull request #8901: fix:
add datasource.changed_on to cache_key
Posted by GitBox <gi...@apache.org>.
mistercrunch merged pull request #8901: fix: add datasource.changed_on to cache_key
URL: https://github.com/apache/incubator-superset/pull/8901
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [incubator-superset] villebro commented on issue #8901: [WIP] Add
sql query to cache_key
Posted by GitBox <gi...@apache.org>.
villebro commented on issue #8901: [WIP] Add sql query to cache_key
URL: https://github.com/apache/incubator-superset/pull/8901#issuecomment-573532448
FYI working on this fix+unit tests surfaced some other issues I will be addressing with this PR, i.e. will take slightly longer to complete than anticipated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [incubator-superset] john-bodley commented on a change in pull
request #8901: fix: add datasource.changed_on to cache_key
Posted by GitBox <gi...@apache.org>.
john-bodley commented on a change in pull request #8901: fix: add datasource.changed_on to cache_key
URL: https://github.com/apache/incubator-superset/pull/8901#discussion_r372541748
##########
File path: superset/common/query_context.py
##########
@@ -157,20 +157,25 @@ def cache_timeout(self) -> int:
return self.datasource.database.cache_timeout
return config["CACHE_DEFAULT_TIMEOUT"]
- def get_df_payload( # pylint: disable=too-many-locals,too-many-statements
- self, query_obj: QueryObject, **kwargs
- ) -> Dict[str, Any]:
- """Handles caching around the df payload retrieval"""
+ def cache_key(self, query_obj: QueryObject, **kwargs) -> Optional[str]:
extra_cache_keys = self.datasource.get_extra_cache_keys(query_obj.to_dict())
cache_key = (
query_obj.cache_key(
datasource=self.datasource.uid,
extra_cache_keys=extra_cache_keys,
+ changed_on=self.datasource.changed_on,
Review comment:
@villebro apologies for the late comment. In your PR description you mention that the cache key should be a function on whether the column or metric definitions associated with a datasource are changed.
On line #166 you merely use the datasource `change_on` and thus I was wondering whether a changes to the columns and/or metrics cascade, i.e., trigger an update to the datasource `changed_on`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [incubator-superset] willbarrett commented on issue #8901: [WIP]
Add sql query to cache_key
Posted by GitBox <gi...@apache.org>.
willbarrett commented on issue #8901: [WIP] Add sql query to cache_key
URL: https://github.com/apache/incubator-superset/pull/8901#issuecomment-569827644
Thanks for this @villebro! My recommendation would be to go with the `changed_on` timestamp instead of the `sql` column. As you mention, Datasources define a number of different things that could affect cache behavior or results. Causing cache misses on every Datasource update feels more future-proof, as this logic won't need to change every time Datasource behavior changes. We may cause some unnecessary cache misses, but I think we should lean in the direction of ensuring correctness first and thinking about performance second.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org