You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@superset.apache.org by GitBox <gi...@apache.org> on 2020/09/02 05:05:09 UTC
[GitHub] [incubator-superset] bkyryliuk opened a new pull request #10761: feat: [WIP] implement cache invalidation api
bkyryliuk opened a new pull request #10761:
URL: https://github.com/apache/incubator-superset/pull/10761
### SUMMARY
<!--- Describe the change below, including rationale and design decisions -->
### BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF
<!--- Skip this if not applicable -->
### TEST PLAN
<!--- What steps should be taken to verify the changes -->
### ADDITIONAL INFORMATION
<!--- Check any relevant boxes with "x" -->
<!--- HINT: Include "Fixes #nnn" if you are fixing an existing issue -->
- [ ] Has associated issue:
- [ ] Changes UI
- [ ] Requires DB Migration.
- [ ] Confirm DB Migration upgrade and downgrade tested.
- [ ] Introduces new feature or API
- [ ] Removes existing feature or API
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [incubator-superset] bkyryliuk commented on a change in pull request #10761: feat: implement cache invalidation api
Posted by GitBox <gi...@apache.org>.
bkyryliuk commented on a change in pull request #10761:
URL: https://github.com/apache/incubator-superset/pull/10761#discussion_r487139370
##########
File path: superset/cachekeys/schemas.py
##########
@@ -0,0 +1,45 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+# RISON/JSON schemas for query parameters
+from marshmallow import fields, Schema, validate
+
+from superset.charts.schemas import (
+ datasource_name_description,
+ datasource_type_description,
+ datasource_uid_description,
+)
+
+
+class Datasource(Schema):
+ database_name = fields.String(description="Datasource name",)
+ datasource_name = fields.String(description=datasource_name_description,)
+ schema = fields.String(description="Datasource schema",)
+ datasource_type = fields.String(
+ description=datasource_type_description,
+ validate=validate.OneOf(choices=("druid", "table", "view")),
+ allow_none=True,
Review comment:
done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [incubator-superset] codecov-commenter edited a comment on pull request #10761: feat: [WIP] implement cache invalidation api
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #10761:
URL: https://github.com/apache/incubator-superset/pull/10761#issuecomment-686252702
# [Codecov](https://codecov.io/gh/apache/incubator-superset/pull/10761?src=pr&el=h1) Report
> Merging [#10761](https://codecov.io/gh/apache/incubator-superset/pull/10761?src=pr&el=desc) into [master](https://codecov.io/gh/apache/incubator-superset/commit/f6858256f496387146c4b535436bedaeceeeee51?el=desc) will **increase** coverage by `0.37%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/incubator-superset/pull/10761?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #10761 +/- ##
==========================================
+ Coverage 61.22% 61.60% +0.37%
==========================================
Files 802 429 -373
Lines 37814 13974 -23840
Branches 3555 3555
==========================================
- Hits 23153 8608 -14545
+ Misses 14475 5180 -9295
Partials 186 186
```
| Flag | Coverage Δ | |
|---|---|---|
| #javascript | `61.60% <ø> (ø)` | |
| #python | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/incubator-superset/pull/10761?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [.../migrations/versions/1a1d627ebd8e\_position\_json.py](https://codecov.io/gh/apache/incubator-superset/pull/10761/diff?src=pr&el=tree#diff-c3VwZXJzZXQvbWlncmF0aW9ucy92ZXJzaW9ucy8xYTFkNjI3ZWJkOGVfcG9zaXRpb25fanNvbi5weQ==) | | |
| [...7bf4f117f9\_adding\_extra\_field\_to\_database\_model.py](https://codecov.io/gh/apache/incubator-superset/pull/10761/diff?src=pr&el=tree#diff-c3VwZXJzZXQvbWlncmF0aW9ucy92ZXJzaW9ucy84NjdiZjRmMTE3ZjlfYWRkaW5nX2V4dHJhX2ZpZWxkX3RvX2RhdGFiYXNlX21vZGVsLnB5) | | |
| [superset/models/sql\_types/presto\_sql\_types.py](https://codecov.io/gh/apache/incubator-superset/pull/10761/diff?src=pr&el=tree#diff-c3VwZXJzZXQvbW9kZWxzL3NxbF90eXBlcy9wcmVzdG9fc3FsX3R5cGVzLnB5) | | |
| [superset/db\_engine\_specs/pinot.py](https://codecov.io/gh/apache/incubator-superset/pull/10761/diff?src=pr&el=tree#diff-c3VwZXJzZXQvZGJfZW5naW5lX3NwZWNzL3Bpbm90LnB5) | | |
| [...rsions/a9c47e2c1547\_add\_impersonate\_user\_to\_dbs.py](https://codecov.io/gh/apache/incubator-superset/pull/10761/diff?src=pr&el=tree#diff-c3VwZXJzZXQvbWlncmF0aW9ucy92ZXJzaW9ucy9hOWM0N2UyYzE1NDdfYWRkX2ltcGVyc29uYXRlX3VzZXJfdG9fZGJzLnB5) | | |
| [superset/stats\_logger.py](https://codecov.io/gh/apache/incubator-superset/pull/10761/diff?src=pr&el=tree#diff-c3VwZXJzZXQvc3RhdHNfbG9nZ2VyLnB5) | | |
| [...rset/migrations/versions/65903709c321\_allow\_dml.py](https://codecov.io/gh/apache/incubator-superset/pull/10761/diff?src=pr&el=tree#diff-c3VwZXJzZXQvbWlncmF0aW9ucy92ZXJzaW9ucy82NTkwMzcwOWMzMjFfYWxsb3dfZG1sLnB5) | | |
| [superset/dashboards/commands/create.py](https://codecov.io/gh/apache/incubator-superset/pull/10761/diff?src=pr&el=tree#diff-c3VwZXJzZXQvZGFzaGJvYXJkcy9jb21tYW5kcy9jcmVhdGUucHk=) | | |
| [superset/models/schedules.py](https://codecov.io/gh/apache/incubator-superset/pull/10761/diff?src=pr&el=tree#diff-c3VwZXJzZXQvbW9kZWxzL3NjaGVkdWxlcy5weQ==) | | |
| [superset/dashboards/commands/delete.py](https://codecov.io/gh/apache/incubator-superset/pull/10761/diff?src=pr&el=tree#diff-c3VwZXJzZXQvZGFzaGJvYXJkcy9jb21tYW5kcy9kZWxldGUucHk=) | | |
| ... and [361 more](https://codecov.io/gh/apache/incubator-superset/pull/10761/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-superset/pull/10761?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/incubator-superset/pull/10761?src=pr&el=footer). Last update [f685825...47eb457](https://codecov.io/gh/apache/incubator-superset/pull/10761?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [incubator-superset] bkyryliuk commented on a change in pull request #10761: feat: implement cache invalidation api
Posted by GitBox <gi...@apache.org>.
bkyryliuk commented on a change in pull request #10761:
URL: https://github.com/apache/incubator-superset/pull/10761#discussion_r486412102
##########
File path: superset/cachekeys/api.py
##########
@@ -0,0 +1,98 @@
+# Licensed to the Apache Software Foundation (ASF) under one
Review comment:
agree, I had the same struggles, definitely open to suggestions.
/api/v1/cache/invalidate could be slightly better
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [incubator-superset] bkyryliuk commented on a change in pull request #10761: feat: implement cache invalidation api
Posted by GitBox <gi...@apache.org>.
bkyryliuk commented on a change in pull request #10761:
URL: https://github.com/apache/incubator-superset/pull/10761#discussion_r487139156
##########
File path: superset/cachekeys/api.py
##########
@@ -0,0 +1,98 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from flask import request, Response
+from flask_appbuilder import expose
+from flask_appbuilder.api import safe
+from flask_appbuilder.models.sqla.interface import SQLAInterface
+from flask_appbuilder.security.decorators import protect
+from jsonschema import ValidationError
+
+from superset.cachekeys.schemas import CacheInvalidationRequestSchema
+from superset.connectors.connector_registry import ConnectorRegistry
+from superset.extensions import cache_manager, db, event_logger
+from superset.models.cache import CacheKey
+from superset.views.base_api import BaseSupersetModelRestApi, statsd_metrics
+
+
+class CacheRestApi(BaseSupersetModelRestApi):
+ datamodel = SQLAInterface(CacheKey)
+ resource_name = "cache"
+ allow_browser_login = True
+ class_permission_name = "CacheRestApi"
+
+ openapi_spec_component_schemas = (CacheInvalidationRequestSchema,)
+
+ @expose("/invalidate", methods=["POST"])
+ @event_logger.log_this
+ @protect()
+ @safe
+ @statsd_metrics
+ def invalidate(self) -> Response:
+ """
+ Takes a list of datasources, finds the associated cache records and
+ invalidates them and removes the database records
+
+ ---
+ post:
+ description: >-
+ Takes a list of datasources, finds the associated cache records and
+ invalidates them and removes the database records
+ requestBody:
+ description: >-
+ A list of datasources uuid or the tuples of database and datasource names
+ required: true
+ content:
+ application/json:
+ schema:
+ $ref: "#/components/schemas/CacheInvalidationRequestSchema"
+ responses:
+ 201:
+ description: cache was successfully invalidated
+ 400:
+ $ref: '#/components/responses/400'
+ 500:
+ $ref: '#/components/responses/500'
+ """
+ try:
+ datasources = CacheInvalidationRequestSchema().load(request.json)
+ except KeyError:
+ return self.response_400(message="Request is incorrect")
+ except ValidationError as error:
+ return self.response_400(message=error.message)
+ datasource_uids = set(datasources.get("datasource_uids", []))
+ for ds in datasources.get("datasources", []):
+ ds_obj = ConnectorRegistry.get_datasource_by_name(
+ session=db.session,
+ datasource_type=ds.get("datasource_type"),
+ datasource_name=ds.get("datasource_name"),
+ schema=ds.get("schema"),
+ database_name=ds.get("database_name"),
+ )
+ if ds_obj:
+ datasource_uids.add(ds_obj.uid)
+
+ cache_keys = (
+ db.session.query(CacheKey)
+ .filter(CacheKey.datasource_uid.in_(datasource_uids))
+ .all()
+ )
+ if cache_keys:
+ cache_manager.cache.delete_many(*[c.cache_key for c in cache_keys])
Review comment:
delete_many doesn't fail, only returns true / false if the keys were deleted. e.g. if the key was not found it will return false. Updated CacheKey logic to delete is a single statement & rollback if fails -> that would be more robust.
deleting case & keeping records is not too bad as there still a need to garbage collect those and cache is not a persistent store.
##########
File path: superset/cachekeys/schemas.py
##########
@@ -0,0 +1,45 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+# RISON/JSON schemas for query parameters
+from marshmallow import fields, Schema, validate
+
+from superset.charts.schemas import (
+ datasource_name_description,
+ datasource_type_description,
+ datasource_uid_description,
+)
+
+
+class Datasource(Schema):
+ database_name = fields.String(description="Datasource name",)
+ datasource_name = fields.String(description=datasource_name_description,)
+ schema = fields.String(description="Datasource schema",)
+ datasource_type = fields.String(
+ description=datasource_type_description,
+ validate=validate.OneOf(choices=("druid", "table", "view")),
+ allow_none=True,
Review comment:
done
##########
File path: superset/cachekeys/api.py
##########
@@ -0,0 +1,98 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from flask import request, Response
+from flask_appbuilder import expose
+from flask_appbuilder.api import safe
+from flask_appbuilder.models.sqla.interface import SQLAInterface
+from flask_appbuilder.security.decorators import protect
+from jsonschema import ValidationError
+
+from superset.cachekeys.schemas import CacheInvalidationRequestSchema
+from superset.connectors.connector_registry import ConnectorRegistry
+from superset.extensions import cache_manager, db, event_logger
+from superset.models.cache import CacheKey
+from superset.views.base_api import BaseSupersetModelRestApi, statsd_metrics
+
+
+class CacheRestApi(BaseSupersetModelRestApi):
+ datamodel = SQLAInterface(CacheKey)
+ resource_name = "cache"
+ allow_browser_login = True
+ class_permission_name = "CacheRestApi"
+
+ openapi_spec_component_schemas = (CacheInvalidationRequestSchema,)
+
+ @expose("/invalidate", methods=["POST"])
+ @event_logger.log_this
+ @protect()
+ @safe
+ @statsd_metrics
+ def invalidate(self) -> Response:
+ """
+ Takes a list of datasources, finds the associated cache records and
+ invalidates them and removes the database records
+
+ ---
+ post:
+ description: >-
+ Takes a list of datasources, finds the associated cache records and
+ invalidates them and removes the database records
+ requestBody:
+ description: >-
+ A list of datasources uuid or the tuples of database and datasource names
+ required: true
+ content:
+ application/json:
+ schema:
+ $ref: "#/components/schemas/CacheInvalidationRequestSchema"
+ responses:
+ 201:
+ description: cache was successfully invalidated
+ 400:
+ $ref: '#/components/responses/400'
+ 500:
+ $ref: '#/components/responses/500'
+ """
+ try:
+ datasources = CacheInvalidationRequestSchema().load(request.json)
+ except KeyError:
+ return self.response_400(message="Request is incorrect")
+ except ValidationError as error:
+ return self.response_400(message=error.message)
+ datasource_uids = set(datasources.get("datasource_uids", []))
+ for ds in datasources.get("datasources", []):
+ ds_obj = ConnectorRegistry.get_datasource_by_name(
+ session=db.session,
+ datasource_type=ds.get("datasource_type"),
+ datasource_name=ds.get("datasource_name"),
+ schema=ds.get("schema"),
+ database_name=ds.get("database_name"),
+ )
+ if ds_obj:
+ datasource_uids.add(ds_obj.uid)
+
+ cache_keys = (
+ db.session.query(CacheKey)
+ .filter(CacheKey.datasource_uid.in_(datasource_uids))
+ .all()
+ )
+ if cache_keys:
+ cache_manager.cache.delete_many(*[c.cache_key for c in cache_keys])
Review comment:
delete_many doesn't fail, only returns true / false if the keys were deleted. e.g. if the key was not found it will return false. Updated CacheKey logic to delete is a single statement & rollback if fails -> that would be more robust.
deleting case & keeping records is not too bad as there still a need to garbage collect those and cache is not a persistent store.
##########
File path: superset/cachekeys/schemas.py
##########
@@ -0,0 +1,45 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+# RISON/JSON schemas for query parameters
+from marshmallow import fields, Schema, validate
+
+from superset.charts.schemas import (
+ datasource_name_description,
+ datasource_type_description,
+ datasource_uid_description,
+)
+
+
+class Datasource(Schema):
+ database_name = fields.String(description="Datasource name",)
+ datasource_name = fields.String(description=datasource_name_description,)
+ schema = fields.String(description="Datasource schema",)
+ datasource_type = fields.String(
+ description=datasource_type_description,
+ validate=validate.OneOf(choices=("druid", "table", "view")),
+ allow_none=True,
Review comment:
done
##########
File path: superset/cachekeys/api.py
##########
@@ -0,0 +1,98 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from flask import request, Response
+from flask_appbuilder import expose
+from flask_appbuilder.api import safe
+from flask_appbuilder.models.sqla.interface import SQLAInterface
+from flask_appbuilder.security.decorators import protect
+from jsonschema import ValidationError
+
+from superset.cachekeys.schemas import CacheInvalidationRequestSchema
+from superset.connectors.connector_registry import ConnectorRegistry
+from superset.extensions import cache_manager, db, event_logger
+from superset.models.cache import CacheKey
+from superset.views.base_api import BaseSupersetModelRestApi, statsd_metrics
+
+
+class CacheRestApi(BaseSupersetModelRestApi):
+ datamodel = SQLAInterface(CacheKey)
+ resource_name = "cache"
+ allow_browser_login = True
+ class_permission_name = "CacheRestApi"
+
+ openapi_spec_component_schemas = (CacheInvalidationRequestSchema,)
+
+ @expose("/invalidate", methods=["POST"])
+ @event_logger.log_this
+ @protect()
+ @safe
+ @statsd_metrics
+ def invalidate(self) -> Response:
+ """
+ Takes a list of datasources, finds the associated cache records and
+ invalidates them and removes the database records
+
+ ---
+ post:
+ description: >-
+ Takes a list of datasources, finds the associated cache records and
+ invalidates them and removes the database records
+ requestBody:
+ description: >-
+ A list of datasources uuid or the tuples of database and datasource names
+ required: true
+ content:
+ application/json:
+ schema:
+ $ref: "#/components/schemas/CacheInvalidationRequestSchema"
+ responses:
+ 201:
+ description: cache was successfully invalidated
+ 400:
+ $ref: '#/components/responses/400'
+ 500:
+ $ref: '#/components/responses/500'
+ """
+ try:
+ datasources = CacheInvalidationRequestSchema().load(request.json)
+ except KeyError:
+ return self.response_400(message="Request is incorrect")
+ except ValidationError as error:
+ return self.response_400(message=error.message)
+ datasource_uids = set(datasources.get("datasource_uids", []))
+ for ds in datasources.get("datasources", []):
+ ds_obj = ConnectorRegistry.get_datasource_by_name(
+ session=db.session,
+ datasource_type=ds.get("datasource_type"),
+ datasource_name=ds.get("datasource_name"),
+ schema=ds.get("schema"),
+ database_name=ds.get("database_name"),
+ )
+ if ds_obj:
+ datasource_uids.add(ds_obj.uid)
+
+ cache_keys = (
+ db.session.query(CacheKey)
+ .filter(CacheKey.datasource_uid.in_(datasource_uids))
+ .all()
+ )
+ if cache_keys:
+ cache_manager.cache.delete_many(*[c.cache_key for c in cache_keys])
Review comment:
delete_many doesn't fail, only returns true / false if the keys were deleted. e.g. if the key was not found it will return false. Updated CacheKey logic to delete is a single statement & rollback if fails -> that would be more robust.
deleting case & keeping records is not too bad as there still a need to garbage collect those and cache is not a persistent store.
##########
File path: superset/cachekeys/schemas.py
##########
@@ -0,0 +1,45 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+# RISON/JSON schemas for query parameters
+from marshmallow import fields, Schema, validate
+
+from superset.charts.schemas import (
+ datasource_name_description,
+ datasource_type_description,
+ datasource_uid_description,
+)
+
+
+class Datasource(Schema):
+ database_name = fields.String(description="Datasource name",)
+ datasource_name = fields.String(description=datasource_name_description,)
+ schema = fields.String(description="Datasource schema",)
+ datasource_type = fields.String(
+ description=datasource_type_description,
+ validate=validate.OneOf(choices=("druid", "table", "view")),
+ allow_none=True,
Review comment:
done
##########
File path: superset/cachekeys/api.py
##########
@@ -0,0 +1,98 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from flask import request, Response
+from flask_appbuilder import expose
+from flask_appbuilder.api import safe
+from flask_appbuilder.models.sqla.interface import SQLAInterface
+from flask_appbuilder.security.decorators import protect
+from jsonschema import ValidationError
+
+from superset.cachekeys.schemas import CacheInvalidationRequestSchema
+from superset.connectors.connector_registry import ConnectorRegistry
+from superset.extensions import cache_manager, db, event_logger
+from superset.models.cache import CacheKey
+from superset.views.base_api import BaseSupersetModelRestApi, statsd_metrics
+
+
+class CacheRestApi(BaseSupersetModelRestApi):
+ datamodel = SQLAInterface(CacheKey)
+ resource_name = "cache"
+ allow_browser_login = True
+ class_permission_name = "CacheRestApi"
+
+ openapi_spec_component_schemas = (CacheInvalidationRequestSchema,)
+
+ @expose("/invalidate", methods=["POST"])
+ @event_logger.log_this
+ @protect()
+ @safe
+ @statsd_metrics
+ def invalidate(self) -> Response:
+ """
+ Takes a list of datasources, finds the associated cache records and
+ invalidates them and removes the database records
+
+ ---
+ post:
+ description: >-
+ Takes a list of datasources, finds the associated cache records and
+ invalidates them and removes the database records
+ requestBody:
+ description: >-
+ A list of datasources uuid or the tuples of database and datasource names
+ required: true
+ content:
+ application/json:
+ schema:
+ $ref: "#/components/schemas/CacheInvalidationRequestSchema"
+ responses:
+ 201:
+ description: cache was successfully invalidated
+ 400:
+ $ref: '#/components/responses/400'
+ 500:
+ $ref: '#/components/responses/500'
+ """
+ try:
+ datasources = CacheInvalidationRequestSchema().load(request.json)
+ except KeyError:
+ return self.response_400(message="Request is incorrect")
+ except ValidationError as error:
+ return self.response_400(message=error.message)
+ datasource_uids = set(datasources.get("datasource_uids", []))
+ for ds in datasources.get("datasources", []):
+ ds_obj = ConnectorRegistry.get_datasource_by_name(
+ session=db.session,
+ datasource_type=ds.get("datasource_type"),
+ datasource_name=ds.get("datasource_name"),
+ schema=ds.get("schema"),
+ database_name=ds.get("database_name"),
+ )
+ if ds_obj:
+ datasource_uids.add(ds_obj.uid)
+
+ cache_keys = (
+ db.session.query(CacheKey)
+ .filter(CacheKey.datasource_uid.in_(datasource_uids))
+ .all()
+ )
+ if cache_keys:
+ cache_manager.cache.delete_many(*[c.cache_key for c in cache_keys])
Review comment:
delete_many doesn't fail, only returns true / false if the keys were deleted. e.g. if the key was not found it will return false. Updated CacheKey logic to delete is a single statement & rollback if fails -> that would be more robust.
deleting case & keeping records is not too bad as there still a need to garbage collect those and cache is not a persistent store.
##########
File path: superset/cachekeys/schemas.py
##########
@@ -0,0 +1,45 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+# RISON/JSON schemas for query parameters
+from marshmallow import fields, Schema, validate
+
+from superset.charts.schemas import (
+ datasource_name_description,
+ datasource_type_description,
+ datasource_uid_description,
+)
+
+
+class Datasource(Schema):
+ database_name = fields.String(description="Datasource name",)
+ datasource_name = fields.String(description=datasource_name_description,)
+ schema = fields.String(description="Datasource schema",)
+ datasource_type = fields.String(
+ description=datasource_type_description,
+ validate=validate.OneOf(choices=("druid", "table", "view")),
+ allow_none=True,
Review comment:
done
##########
File path: superset/cachekeys/api.py
##########
@@ -0,0 +1,98 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from flask import request, Response
+from flask_appbuilder import expose
+from flask_appbuilder.api import safe
+from flask_appbuilder.models.sqla.interface import SQLAInterface
+from flask_appbuilder.security.decorators import protect
+from jsonschema import ValidationError
+
+from superset.cachekeys.schemas import CacheInvalidationRequestSchema
+from superset.connectors.connector_registry import ConnectorRegistry
+from superset.extensions import cache_manager, db, event_logger
+from superset.models.cache import CacheKey
+from superset.views.base_api import BaseSupersetModelRestApi, statsd_metrics
+
+
+class CacheRestApi(BaseSupersetModelRestApi):
+ datamodel = SQLAInterface(CacheKey)
+ resource_name = "cache"
+ allow_browser_login = True
+ class_permission_name = "CacheRestApi"
+
+ openapi_spec_component_schemas = (CacheInvalidationRequestSchema,)
+
+ @expose("/invalidate", methods=["POST"])
+ @event_logger.log_this
+ @protect()
+ @safe
+ @statsd_metrics
+ def invalidate(self) -> Response:
+ """
+ Takes a list of datasources, finds the associated cache records and
+ invalidates them and removes the database records
+
+ ---
+ post:
+ description: >-
+ Takes a list of datasources, finds the associated cache records and
+ invalidates them and removes the database records
+ requestBody:
+ description: >-
+ A list of datasources uuid or the tuples of database and datasource names
+ required: true
+ content:
+ application/json:
+ schema:
+ $ref: "#/components/schemas/CacheInvalidationRequestSchema"
+ responses:
+ 201:
+ description: cache was successfully invalidated
+ 400:
+ $ref: '#/components/responses/400'
+ 500:
+ $ref: '#/components/responses/500'
+ """
+ try:
+ datasources = CacheInvalidationRequestSchema().load(request.json)
+ except KeyError:
+ return self.response_400(message="Request is incorrect")
+ except ValidationError as error:
+ return self.response_400(message=error.message)
+ datasource_uids = set(datasources.get("datasource_uids", []))
+ for ds in datasources.get("datasources", []):
+ ds_obj = ConnectorRegistry.get_datasource_by_name(
+ session=db.session,
+ datasource_type=ds.get("datasource_type"),
+ datasource_name=ds.get("datasource_name"),
+ schema=ds.get("schema"),
+ database_name=ds.get("database_name"),
+ )
+ if ds_obj:
+ datasource_uids.add(ds_obj.uid)
+
+ cache_keys = (
+ db.session.query(CacheKey)
+ .filter(CacheKey.datasource_uid.in_(datasource_uids))
+ .all()
+ )
+ if cache_keys:
+ cache_manager.cache.delete_many(*[c.cache_key for c in cache_keys])
Review comment:
delete_many doesn't fail, only returns true / false if the keys were deleted. e.g. if the key was not found it will return false. Updated CacheKey logic to delete is a single statement & rollback if fails -> that would be more robust.
deleting case & keeping records is not too bad as there still a need to garbage collect those and cache is not a persistent store.
##########
File path: superset/cachekeys/schemas.py
##########
@@ -0,0 +1,45 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+# RISON/JSON schemas for query parameters
+from marshmallow import fields, Schema, validate
+
+from superset.charts.schemas import (
+ datasource_name_description,
+ datasource_type_description,
+ datasource_uid_description,
+)
+
+
+class Datasource(Schema):
+ database_name = fields.String(description="Datasource name",)
+ datasource_name = fields.String(description=datasource_name_description,)
+ schema = fields.String(description="Datasource schema",)
+ datasource_type = fields.String(
+ description=datasource_type_description,
+ validate=validate.OneOf(choices=("druid", "table", "view")),
+ allow_none=True,
Review comment:
done
##########
File path: superset/cachekeys/api.py
##########
@@ -0,0 +1,98 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from flask import request, Response
+from flask_appbuilder import expose
+from flask_appbuilder.api import safe
+from flask_appbuilder.models.sqla.interface import SQLAInterface
+from flask_appbuilder.security.decorators import protect
+from jsonschema import ValidationError
+
+from superset.cachekeys.schemas import CacheInvalidationRequestSchema
+from superset.connectors.connector_registry import ConnectorRegistry
+from superset.extensions import cache_manager, db, event_logger
+from superset.models.cache import CacheKey
+from superset.views.base_api import BaseSupersetModelRestApi, statsd_metrics
+
+
+class CacheRestApi(BaseSupersetModelRestApi):
+ datamodel = SQLAInterface(CacheKey)
+ resource_name = "cache"
+ allow_browser_login = True
+ class_permission_name = "CacheRestApi"
+
+ openapi_spec_component_schemas = (CacheInvalidationRequestSchema,)
+
+ @expose("/invalidate", methods=["POST"])
+ @event_logger.log_this
+ @protect()
+ @safe
+ @statsd_metrics
+ def invalidate(self) -> Response:
+ """
+ Takes a list of datasources, finds the associated cache records and
+ invalidates them and removes the database records
+
+ ---
+ post:
+ description: >-
+ Takes a list of datasources, finds the associated cache records and
+ invalidates them and removes the database records
+ requestBody:
+ description: >-
+ A list of datasources uuid or the tuples of database and datasource names
+ required: true
+ content:
+ application/json:
+ schema:
+ $ref: "#/components/schemas/CacheInvalidationRequestSchema"
+ responses:
+ 201:
+ description: cache was successfully invalidated
+ 400:
+ $ref: '#/components/responses/400'
+ 500:
+ $ref: '#/components/responses/500'
+ """
+ try:
+ datasources = CacheInvalidationRequestSchema().load(request.json)
+ except KeyError:
+ return self.response_400(message="Request is incorrect")
+ except ValidationError as error:
+ return self.response_400(message=error.message)
+ datasource_uids = set(datasources.get("datasource_uids", []))
+ for ds in datasources.get("datasources", []):
+ ds_obj = ConnectorRegistry.get_datasource_by_name(
+ session=db.session,
+ datasource_type=ds.get("datasource_type"),
+ datasource_name=ds.get("datasource_name"),
+ schema=ds.get("schema"),
+ database_name=ds.get("database_name"),
+ )
+ if ds_obj:
+ datasource_uids.add(ds_obj.uid)
+
+ cache_keys = (
+ db.session.query(CacheKey)
+ .filter(CacheKey.datasource_uid.in_(datasource_uids))
+ .all()
+ )
+ if cache_keys:
+ cache_manager.cache.delete_many(*[c.cache_key for c in cache_keys])
Review comment:
delete_many doesn't fail, only returns true / false if the keys were deleted. e.g. if the key was not found it will return false. Updated CacheKey logic to delete is a single statement & rollback if fails -> that would be more robust.
deleting case & keeping records is not too bad as there still a need to garbage collect those and cache is not a persistent store.
##########
File path: superset/cachekeys/schemas.py
##########
@@ -0,0 +1,45 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+# RISON/JSON schemas for query parameters
+from marshmallow import fields, Schema, validate
+
+from superset.charts.schemas import (
+ datasource_name_description,
+ datasource_type_description,
+ datasource_uid_description,
+)
+
+
+class Datasource(Schema):
+ database_name = fields.String(description="Datasource name",)
+ datasource_name = fields.String(description=datasource_name_description,)
+ schema = fields.String(description="Datasource schema",)
+ datasource_type = fields.String(
+ description=datasource_type_description,
+ validate=validate.OneOf(choices=("druid", "table", "view")),
+ allow_none=True,
Review comment:
done
##########
File path: superset/cachekeys/api.py
##########
@@ -0,0 +1,98 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from flask import request, Response
+from flask_appbuilder import expose
+from flask_appbuilder.api import safe
+from flask_appbuilder.models.sqla.interface import SQLAInterface
+from flask_appbuilder.security.decorators import protect
+from jsonschema import ValidationError
+
+from superset.cachekeys.schemas import CacheInvalidationRequestSchema
+from superset.connectors.connector_registry import ConnectorRegistry
+from superset.extensions import cache_manager, db, event_logger
+from superset.models.cache import CacheKey
+from superset.views.base_api import BaseSupersetModelRestApi, statsd_metrics
+
+
+class CacheRestApi(BaseSupersetModelRestApi):
+ datamodel = SQLAInterface(CacheKey)
+ resource_name = "cache"
+ allow_browser_login = True
+ class_permission_name = "CacheRestApi"
+
+ openapi_spec_component_schemas = (CacheInvalidationRequestSchema,)
+
+ @expose("/invalidate", methods=["POST"])
+ @event_logger.log_this
+ @protect()
+ @safe
+ @statsd_metrics
+ def invalidate(self) -> Response:
+ """
+ Takes a list of datasources, finds the associated cache records and
+ invalidates them and removes the database records
+
+ ---
+ post:
+ description: >-
+ Takes a list of datasources, finds the associated cache records and
+ invalidates them and removes the database records
+ requestBody:
+ description: >-
+ A list of datasources uuid or the tuples of database and datasource names
+ required: true
+ content:
+ application/json:
+ schema:
+ $ref: "#/components/schemas/CacheInvalidationRequestSchema"
+ responses:
+ 201:
+ description: cache was successfully invalidated
+ 400:
+ $ref: '#/components/responses/400'
+ 500:
+ $ref: '#/components/responses/500'
+ """
+ try:
+ datasources = CacheInvalidationRequestSchema().load(request.json)
+ except KeyError:
+ return self.response_400(message="Request is incorrect")
+ except ValidationError as error:
+ return self.response_400(message=error.message)
+ datasource_uids = set(datasources.get("datasource_uids", []))
+ for ds in datasources.get("datasources", []):
+ ds_obj = ConnectorRegistry.get_datasource_by_name(
+ session=db.session,
+ datasource_type=ds.get("datasource_type"),
+ datasource_name=ds.get("datasource_name"),
+ schema=ds.get("schema"),
+ database_name=ds.get("database_name"),
+ )
+ if ds_obj:
+ datasource_uids.add(ds_obj.uid)
+
+ cache_keys = (
+ db.session.query(CacheKey)
+ .filter(CacheKey.datasource_uid.in_(datasource_uids))
+ .all()
+ )
+ if cache_keys:
+ cache_manager.cache.delete_many(*[c.cache_key for c in cache_keys])
Review comment:
delete_many doesn't fail, only returns true / false if the keys were deleted. e.g. if the key was not found it will return false. Updated CacheKey logic to delete is a single statement & rollback if fails -> that would be more robust.
deleting case & keeping records is not too bad as there still a need to garbage collect those and cache is not a persistent store.
##########
File path: superset/cachekeys/schemas.py
##########
@@ -0,0 +1,45 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+# RISON/JSON schemas for query parameters
+from marshmallow import fields, Schema, validate
+
+from superset.charts.schemas import (
+ datasource_name_description,
+ datasource_type_description,
+ datasource_uid_description,
+)
+
+
+class Datasource(Schema):
+ database_name = fields.String(description="Datasource name",)
+ datasource_name = fields.String(description=datasource_name_description,)
+ schema = fields.String(description="Datasource schema",)
+ datasource_type = fields.String(
+ description=datasource_type_description,
+ validate=validate.OneOf(choices=("druid", "table", "view")),
+ allow_none=True,
Review comment:
done
##########
File path: superset/cachekeys/api.py
##########
@@ -0,0 +1,98 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from flask import request, Response
+from flask_appbuilder import expose
+from flask_appbuilder.api import safe
+from flask_appbuilder.models.sqla.interface import SQLAInterface
+from flask_appbuilder.security.decorators import protect
+from jsonschema import ValidationError
+
+from superset.cachekeys.schemas import CacheInvalidationRequestSchema
+from superset.connectors.connector_registry import ConnectorRegistry
+from superset.extensions import cache_manager, db, event_logger
+from superset.models.cache import CacheKey
+from superset.views.base_api import BaseSupersetModelRestApi, statsd_metrics
+
+
+class CacheRestApi(BaseSupersetModelRestApi):
+ datamodel = SQLAInterface(CacheKey)
+ resource_name = "cache"
+ allow_browser_login = True
+ class_permission_name = "CacheRestApi"
+
+ openapi_spec_component_schemas = (CacheInvalidationRequestSchema,)
+
+ @expose("/invalidate", methods=["POST"])
+ @event_logger.log_this
+ @protect()
+ @safe
+ @statsd_metrics
+ def invalidate(self) -> Response:
+ """
+ Takes a list of datasources, finds the associated cache records and
+ invalidates them and removes the database records
+
+ ---
+ post:
+ description: >-
+ Takes a list of datasources, finds the associated cache records and
+ invalidates them and removes the database records
+ requestBody:
+ description: >-
+ A list of datasources uuid or the tuples of database and datasource names
+ required: true
+ content:
+ application/json:
+ schema:
+ $ref: "#/components/schemas/CacheInvalidationRequestSchema"
+ responses:
+ 201:
+ description: cache was successfully invalidated
+ 400:
+ $ref: '#/components/responses/400'
+ 500:
+ $ref: '#/components/responses/500'
+ """
+ try:
+ datasources = CacheInvalidationRequestSchema().load(request.json)
+ except KeyError:
+ return self.response_400(message="Request is incorrect")
+ except ValidationError as error:
+ return self.response_400(message=error.message)
+ datasource_uids = set(datasources.get("datasource_uids", []))
+ for ds in datasources.get("datasources", []):
+ ds_obj = ConnectorRegistry.get_datasource_by_name(
+ session=db.session,
+ datasource_type=ds.get("datasource_type"),
+ datasource_name=ds.get("datasource_name"),
+ schema=ds.get("schema"),
+ database_name=ds.get("database_name"),
+ )
+ if ds_obj:
+ datasource_uids.add(ds_obj.uid)
+
+ cache_keys = (
+ db.session.query(CacheKey)
+ .filter(CacheKey.datasource_uid.in_(datasource_uids))
+ .all()
+ )
+ if cache_keys:
+ cache_manager.cache.delete_many(*[c.cache_key for c in cache_keys])
Review comment:
delete_many doesn't fail, only returns true / false if the keys were deleted. e.g. if the key was not found it will return false. Updated CacheKey logic to delete is a single statement & rollback if fails -> that would be more robust.
deleting case & keeping records is not too bad as there still a need to garbage collect those and cache is not a persistent store.
##########
File path: superset/cachekeys/schemas.py
##########
@@ -0,0 +1,45 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+# RISON/JSON schemas for query parameters
+from marshmallow import fields, Schema, validate
+
+from superset.charts.schemas import (
+ datasource_name_description,
+ datasource_type_description,
+ datasource_uid_description,
+)
+
+
+class Datasource(Schema):
+ database_name = fields.String(description="Datasource name",)
+ datasource_name = fields.String(description=datasource_name_description,)
+ schema = fields.String(description="Datasource schema",)
+ datasource_type = fields.String(
+ description=datasource_type_description,
+ validate=validate.OneOf(choices=("druid", "table", "view")),
+ allow_none=True,
Review comment:
done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [incubator-superset] codecov-commenter edited a comment on pull request #10761: feat: [WIP] implement cache invalidation api
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #10761:
URL: https://github.com/apache/incubator-superset/pull/10761#issuecomment-686252702
# [Codecov](https://codecov.io/gh/apache/incubator-superset/pull/10761?src=pr&el=h1) Report
> Merging [#10761](https://codecov.io/gh/apache/incubator-superset/pull/10761?src=pr&el=desc) into [master](https://codecov.io/gh/apache/incubator-superset/commit/1ee87cc4d1810163bd00788636291588ecbfcbdf?el=desc) will **increase** coverage by `2.36%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/incubator-superset/pull/10761?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #10761 +/- ##
==========================================
+ Coverage 59.23% 61.59% +2.36%
==========================================
Files 768 429 -339
Lines 36651 13971 -22680
Branches 3302 3555 +253
==========================================
- Hits 21710 8606 -13104
+ Misses 14759 5179 -9580
- Partials 182 186 +4
```
| Flag | Coverage Δ | |
|---|---|---|
| #cypress | `?` | |
| #javascript | `61.59% <ø> (?)` | |
| #python | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/incubator-superset/pull/10761?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [superset-frontend/src/views/App.tsx](https://codecov.io/gh/apache/incubator-superset/pull/10761/diff?src=pr&el=tree#diff-c3VwZXJzZXQtZnJvbnRlbmQvc3JjL3ZpZXdzL0FwcC50c3g=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [superset-frontend/src/SqlLab/App.jsx](https://codecov.io/gh/apache/incubator-superset/pull/10761/diff?src=pr&el=tree#diff-c3VwZXJzZXQtZnJvbnRlbmQvc3JjL1NxbExhYi9BcHAuanN4) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [superset-frontend/src/views/menu.tsx](https://codecov.io/gh/apache/incubator-superset/pull/10761/diff?src=pr&el=tree#diff-c3VwZXJzZXQtZnJvbnRlbmQvc3JjL3ZpZXdzL21lbnUudHN4) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [superset-frontend/src/explore/App.jsx](https://codecov.io/gh/apache/incubator-superset/pull/10761/diff?src=pr&el=tree#diff-c3VwZXJzZXQtZnJvbnRlbmQvc3JjL2V4cGxvcmUvQXBwLmpzeA==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [superset-frontend/src/views/index.tsx](https://codecov.io/gh/apache/incubator-superset/pull/10761/diff?src=pr&el=tree#diff-c3VwZXJzZXQtZnJvbnRlbmQvc3JjL3ZpZXdzL2luZGV4LnRzeA==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [superset-frontend/src/SqlLab/index.tsx](https://codecov.io/gh/apache/incubator-superset/pull/10761/diff?src=pr&el=tree#diff-c3VwZXJzZXQtZnJvbnRlbmQvc3JjL1NxbExhYi9pbmRleC50c3g=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [superset-frontend/src/dashboard/App.jsx](https://codecov.io/gh/apache/incubator-superset/pull/10761/diff?src=pr&el=tree#diff-c3VwZXJzZXQtZnJvbnRlbmQvc3JjL2Rhc2hib2FyZC9BcHAuanN4) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [superset-frontend/src/explore/index.jsx](https://codecov.io/gh/apache/incubator-superset/pull/10761/diff?src=pr&el=tree#diff-c3VwZXJzZXQtZnJvbnRlbmQvc3JjL2V4cGxvcmUvaW5kZXguanN4) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [superset-frontend/src/dashboard/index.jsx](https://codecov.io/gh/apache/incubator-superset/pull/10761/diff?src=pr&el=tree#diff-c3VwZXJzZXQtZnJvbnRlbmQvc3JjL2Rhc2hib2FyZC9pbmRleC5qc3g=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [superset-frontend/src/setup/setupColors.js](https://codecov.io/gh/apache/incubator-superset/pull/10761/diff?src=pr&el=tree#diff-c3VwZXJzZXQtZnJvbnRlbmQvc3JjL3NldHVwL3NldHVwQ29sb3JzLmpz) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| ... and [691 more](https://codecov.io/gh/apache/incubator-superset/pull/10761/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-superset/pull/10761?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/incubator-superset/pull/10761?src=pr&el=footer). Last update [1ee87cc...2dc4543](https://codecov.io/gh/apache/incubator-superset/pull/10761?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [incubator-superset] benceorlai commented on pull request #10761: feat: implement cache invalidation api
Posted by GitBox <gi...@apache.org>.
benceorlai commented on pull request #10761:
URL: https://github.com/apache/incubator-superset/pull/10761#issuecomment-735502767
Roadmap: https://github.com/apache-superset/superset-roadmap/issues/74
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [incubator-superset] codecov-commenter commented on pull request #10761: feat: [WIP] implement cache invalidation api
Posted by GitBox <gi...@apache.org>.
codecov-commenter commented on pull request #10761:
URL: https://github.com/apache/incubator-superset/pull/10761#issuecomment-686252702
# [Codecov](https://codecov.io/gh/apache/incubator-superset/pull/10761?src=pr&el=h1) Report
> Merging [#10761](https://codecov.io/gh/apache/incubator-superset/pull/10761?src=pr&el=desc) into [master](https://codecov.io/gh/apache/incubator-superset/commit/e0ad9935c145493b5cb6048f1aeab9018edd363a?el=desc) will **decrease** coverage by `3.46%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/incubator-superset/pull/10761?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #10761 +/- ##
==========================================
- Coverage 65.06% 61.59% -3.47%
==========================================
Files 795 429 -366
Lines 37403 13971 -23432
Branches 3555 3555
==========================================
- Hits 24337 8606 -15731
+ Misses 12959 5179 -7780
- Partials 107 186 +79
```
| Flag | Coverage Δ | |
|---|---|---|
| #cypress | `?` | |
| #javascript | `61.59% <ø> (+0.10%)` | :arrow_up: |
| #python | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/incubator-superset/pull/10761?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [superset-frontend/src/SqlLab/App.jsx](https://codecov.io/gh/apache/incubator-superset/pull/10761/diff?src=pr&el=tree#diff-c3VwZXJzZXQtZnJvbnRlbmQvc3JjL1NxbExhYi9BcHAuanN4) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [superset-frontend/src/explore/App.jsx](https://codecov.io/gh/apache/incubator-superset/pull/10761/diff?src=pr&el=tree#diff-c3VwZXJzZXQtZnJvbnRlbmQvc3JjL2V4cGxvcmUvQXBwLmpzeA==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [superset-frontend/src/dashboard/App.jsx](https://codecov.io/gh/apache/incubator-superset/pull/10761/diff?src=pr&el=tree#diff-c3VwZXJzZXQtZnJvbnRlbmQvc3JjL2Rhc2hib2FyZC9BcHAuanN4) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [superset-frontend/src/explore/index.jsx](https://codecov.io/gh/apache/incubator-superset/pull/10761/diff?src=pr&el=tree#diff-c3VwZXJzZXQtZnJvbnRlbmQvc3JjL2V4cGxvcmUvaW5kZXguanN4) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [superset-frontend/src/dashboard/index.jsx](https://codecov.io/gh/apache/incubator-superset/pull/10761/diff?src=pr&el=tree#diff-c3VwZXJzZXQtZnJvbnRlbmQvc3JjL2Rhc2hib2FyZC9pbmRleC5qc3g=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [superset-frontend/src/setup/setupColors.js](https://codecov.io/gh/apache/incubator-superset/pull/10761/diff?src=pr&el=tree#diff-c3VwZXJzZXQtZnJvbnRlbmQvc3JjL3NldHVwL3NldHVwQ29sb3JzLmpz) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [superset-frontend/src/chart/ChartContainer.jsx](https://codecov.io/gh/apache/incubator-superset/pull/10761/diff?src=pr&el=tree#diff-c3VwZXJzZXQtZnJvbnRlbmQvc3JjL2NoYXJ0L0NoYXJ0Q29udGFpbmVyLmpzeA==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [superset-frontend/src/setup/setupFormatters.js](https://codecov.io/gh/apache/incubator-superset/pull/10761/diff?src=pr&el=tree#diff-c3VwZXJzZXQtZnJvbnRlbmQvc3JjL3NldHVwL3NldHVwRm9ybWF0dGVycy5qcw==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [superset-frontend/src/explore/reducers/index.js](https://codecov.io/gh/apache/incubator-superset/pull/10761/diff?src=pr&el=tree#diff-c3VwZXJzZXQtZnJvbnRlbmQvc3JjL2V4cGxvcmUvcmVkdWNlcnMvaW5kZXguanM=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [superset-frontend/src/setup/setupPluginsExtra.js](https://codecov.io/gh/apache/incubator-superset/pull/10761/diff?src=pr&el=tree#diff-c3VwZXJzZXQtZnJvbnRlbmQvc3JjL3NldHVwL3NldHVwUGx1Z2luc0V4dHJhLmpz) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| ... and [524 more](https://codecov.io/gh/apache/incubator-superset/pull/10761/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-superset/pull/10761?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/incubator-superset/pull/10761?src=pr&el=footer). Last update [e0ad993...2028348](https://codecov.io/gh/apache/incubator-superset/pull/10761?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [incubator-superset] bkyryliuk commented on pull request #10761: feat: [WIP] implement cache invalidation api
Posted by GitBox <gi...@apache.org>.
bkyryliuk commented on pull request #10761:
URL: https://github.com/apache/incubator-superset/pull/10761#issuecomment-687282114
@willbarrett , @dpgaspar - ready for review
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [incubator-superset] bkyryliuk merged pull request #10761: feat: implement cache invalidation api
Posted by GitBox <gi...@apache.org>.
bkyryliuk merged pull request #10761:
URL: https://github.com/apache/incubator-superset/pull/10761
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [incubator-superset] mistercrunch commented on a change in pull request #10761: feat: implement cache invalidation api
Posted by GitBox <gi...@apache.org>.
mistercrunch commented on a change in pull request #10761:
URL: https://github.com/apache/incubator-superset/pull/10761#discussion_r484002542
##########
File path: superset/cachekeys/api.py
##########
@@ -0,0 +1,98 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from flask import request, Response
+from flask_appbuilder import expose
+from flask_appbuilder.api import safe
+from flask_appbuilder.models.sqla.interface import SQLAInterface
+from flask_appbuilder.security.decorators import protect
+from jsonschema import ValidationError
+
+from superset.cachekeys.schemas import CacheInvalidationRequestSchema
+from superset.connectors.connector_registry import ConnectorRegistry
+from superset.extensions import cache_manager, db, event_logger
+from superset.models.cache import CacheKey
+from superset.views.base_api import BaseSupersetModelRestApi, statsd_metrics
+
+
+class CacheRestApi(BaseSupersetModelRestApi):
+ datamodel = SQLAInterface(CacheKey)
+ resource_name = "cache"
+ allow_browser_login = True
+ class_permission_name = "CacheRestApi"
+
+ openapi_spec_component_schemas = (CacheInvalidationRequestSchema,)
+
+ @expose("/invalidate", methods=["POST"])
+ @event_logger.log_this
+ @protect()
+ @safe
+ @statsd_metrics
+ def invalidate(self) -> Response:
+ """
+ Takes a list of datasources, finds the associated cache records and
Review comment:
Side note (outside the scope of this PR), we'll be renaming datasource to dataset. I think it's mostly done in the UI layer, but has yet to be done in the code.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [incubator-superset] dpgaspar commented on a change in pull request #10761: feat: [WIP] implement cache invalidation api
Posted by GitBox <gi...@apache.org>.
dpgaspar commented on a change in pull request #10761:
URL: https://github.com/apache/incubator-superset/pull/10761#discussion_r482152879
##########
File path: superset/cache/api.py
##########
@@ -0,0 +1,91 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from flask import request, Response
+from flask_appbuilder import expose
+from flask_appbuilder.api import safe
+from flask_appbuilder.models.sqla.interface import SQLAInterface
+from flask_appbuilder.security.decorators import protect
+from flask_babel import gettext as _
+from jsonschema import ValidationError
+
+from superset.cache.schemas import CacheInvalidationRequestSchema
+from superset.connectors.connector_registry import ConnectorRegistry
+from superset.extensions import cache_manager, db, event_logger
+from superset.models.cache import CacheKey
+from superset.views.base_api import BaseSupersetModelRestApi, statsd_metrics
+
+
+class CacheRestApi(BaseSupersetModelRestApi):
+ datamodel = SQLAInterface(CacheKey)
+ resource_name = "cache"
+ allow_browser_login = True
+ class_permission_name = "CacheRestApi"
+
Review comment:
use `openapi_spec_component_schemas = CacheInvalidateRequestSchema` to add the schema to the OpenAPI spec
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [incubator-superset] bkyryliuk commented on pull request #10761: feat: implement cache invalidation api
Posted by GitBox <gi...@apache.org>.
bkyryliuk commented on pull request #10761:
URL: https://github.com/apache/incubator-superset/pull/10761#issuecomment-692213931
> Do you want all the `ModelRestApi` route methods to be available? If not include `include_route_methods = {"invalidate",}` also note this will create a new specific permission `can invalidate on Cache key`.
> Check `https://localhost:8080/swagger/v1`
addressed, thanks
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [incubator-superset] willbarrett commented on a change in pull request #10761: feat: implement cache invalidation api
Posted by GitBox <gi...@apache.org>.
willbarrett commented on a change in pull request #10761:
URL: https://github.com/apache/incubator-superset/pull/10761#discussion_r488349155
##########
File path: superset/cachekeys/api.py
##########
@@ -0,0 +1,98 @@
+# Licensed to the Apache Software Foundation (ASF) under one
Review comment:
How about `DELETE /api/v1/cachekey/<cachekey>`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [incubator-superset] bkyryliuk commented on a change in pull request #10761: feat: implement cache invalidation api
Posted by GitBox <gi...@apache.org>.
bkyryliuk commented on a change in pull request #10761:
URL: https://github.com/apache/incubator-superset/pull/10761#discussion_r487139156
##########
File path: superset/cachekeys/api.py
##########
@@ -0,0 +1,98 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from flask import request, Response
+from flask_appbuilder import expose
+from flask_appbuilder.api import safe
+from flask_appbuilder.models.sqla.interface import SQLAInterface
+from flask_appbuilder.security.decorators import protect
+from jsonschema import ValidationError
+
+from superset.cachekeys.schemas import CacheInvalidationRequestSchema
+from superset.connectors.connector_registry import ConnectorRegistry
+from superset.extensions import cache_manager, db, event_logger
+from superset.models.cache import CacheKey
+from superset.views.base_api import BaseSupersetModelRestApi, statsd_metrics
+
+
+class CacheRestApi(BaseSupersetModelRestApi):
+ datamodel = SQLAInterface(CacheKey)
+ resource_name = "cache"
+ allow_browser_login = True
+ class_permission_name = "CacheRestApi"
+
+ openapi_spec_component_schemas = (CacheInvalidationRequestSchema,)
+
+ @expose("/invalidate", methods=["POST"])
+ @event_logger.log_this
+ @protect()
+ @safe
+ @statsd_metrics
+ def invalidate(self) -> Response:
+ """
+ Takes a list of datasources, finds the associated cache records and
+ invalidates them and removes the database records
+
+ ---
+ post:
+ description: >-
+ Takes a list of datasources, finds the associated cache records and
+ invalidates them and removes the database records
+ requestBody:
+ description: >-
+ A list of datasources uuid or the tuples of database and datasource names
+ required: true
+ content:
+ application/json:
+ schema:
+ $ref: "#/components/schemas/CacheInvalidationRequestSchema"
+ responses:
+ 201:
+ description: cache was successfully invalidated
+ 400:
+ $ref: '#/components/responses/400'
+ 500:
+ $ref: '#/components/responses/500'
+ """
+ try:
+ datasources = CacheInvalidationRequestSchema().load(request.json)
+ except KeyError:
+ return self.response_400(message="Request is incorrect")
+ except ValidationError as error:
+ return self.response_400(message=error.message)
+ datasource_uids = set(datasources.get("datasource_uids", []))
+ for ds in datasources.get("datasources", []):
+ ds_obj = ConnectorRegistry.get_datasource_by_name(
+ session=db.session,
+ datasource_type=ds.get("datasource_type"),
+ datasource_name=ds.get("datasource_name"),
+ schema=ds.get("schema"),
+ database_name=ds.get("database_name"),
+ )
+ if ds_obj:
+ datasource_uids.add(ds_obj.uid)
+
+ cache_keys = (
+ db.session.query(CacheKey)
+ .filter(CacheKey.datasource_uid.in_(datasource_uids))
+ .all()
+ )
+ if cache_keys:
+ cache_manager.cache.delete_many(*[c.cache_key for c in cache_keys])
Review comment:
delete_many doesn't fail, only returns true / false if the keys were deleted. e.g. if the key was not found it will return false. Updated CacheKey logic to delete is a single statement & rollback if fails -> that would be more robust.
deleting case & keeping records is not too bad as there still a need to garbage collect those and cache is not a persistent store.
##########
File path: superset/cachekeys/schemas.py
##########
@@ -0,0 +1,45 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+# RISON/JSON schemas for query parameters
+from marshmallow import fields, Schema, validate
+
+from superset.charts.schemas import (
+ datasource_name_description,
+ datasource_type_description,
+ datasource_uid_description,
+)
+
+
+class Datasource(Schema):
+ database_name = fields.String(description="Datasource name",)
+ datasource_name = fields.String(description=datasource_name_description,)
+ schema = fields.String(description="Datasource schema",)
+ datasource_type = fields.String(
+ description=datasource_type_description,
+ validate=validate.OneOf(choices=("druid", "table", "view")),
+ allow_none=True,
Review comment:
done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [incubator-superset] bkyryliuk commented on a change in pull request #10761: feat: implement cache invalidation api
Posted by GitBox <gi...@apache.org>.
bkyryliuk commented on a change in pull request #10761:
URL: https://github.com/apache/incubator-superset/pull/10761#discussion_r486693571
##########
File path: superset/cachekeys/api.py
##########
@@ -0,0 +1,98 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from flask import request, Response
+from flask_appbuilder import expose
+from flask_appbuilder.api import safe
+from flask_appbuilder.models.sqla.interface import SQLAInterface
+from flask_appbuilder.security.decorators import protect
+from jsonschema import ValidationError
+
+from superset.cachekeys.schemas import CacheInvalidationRequestSchema
+from superset.connectors.connector_registry import ConnectorRegistry
+from superset.extensions import cache_manager, db, event_logger
+from superset.models.cache import CacheKey
+from superset.views.base_api import BaseSupersetModelRestApi, statsd_metrics
+
+
+class CacheRestApi(BaseSupersetModelRestApi):
+ datamodel = SQLAInterface(CacheKey)
+ resource_name = "cache"
Review comment:
renamed to cachekey, I am happy with either way naming it `cachekey` or `cache`
I was thinking about moving it into separate class from the cache key, but most of the instrumentation lives in BaseSupersetModelRestApi that requires to model.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [incubator-superset] bkyryliuk commented on a change in pull request #10761: feat: [WIP] implement cache invalidation api
Posted by GitBox <gi...@apache.org>.
bkyryliuk commented on a change in pull request #10761:
URL: https://github.com/apache/incubator-superset/pull/10761#discussion_r483758459
##########
File path: tests/superset_test_config.py
##########
@@ -76,6 +76,14 @@ def GET_FEATURE_FLAGS_FUNC(ff):
REDIS_PORT = os.environ.get("REDIS_PORT", "6379")
REDIS_CELERY_DB = os.environ.get("REDIS_CELERY_DB", 2)
REDIS_RESULTS_DB = os.environ.get("REDIS_RESULTS_DB", 3)
+REDIS_CACHE_DB = os.environ.get("REDIS_CACHE_DB", 4)
+
+CACHE_CONFIG = {
Review comment:
needed for cache to work on CI
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [incubator-superset] bkyryliuk commented on a change in pull request #10761: feat: implement cache invalidation api
Posted by GitBox <gi...@apache.org>.
bkyryliuk commented on a change in pull request #10761:
URL: https://github.com/apache/incubator-superset/pull/10761#discussion_r487139156
##########
File path: superset/cachekeys/api.py
##########
@@ -0,0 +1,98 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from flask import request, Response
+from flask_appbuilder import expose
+from flask_appbuilder.api import safe
+from flask_appbuilder.models.sqla.interface import SQLAInterface
+from flask_appbuilder.security.decorators import protect
+from jsonschema import ValidationError
+
+from superset.cachekeys.schemas import CacheInvalidationRequestSchema
+from superset.connectors.connector_registry import ConnectorRegistry
+from superset.extensions import cache_manager, db, event_logger
+from superset.models.cache import CacheKey
+from superset.views.base_api import BaseSupersetModelRestApi, statsd_metrics
+
+
+class CacheRestApi(BaseSupersetModelRestApi):
+ datamodel = SQLAInterface(CacheKey)
+ resource_name = "cache"
+ allow_browser_login = True
+ class_permission_name = "CacheRestApi"
+
+ openapi_spec_component_schemas = (CacheInvalidationRequestSchema,)
+
+ @expose("/invalidate", methods=["POST"])
+ @event_logger.log_this
+ @protect()
+ @safe
+ @statsd_metrics
+ def invalidate(self) -> Response:
+ """
+ Takes a list of datasources, finds the associated cache records and
+ invalidates them and removes the database records
+
+ ---
+ post:
+ description: >-
+ Takes a list of datasources, finds the associated cache records and
+ invalidates them and removes the database records
+ requestBody:
+ description: >-
+ A list of datasources uuid or the tuples of database and datasource names
+ required: true
+ content:
+ application/json:
+ schema:
+ $ref: "#/components/schemas/CacheInvalidationRequestSchema"
+ responses:
+ 201:
+ description: cache was successfully invalidated
+ 400:
+ $ref: '#/components/responses/400'
+ 500:
+ $ref: '#/components/responses/500'
+ """
+ try:
+ datasources = CacheInvalidationRequestSchema().load(request.json)
+ except KeyError:
+ return self.response_400(message="Request is incorrect")
+ except ValidationError as error:
+ return self.response_400(message=error.message)
+ datasource_uids = set(datasources.get("datasource_uids", []))
+ for ds in datasources.get("datasources", []):
+ ds_obj = ConnectorRegistry.get_datasource_by_name(
+ session=db.session,
+ datasource_type=ds.get("datasource_type"),
+ datasource_name=ds.get("datasource_name"),
+ schema=ds.get("schema"),
+ database_name=ds.get("database_name"),
+ )
+ if ds_obj:
+ datasource_uids.add(ds_obj.uid)
+
+ cache_keys = (
+ db.session.query(CacheKey)
+ .filter(CacheKey.datasource_uid.in_(datasource_uids))
+ .all()
+ )
+ if cache_keys:
+ cache_manager.cache.delete_many(*[c.cache_key for c in cache_keys])
Review comment:
delete_many doesn't fail, only returns true / false if the keys were deleted. e.g. if the key was not found it will return false. Updated CacheKey logic to delete is a single statement & rollback if fails -> that would be more robust.
deleting case & keeping records is not too bad as there still a need to garbage collect those and cache is not a persistent store.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [incubator-superset] codecov-commenter edited a comment on pull request #10761: feat: [WIP] implement cache invalidation api
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #10761:
URL: https://github.com/apache/incubator-superset/pull/10761#issuecomment-686252702
# [Codecov](https://codecov.io/gh/apache/incubator-superset/pull/10761?src=pr&el=h1) Report
> Merging [#10761](https://codecov.io/gh/apache/incubator-superset/pull/10761?src=pr&el=desc) into [master](https://codecov.io/gh/apache/incubator-superset/commit/e0ad9935c145493b5cb6048f1aeab9018edd363a?el=desc) will **increase** coverage by `7.55%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/incubator-superset/pull/10761?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #10761 +/- ##
==========================================
+ Coverage 65.06% 72.61% +7.55%
==========================================
Files 795 429 -366
Lines 37403 13973 -23430
Branches 3555 3555
==========================================
- Hits 24337 10147 -14190
+ Misses 12959 3717 -9242
- Partials 107 109 +2
```
| Flag | Coverage Δ | |
|---|---|---|
| #cypress | `55.36% <ø> (-0.70%)` | :arrow_down: |
| #javascript | `61.59% <ø> (+0.10%)` | :arrow_up: |
| #python | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/incubator-superset/pull/10761?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [...et-frontend/src/SqlLab/reducers/getInitialState.js](https://codecov.io/gh/apache/incubator-superset/pull/10761/diff?src=pr&el=tree#diff-c3VwZXJzZXQtZnJvbnRlbmQvc3JjL1NxbExhYi9yZWR1Y2Vycy9nZXRJbml0aWFsU3RhdGUuanM=) | `33.33% <0.00%> (-16.67%)` | :arrow_down: |
| [...rontend/src/visualizations/FilterBox/FilterBox.jsx](https://codecov.io/gh/apache/incubator-superset/pull/10761/diff?src=pr&el=tree#diff-c3VwZXJzZXQtZnJvbnRlbmQvc3JjL3Zpc3VhbGl6YXRpb25zL0ZpbHRlckJveC9GaWx0ZXJCb3guanN4) | `56.52% <0.00%> (-8.70%)` | :arrow_down: |
| [...rontend/src/SqlLab/components/QueryAutoRefresh.jsx](https://codecov.io/gh/apache/incubator-superset/pull/10761/diff?src=pr&el=tree#diff-c3VwZXJzZXQtZnJvbnRlbmQvc3JjL1NxbExhYi9jb21wb25lbnRzL1F1ZXJ5QXV0b1JlZnJlc2guanN4) | `65.90% <0.00%> (-6.82%)` | :arrow_down: |
| [...rontend/src/SqlLab/components/TabbedSqlEditors.jsx](https://codecov.io/gh/apache/incubator-superset/pull/10761/diff?src=pr&el=tree#diff-c3VwZXJzZXQtZnJvbnRlbmQvc3JjL1NxbExhYi9jb21wb25lbnRzL1RhYmJlZFNxbEVkaXRvcnMuanN4) | `77.27% <0.00%> (-5.20%)` | :arrow_down: |
| [...rontend/src/SqlLab/components/SqlEditorLeftBar.jsx](https://codecov.io/gh/apache/incubator-superset/pull/10761/diff?src=pr&el=tree#diff-c3VwZXJzZXQtZnJvbnRlbmQvc3JjL1NxbExhYi9jb21wb25lbnRzL1NxbEVkaXRvckxlZnRCYXIuanN4) | `44.00% <0.00%> (-4.00%)` | :arrow_down: |
| [...et-frontend/src/components/ConfirmStatusChange.tsx](https://codecov.io/gh/apache/incubator-superset/pull/10761/diff?src=pr&el=tree#diff-c3VwZXJzZXQtZnJvbnRlbmQvc3JjL2NvbXBvbmVudHMvQ29uZmlybVN0YXR1c0NoYW5nZS50c3g=) | `96.15% <0.00%> (-3.85%)` | :arrow_down: |
| [...end/src/SqlLab/components/TemplateParamsEditor.jsx](https://codecov.io/gh/apache/incubator-superset/pull/10761/diff?src=pr&el=tree#diff-c3VwZXJzZXQtZnJvbnRlbmQvc3JjL1NxbExhYi9jb21wb25lbnRzL1RlbXBsYXRlUGFyYW1zRWRpdG9yLmpzeA==) | `88.57% <0.00%> (-2.86%)` | :arrow_down: |
| [...ontend/src/views/CRUD/data/dataset/DatasetList.tsx](https://codecov.io/gh/apache/incubator-superset/pull/10761/diff?src=pr&el=tree#diff-c3VwZXJzZXQtZnJvbnRlbmQvc3JjL3ZpZXdzL0NSVUQvZGF0YS9kYXRhc2V0L0RhdGFzZXRMaXN0LnRzeA==) | `72.58% <0.00%> (-2.78%)` | :arrow_down: |
| [superset-frontend/src/SqlLab/actions/sqlLab.js](https://codecov.io/gh/apache/incubator-superset/pull/10761/diff?src=pr&el=tree#diff-c3VwZXJzZXQtZnJvbnRlbmQvc3JjL1NxbExhYi9hY3Rpb25zL3NxbExhYi5qcw==) | `60.25% <0.00%> (-2.36%)` | :arrow_down: |
| [.../src/dashboard/components/gridComponents/Chart.jsx](https://codecov.io/gh/apache/incubator-superset/pull/10761/diff?src=pr&el=tree#diff-c3VwZXJzZXQtZnJvbnRlbmQvc3JjL2Rhc2hib2FyZC9jb21wb25lbnRzL2dyaWRDb21wb25lbnRzL0NoYXJ0LmpzeA==) | `83.67% <0.00%> (-2.05%)` | :arrow_down: |
| ... and [374 more](https://codecov.io/gh/apache/incubator-superset/pull/10761/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-superset/pull/10761?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/incubator-superset/pull/10761?src=pr&el=footer). Last update [e0ad993...2028348](https://codecov.io/gh/apache/incubator-superset/pull/10761?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [incubator-superset] bkyryliuk commented on a change in pull request #10761: feat: implement cache invalidation api
Posted by GitBox <gi...@apache.org>.
bkyryliuk commented on a change in pull request #10761:
URL: https://github.com/apache/incubator-superset/pull/10761#discussion_r486413655
##########
File path: superset/cachekeys/api.py
##########
@@ -0,0 +1,98 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from flask import request, Response
+from flask_appbuilder import expose
+from flask_appbuilder.api import safe
+from flask_appbuilder.models.sqla.interface import SQLAInterface
+from flask_appbuilder.security.decorators import protect
+from jsonschema import ValidationError
+
+from superset.cachekeys.schemas import CacheInvalidationRequestSchema
+from superset.connectors.connector_registry import ConnectorRegistry
+from superset.extensions import cache_manager, db, event_logger
+from superset.models.cache import CacheKey
+from superset.views.base_api import BaseSupersetModelRestApi, statsd_metrics
+
+
+class CacheRestApi(BaseSupersetModelRestApi):
+ datamodel = SQLAInterface(CacheKey)
+ resource_name = "cache"
+ allow_browser_login = True
+ class_permission_name = "CacheRestApi"
+
+ openapi_spec_component_schemas = (CacheInvalidationRequestSchema,)
+
+ @expose("/invalidate", methods=["POST"])
+ @event_logger.log_this
+ @protect()
+ @safe
+ @statsd_metrics
+ def invalidate(self) -> Response:
+ """
+ Takes a list of datasources, finds the associated cache records and
Review comment:
good catch, I can do it in the separate PR if that works for you, would rename it in API & CacheKey model for consistency.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [incubator-superset] dpgaspar commented on a change in pull request #10761: feat: [WIP] implement cache invalidation api
Posted by GitBox <gi...@apache.org>.
dpgaspar commented on a change in pull request #10761:
URL: https://github.com/apache/incubator-superset/pull/10761#discussion_r482152879
##########
File path: superset/cache/api.py
##########
@@ -0,0 +1,91 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from flask import request, Response
+from flask_appbuilder import expose
+from flask_appbuilder.api import safe
+from flask_appbuilder.models.sqla.interface import SQLAInterface
+from flask_appbuilder.security.decorators import protect
+from flask_babel import gettext as _
+from jsonschema import ValidationError
+
+from superset.cache.schemas import CacheInvalidationRequestSchema
+from superset.connectors.connector_registry import ConnectorRegistry
+from superset.extensions import cache_manager, db, event_logger
+from superset.models.cache import CacheKey
+from superset.views.base_api import BaseSupersetModelRestApi, statsd_metrics
+
+
+class CacheRestApi(BaseSupersetModelRestApi):
+ datamodel = SQLAInterface(CacheKey)
+ resource_name = "cache"
+ allow_browser_login = True
+ class_permission_name = "CacheRestApi"
+
Review comment:
use `openapi_spec_component_schemas = (CacheInvalidateRequestSchema,)` to add the schema to the OpenAPI spec
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [incubator-superset] bkyryliuk commented on a change in pull request #10761: feat: implement cache invalidation api
Posted by GitBox <gi...@apache.org>.
bkyryliuk commented on a change in pull request #10761:
URL: https://github.com/apache/incubator-superset/pull/10761#discussion_r486613012
##########
File path: tests/base_tests.py
##########
@@ -69,6 +71,39 @@ def get_resp(
return resp.data.decode("utf-8")
+def post_assert_metric(
Review comment:
yes same code, moved it into this function for reuse in the pytests
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [incubator-superset] bkyryliuk commented on pull request #10761: feat: implement cache invalidation api
Posted by GitBox <gi...@apache.org>.
bkyryliuk commented on pull request #10761:
URL: https://github.com/apache/incubator-superset/pull/10761#issuecomment-692216023

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [incubator-superset] bkyryliuk commented on a change in pull request #10761: feat: implement cache invalidation api
Posted by GitBox <gi...@apache.org>.
bkyryliuk commented on a change in pull request #10761:
URL: https://github.com/apache/incubator-superset/pull/10761#discussion_r488870421
##########
File path: superset/cachekeys/api.py
##########
@@ -0,0 +1,98 @@
+# Licensed to the Apache Software Foundation (ASF) under one
Review comment:
it would be too much struggle to fetch cache keys though the API and then issue deletes for those.
I think we could expose GET / DELETE later once those use cases will arise
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [incubator-superset] dpgaspar commented on a change in pull request #10761: feat: [WIP] implement cache invalidation api
Posted by GitBox <gi...@apache.org>.
dpgaspar commented on a change in pull request #10761:
URL: https://github.com/apache/incubator-superset/pull/10761#discussion_r482154191
##########
File path: superset/cache/api.py
##########
@@ -0,0 +1,91 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from flask import request, Response
+from flask_appbuilder import expose
+from flask_appbuilder.api import safe
+from flask_appbuilder.models.sqla.interface import SQLAInterface
+from flask_appbuilder.security.decorators import protect
+from flask_babel import gettext as _
+from jsonschema import ValidationError
+
+from superset.cache.schemas import CacheInvalidationRequestSchema
+from superset.connectors.connector_registry import ConnectorRegistry
+from superset.extensions import cache_manager, db, event_logger
+from superset.models.cache import CacheKey
+from superset.views.base_api import BaseSupersetModelRestApi, statsd_metrics
+
+
+class CacheRestApi(BaseSupersetModelRestApi):
+ datamodel = SQLAInterface(CacheKey)
+ resource_name = "cache"
+ allow_browser_login = True
+ class_permission_name = "CacheRestApi"
+
+ @expose("/invalidate", methods=["POST"])
+ @event_logger.log_this
+ @protect()
+ @safe
+ @statsd_metrics
+ def invalidate(self) -> Response:
+ """
+ Takes a list of datasources, finds the associated cache records and invalidates them.
+
+ ---
+ post:
+ description: >-
+ Takes a list of datasources, finds the associated cache records and invalidates them.
+ requestBody:
+ description: >-
+ A list of datasources uuid or the tuples of the database and datasource names
+ required: true
+ content:
+ application/json:
+ schema:
+ $ref: "#/components/schemas/CacheInvalidateRequestSchema"
+ responses:
+ 201:
+ $ref: '#/components/responses/201'
+ 400:
+ $ref: '#/components/responses/400'
+ 500:
+ $ref: '#/components/responses/500'
+ """
+ try:
+ datasources = CacheInvalidationRequestSchema().load(request.json)
+ except KeyError:
+ return self.response_400(message="Request is incorrect")
+ except ValidationError as error:
+ return self.response_400(
+ message=_("Request is incorrect: %(error)s", error=error.messages)
Review comment:
nit: Just use: `return self.response_400(message=error.messages)`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [incubator-superset] mistercrunch commented on a change in pull request #10761: feat: implement cache invalidation api
Posted by GitBox <gi...@apache.org>.
mistercrunch commented on a change in pull request #10761:
URL: https://github.com/apache/incubator-superset/pull/10761#discussion_r484002444
##########
File path: superset/cachekeys/api.py
##########
@@ -0,0 +1,98 @@
+# Licensed to the Apache Software Foundation (ASF) under one
Review comment:
I think the route `/api/v1/cachekey/invalidate` is not ideal. I'm struggling to find something good though
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [incubator-superset] bkyryliuk commented on a change in pull request #10761: feat: implement cache invalidation api
Posted by GitBox <gi...@apache.org>.
bkyryliuk commented on a change in pull request #10761:
URL: https://github.com/apache/incubator-superset/pull/10761#discussion_r486693694
##########
File path: tests/cachekeys/api_tests.py
##########
@@ -0,0 +1,131 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+# isort:skip_file
+"""Unit tests for Superset"""
+from tests.test_app import app # noqa
+
+from superset.extensions import cache_manager, db
+from superset.models.cache import CacheKey
+from tests.base_tests import (
+ SupersetTestCase,
+ post_assert_metric,
+ test_client,
+ logged_in_admin,
+) # noqa
+
+
+def test_invalidate_cache(logged_in_admin):
+ rv = post_assert_metric(
+ test_client,
+ "api/v1/cache/invalidate",
+ {"datasource_uids": ["3__table"]},
+ "invalidate",
+ )
+ assert rv.status_code == 201
+
+
+def test_invalidate_existing_cache(logged_in_admin):
+ db.session.add(CacheKey(cache_key="cache_key", datasource_uid="3__table"))
+ db.session.commit()
+ cache_manager.cache.set("cache_key", "value")
+
+ rv = post_assert_metric(
+ test_client,
+ "api/v1/cache/invalidate",
+ {"datasource_uids": ["3__table"]},
+ "invalidate",
+ )
+
+ assert rv.status_code == 201
+ assert cache_manager.cache.get("cache_key") == None
+ assert (
+ not db.session.query(CacheKey).filter(CacheKey.cache_key == "cache_key").first()
+ )
+
+
+def test_invalidate_existing_caches(logged_in_admin):
+ bn = SupersetTestCase.get_birth_names_dataset()
+
+ db.session.add(CacheKey(cache_key="cache_key1", datasource_uid="3__druid"))
+ db.session.add(CacheKey(cache_key="cache_key2", datasource_uid="3__druid"))
+ db.session.add(CacheKey(cache_key="cache_key4", datasource_uid=f"{bn.id}__table"))
+ db.session.add(CacheKey(cache_key="cache_keyX", datasource_uid="X__table"))
+ db.session.commit()
+
+ cache_manager.cache.set("cache_key1", "value")
+ cache_manager.cache.set("cache_key2", "value")
+ cache_manager.cache.set("cache_key4", "value")
+ cache_manager.cache.set("cache_keyX", "value")
+
+ rv = post_assert_metric(
+ test_client,
+ "api/v1/cache/invalidate",
+ {
+ "datasource_uids": ["3__druid", "4__druid"],
Review comment:
done, it caught the bug by the way :) thanks !
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [incubator-superset] codecov-commenter edited a comment on pull request #10761: feat: implement cache invalidation api
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #10761:
URL: https://github.com/apache/incubator-superset/pull/10761#issuecomment-686252702
# [Codecov](https://codecov.io/gh/apache/incubator-superset/pull/10761?src=pr&el=h1) Report
> Merging [#10761](https://codecov.io/gh/apache/incubator-superset/pull/10761?src=pr&el=desc) into [master](https://codecov.io/gh/apache/incubator-superset/commit/8a9ae811d006c886fa43383cf3c351ac3be3010b?el=desc) will **decrease** coverage by `0.32%`.
> The diff coverage is `96.77%`.
[](https://codecov.io/gh/apache/incubator-superset/pull/10761?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #10761 +/- ##
==========================================
- Coverage 61.32% 60.99% -0.33%
==========================================
Files 803 806 +3
Lines 37927 38002 +75
Branches 3561 3561
==========================================
- Hits 23258 23180 -78
- Misses 14483 14636 +153
Partials 186 186
```
| Flag | Coverage Δ | |
|---|---|---|
| #javascript | `61.59% <ø> (+<0.01%)` | :arrow_up: |
| #python | `60.65% <96.77%> (-0.52%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/incubator-superset/pull/10761?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [superset/cachekeys/api.py](https://codecov.io/gh/apache/incubator-superset/pull/10761/diff?src=pr&el=tree#diff-c3VwZXJzZXQvY2FjaGVrZXlzL2FwaS5weQ==) | `95.91% <95.91%> (ø)` | |
| [superset/app.py](https://codecov.io/gh/apache/incubator-superset/pull/10761/diff?src=pr&el=tree#diff-c3VwZXJzZXQvYXBwLnB5) | `80.30% <100.00%> (+0.15%)` | :arrow_up: |
| [superset/cachekeys/schemas.py](https://codecov.io/gh/apache/incubator-superset/pull/10761/diff?src=pr&el=tree#diff-c3VwZXJzZXQvY2FjaGVrZXlzL3NjaGVtYXMucHk=) | `100.00% <100.00%> (ø)` | |
| [superset/charts/schemas.py](https://codecov.io/gh/apache/incubator-superset/pull/10761/diff?src=pr&el=tree#diff-c3VwZXJzZXQvY2hhcnRzL3NjaGVtYXMucHk=) | `100.00% <100.00%> (ø)` | |
| [superset/db\_engines/hive.py](https://codecov.io/gh/apache/incubator-superset/pull/10761/diff?src=pr&el=tree#diff-c3VwZXJzZXQvZGJfZW5naW5lcy9oaXZlLnB5) | `0.00% <0.00%> (-85.72%)` | :arrow_down: |
| [superset/db\_engine\_specs/hive.py](https://codecov.io/gh/apache/incubator-superset/pull/10761/diff?src=pr&el=tree#diff-c3VwZXJzZXQvZGJfZW5naW5lX3NwZWNzL2hpdmUucHk=) | `53.90% <0.00%> (-30.08%)` | :arrow_down: |
| [superset/db\_engine\_specs/presto.py](https://codecov.io/gh/apache/incubator-superset/pull/10761/diff?src=pr&el=tree#diff-c3VwZXJzZXQvZGJfZW5naW5lX3NwZWNzL3ByZXN0by5weQ==) | `70.85% <0.00%> (-10.77%)` | :arrow_down: |
| [superset/connectors/base/views.py](https://codecov.io/gh/apache/incubator-superset/pull/10761/diff?src=pr&el=tree#diff-c3VwZXJzZXQvY29ubmVjdG9ycy9iYXNlL3ZpZXdzLnB5) | `71.42% <0.00%> (-3.58%)` | :arrow_down: |
| [superset/examples/world\_bank.py](https://codecov.io/gh/apache/incubator-superset/pull/10761/diff?src=pr&el=tree#diff-c3VwZXJzZXQvZXhhbXBsZXMvd29ybGRfYmFuay5weQ==) | `97.10% <0.00%> (-2.90%)` | :arrow_down: |
| [superset/examples/birth\_names.py](https://codecov.io/gh/apache/incubator-superset/pull/10761/diff?src=pr&el=tree#diff-c3VwZXJzZXQvZXhhbXBsZXMvYmlydGhfbmFtZXMucHk=) | `97.36% <0.00%> (-2.64%)` | :arrow_down: |
| ... and [15 more](https://codecov.io/gh/apache/incubator-superset/pull/10761/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-superset/pull/10761?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/incubator-superset/pull/10761?src=pr&el=footer). Last update [8a9ae81...115de31](https://codecov.io/gh/apache/incubator-superset/pull/10761?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [incubator-superset] dpgaspar commented on a change in pull request #10761: feat: implement cache invalidation api
Posted by GitBox <gi...@apache.org>.
dpgaspar commented on a change in pull request #10761:
URL: https://github.com/apache/incubator-superset/pull/10761#discussion_r486469842
##########
File path: superset/cachekeys/schemas.py
##########
@@ -0,0 +1,45 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+# RISON/JSON schemas for query parameters
+from marshmallow import fields, Schema, validate
+
+from superset.charts.schemas import (
+ datasource_name_description,
+ datasource_type_description,
+ datasource_uid_description,
+)
+
+
+class Datasource(Schema):
+ database_name = fields.String(description="Datasource name",)
+ datasource_name = fields.String(description=datasource_name_description,)
+ schema = fields.String(description="Datasource schema",)
+ datasource_type = fields.String(
+ description=datasource_type_description,
+ validate=validate.OneOf(choices=("druid", "table", "view")),
+ allow_none=True,
Review comment:
I think that `get_datasource_by_name` will not handle well a `None` here, replace by `required=True`.
##########
File path: superset/cachekeys/api.py
##########
@@ -0,0 +1,98 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from flask import request, Response
+from flask_appbuilder import expose
+from flask_appbuilder.api import safe
+from flask_appbuilder.models.sqla.interface import SQLAInterface
+from flask_appbuilder.security.decorators import protect
+from jsonschema import ValidationError
+
+from superset.cachekeys.schemas import CacheInvalidationRequestSchema
+from superset.connectors.connector_registry import ConnectorRegistry
+from superset.extensions import cache_manager, db, event_logger
+from superset.models.cache import CacheKey
+from superset.views.base_api import BaseSupersetModelRestApi, statsd_metrics
+
+
+class CacheRestApi(BaseSupersetModelRestApi):
+ datamodel = SQLAInterface(CacheKey)
+ resource_name = "cache"
Review comment:
nit: since this was renamed to cache, evaluate if placing everything on `cachekeys` still makes sense
##########
File path: tests/base_tests.py
##########
@@ -69,6 +71,39 @@ def get_resp(
return resp.data.decode("utf-8")
+def post_assert_metric(
Review comment:
Is this the same but on a different place?
##########
File path: tests/cachekeys/api_tests.py
##########
@@ -0,0 +1,131 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+# isort:skip_file
+"""Unit tests for Superset"""
+from tests.test_app import app # noqa
+
+from superset.extensions import cache_manager, db
+from superset.models.cache import CacheKey
+from tests.base_tests import (
+ SupersetTestCase,
+ post_assert_metric,
+ test_client,
+ logged_in_admin,
+) # noqa
+
+
+def test_invalidate_cache(logged_in_admin):
+ rv = post_assert_metric(
+ test_client,
+ "api/v1/cache/invalidate",
+ {"datasource_uids": ["3__table"]},
+ "invalidate",
+ )
+ assert rv.status_code == 201
+
+
+def test_invalidate_existing_cache(logged_in_admin):
+ db.session.add(CacheKey(cache_key="cache_key", datasource_uid="3__table"))
+ db.session.commit()
+ cache_manager.cache.set("cache_key", "value")
+
+ rv = post_assert_metric(
+ test_client,
+ "api/v1/cache/invalidate",
+ {"datasource_uids": ["3__table"]},
+ "invalidate",
+ )
+
+ assert rv.status_code == 201
+ assert cache_manager.cache.get("cache_key") == None
+ assert (
+ not db.session.query(CacheKey).filter(CacheKey.cache_key == "cache_key").first()
+ )
+
+
+def test_invalidate_existing_caches(logged_in_admin):
+ bn = SupersetTestCase.get_birth_names_dataset()
+
+ db.session.add(CacheKey(cache_key="cache_key1", datasource_uid="3__druid"))
+ db.session.add(CacheKey(cache_key="cache_key2", datasource_uid="3__druid"))
+ db.session.add(CacheKey(cache_key="cache_key4", datasource_uid=f"{bn.id}__table"))
+ db.session.add(CacheKey(cache_key="cache_keyX", datasource_uid="X__table"))
+ db.session.commit()
+
+ cache_manager.cache.set("cache_key1", "value")
+ cache_manager.cache.set("cache_key2", "value")
+ cache_manager.cache.set("cache_key4", "value")
+ cache_manager.cache.set("cache_keyX", "value")
+
+ rv = post_assert_metric(
+ test_client,
+ "api/v1/cache/invalidate",
+ {
+ "datasource_uids": ["3__druid", "4__druid"],
Review comment:
can we add a test with edge case POST's: `{ "datasource_uids": [], "datasources": []}`
`{ "datasource_uids": [], "datasources": [{"datasource_name": "", "datasource_type": null}]}`
etc
##########
File path: superset/cachekeys/api.py
##########
@@ -0,0 +1,98 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from flask import request, Response
+from flask_appbuilder import expose
+from flask_appbuilder.api import safe
+from flask_appbuilder.models.sqla.interface import SQLAInterface
+from flask_appbuilder.security.decorators import protect
+from jsonschema import ValidationError
+
+from superset.cachekeys.schemas import CacheInvalidationRequestSchema
+from superset.connectors.connector_registry import ConnectorRegistry
+from superset.extensions import cache_manager, db, event_logger
+from superset.models.cache import CacheKey
+from superset.views.base_api import BaseSupersetModelRestApi, statsd_metrics
+
+
+class CacheRestApi(BaseSupersetModelRestApi):
+ datamodel = SQLAInterface(CacheKey)
+ resource_name = "cache"
+ allow_browser_login = True
+ class_permission_name = "CacheRestApi"
+
+ openapi_spec_component_schemas = (CacheInvalidationRequestSchema,)
+
+ @expose("/invalidate", methods=["POST"])
+ @event_logger.log_this
+ @protect()
+ @safe
+ @statsd_metrics
+ def invalidate(self) -> Response:
+ """
+ Takes a list of datasources, finds the associated cache records and
+ invalidates them and removes the database records
+
+ ---
+ post:
+ description: >-
+ Takes a list of datasources, finds the associated cache records and
+ invalidates them and removes the database records
+ requestBody:
+ description: >-
+ A list of datasources uuid or the tuples of database and datasource names
+ required: true
+ content:
+ application/json:
+ schema:
+ $ref: "#/components/schemas/CacheInvalidationRequestSchema"
+ responses:
+ 201:
+ description: cache was successfully invalidated
+ 400:
+ $ref: '#/components/responses/400'
+ 500:
+ $ref: '#/components/responses/500'
+ """
+ try:
+ datasources = CacheInvalidationRequestSchema().load(request.json)
+ except KeyError:
+ return self.response_400(message="Request is incorrect")
+ except ValidationError as error:
+ return self.response_400(message=error.message)
+ datasource_uids = set(datasources.get("datasource_uids", []))
+ for ds in datasources.get("datasources", []):
+ ds_obj = ConnectorRegistry.get_datasource_by_name(
+ session=db.session,
+ datasource_type=ds.get("datasource_type"),
+ datasource_name=ds.get("datasource_name"),
+ schema=ds.get("schema"),
+ database_name=ds.get("database_name"),
+ )
+ if ds_obj:
+ datasource_uids.add(ds_obj.uid)
+
+ cache_keys = (
+ db.session.query(CacheKey)
+ .filter(CacheKey.datasource_uid.in_(datasource_uids))
+ .all()
+ )
+ if cache_keys:
+ cache_manager.cache.delete_many(*[c.cache_key for c in cache_keys])
Review comment:
A couple of questions:
- if a cache `delete_many` fails, we may end up with delete keys on the cache that still exist on `CacheKey` will a next run of `delete_many` fail if it can't find a key?
- the same applies if a delete `CacheKey` fails, would also be safer to try/except and rollback
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org