You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@superset.apache.org by GitBox <gi...@apache.org> on 2020/09/02 05:05:09 UTC

[GitHub] [incubator-superset] bkyryliuk opened a new pull request #10761: feat: [WIP] implement cache invalidation api

bkyryliuk opened a new pull request #10761:
URL: https://github.com/apache/incubator-superset/pull/10761


   ### SUMMARY
   <!--- Describe the change below, including rationale and design decisions -->
   
   ### BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF
   <!--- Skip this if not applicable -->
   
   ### TEST PLAN
   <!--- What steps should be taken to verify the changes -->
   
   ### ADDITIONAL INFORMATION
   <!--- Check any relevant boxes with "x" -->
   <!--- HINT: Include "Fixes #nnn" if you are fixing an existing issue -->
   - [ ] Has associated issue:
   - [ ] Changes UI
   - [ ] Requires DB Migration.
   - [ ] Confirm DB Migration upgrade and downgrade tested.
   - [ ] Introduces new feature or API
   - [ ] Removes existing feature or API
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


[GitHub] [incubator-superset] bkyryliuk commented on a change in pull request #10761: feat: implement cache invalidation api

Posted by GitBox <gi...@apache.org>.
bkyryliuk commented on a change in pull request #10761:
URL: https://github.com/apache/incubator-superset/pull/10761#discussion_r487139370



##########
File path: superset/cachekeys/schemas.py
##########
@@ -0,0 +1,45 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+# RISON/JSON schemas for query parameters
+from marshmallow import fields, Schema, validate
+
+from superset.charts.schemas import (
+    datasource_name_description,
+    datasource_type_description,
+    datasource_uid_description,
+)
+
+
+class Datasource(Schema):
+    database_name = fields.String(description="Datasource name",)
+    datasource_name = fields.String(description=datasource_name_description,)
+    schema = fields.String(description="Datasource schema",)
+    datasource_type = fields.String(
+        description=datasource_type_description,
+        validate=validate.OneOf(choices=("druid", "table", "view")),
+        allow_none=True,

Review comment:
       done




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


[GitHub] [incubator-superset] codecov-commenter edited a comment on pull request #10761: feat: [WIP] implement cache invalidation api

Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #10761:
URL: https://github.com/apache/incubator-superset/pull/10761#issuecomment-686252702


   # [Codecov](https://codecov.io/gh/apache/incubator-superset/pull/10761?src=pr&el=h1) Report
   > Merging [#10761](https://codecov.io/gh/apache/incubator-superset/pull/10761?src=pr&el=desc) into [master](https://codecov.io/gh/apache/incubator-superset/commit/f6858256f496387146c4b535436bedaeceeeee51?el=desc) will **increase** coverage by `0.37%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/incubator-superset/pull/10761/graphs/tree.svg?width=650&height=150&src=pr&token=KsB0fHcx6l)](https://codecov.io/gh/apache/incubator-superset/pull/10761?src=pr&el=tree)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master   #10761      +/-   ##
   ==========================================
   + Coverage   61.22%   61.60%   +0.37%     
   ==========================================
     Files         802      429     -373     
     Lines       37814    13974   -23840     
     Branches     3555     3555              
   ==========================================
   - Hits        23153     8608   -14545     
   + Misses      14475     5180    -9295     
     Partials      186      186              
   ```
   
   | Flag | Coverage Δ | |
   |---|---|---|
   | #javascript | `61.60% <ø> (ø)` | |
   | #python | `?` | |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   | [Impacted Files](https://codecov.io/gh/apache/incubator-superset/pull/10761?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [.../migrations/versions/1a1d627ebd8e\_position\_json.py](https://codecov.io/gh/apache/incubator-superset/pull/10761/diff?src=pr&el=tree#diff-c3VwZXJzZXQvbWlncmF0aW9ucy92ZXJzaW9ucy8xYTFkNjI3ZWJkOGVfcG9zaXRpb25fanNvbi5weQ==) | | |
   | [...7bf4f117f9\_adding\_extra\_field\_to\_database\_model.py](https://codecov.io/gh/apache/incubator-superset/pull/10761/diff?src=pr&el=tree#diff-c3VwZXJzZXQvbWlncmF0aW9ucy92ZXJzaW9ucy84NjdiZjRmMTE3ZjlfYWRkaW5nX2V4dHJhX2ZpZWxkX3RvX2RhdGFiYXNlX21vZGVsLnB5) | | |
   | [superset/models/sql\_types/presto\_sql\_types.py](https://codecov.io/gh/apache/incubator-superset/pull/10761/diff?src=pr&el=tree#diff-c3VwZXJzZXQvbW9kZWxzL3NxbF90eXBlcy9wcmVzdG9fc3FsX3R5cGVzLnB5) | | |
   | [superset/db\_engine\_specs/pinot.py](https://codecov.io/gh/apache/incubator-superset/pull/10761/diff?src=pr&el=tree#diff-c3VwZXJzZXQvZGJfZW5naW5lX3NwZWNzL3Bpbm90LnB5) | | |
   | [...rsions/a9c47e2c1547\_add\_impersonate\_user\_to\_dbs.py](https://codecov.io/gh/apache/incubator-superset/pull/10761/diff?src=pr&el=tree#diff-c3VwZXJzZXQvbWlncmF0aW9ucy92ZXJzaW9ucy9hOWM0N2UyYzE1NDdfYWRkX2ltcGVyc29uYXRlX3VzZXJfdG9fZGJzLnB5) | | |
   | [superset/stats\_logger.py](https://codecov.io/gh/apache/incubator-superset/pull/10761/diff?src=pr&el=tree#diff-c3VwZXJzZXQvc3RhdHNfbG9nZ2VyLnB5) | | |
   | [...rset/migrations/versions/65903709c321\_allow\_dml.py](https://codecov.io/gh/apache/incubator-superset/pull/10761/diff?src=pr&el=tree#diff-c3VwZXJzZXQvbWlncmF0aW9ucy92ZXJzaW9ucy82NTkwMzcwOWMzMjFfYWxsb3dfZG1sLnB5) | | |
   | [superset/dashboards/commands/create.py](https://codecov.io/gh/apache/incubator-superset/pull/10761/diff?src=pr&el=tree#diff-c3VwZXJzZXQvZGFzaGJvYXJkcy9jb21tYW5kcy9jcmVhdGUucHk=) | | |
   | [superset/models/schedules.py](https://codecov.io/gh/apache/incubator-superset/pull/10761/diff?src=pr&el=tree#diff-c3VwZXJzZXQvbW9kZWxzL3NjaGVkdWxlcy5weQ==) | | |
   | [superset/dashboards/commands/delete.py](https://codecov.io/gh/apache/incubator-superset/pull/10761/diff?src=pr&el=tree#diff-c3VwZXJzZXQvZGFzaGJvYXJkcy9jb21tYW5kcy9kZWxldGUucHk=) | | |
   | ... and [361 more](https://codecov.io/gh/apache/incubator-superset/pull/10761/diff?src=pr&el=tree-more) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-superset/pull/10761?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/incubator-superset/pull/10761?src=pr&el=footer). Last update [f685825...47eb457](https://codecov.io/gh/apache/incubator-superset/pull/10761?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


[GitHub] [incubator-superset] bkyryliuk commented on a change in pull request #10761: feat: implement cache invalidation api

Posted by GitBox <gi...@apache.org>.
bkyryliuk commented on a change in pull request #10761:
URL: https://github.com/apache/incubator-superset/pull/10761#discussion_r486412102



##########
File path: superset/cachekeys/api.py
##########
@@ -0,0 +1,98 @@
+# Licensed to the Apache Software Foundation (ASF) under one

Review comment:
       agree, I had the same struggles, definitely open to suggestions. 
   /api/v1/cache/invalidate could be slightly better




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


[GitHub] [incubator-superset] bkyryliuk commented on a change in pull request #10761: feat: implement cache invalidation api

Posted by GitBox <gi...@apache.org>.
bkyryliuk commented on a change in pull request #10761:
URL: https://github.com/apache/incubator-superset/pull/10761#discussion_r487139156



##########
File path: superset/cachekeys/api.py
##########
@@ -0,0 +1,98 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from flask import request, Response
+from flask_appbuilder import expose
+from flask_appbuilder.api import safe
+from flask_appbuilder.models.sqla.interface import SQLAInterface
+from flask_appbuilder.security.decorators import protect
+from jsonschema import ValidationError
+
+from superset.cachekeys.schemas import CacheInvalidationRequestSchema
+from superset.connectors.connector_registry import ConnectorRegistry
+from superset.extensions import cache_manager, db, event_logger
+from superset.models.cache import CacheKey
+from superset.views.base_api import BaseSupersetModelRestApi, statsd_metrics
+
+
+class CacheRestApi(BaseSupersetModelRestApi):
+    datamodel = SQLAInterface(CacheKey)
+    resource_name = "cache"
+    allow_browser_login = True
+    class_permission_name = "CacheRestApi"
+
+    openapi_spec_component_schemas = (CacheInvalidationRequestSchema,)
+
+    @expose("/invalidate", methods=["POST"])
+    @event_logger.log_this
+    @protect()
+    @safe
+    @statsd_metrics
+    def invalidate(self) -> Response:
+        """
+        Takes a list of datasources, finds the associated cache records and
+        invalidates them and removes the database records
+
+        ---
+        post:
+          description: >-
+            Takes a list of datasources, finds the associated cache records and
+            invalidates them and removes the database records
+          requestBody:
+            description: >-
+              A list of datasources uuid or the tuples of database and datasource names
+            required: true
+            content:
+              application/json:
+                schema:
+                  $ref: "#/components/schemas/CacheInvalidationRequestSchema"
+          responses:
+            201:
+              description: cache was successfully invalidated
+            400:
+              $ref: '#/components/responses/400'
+            500:
+              $ref: '#/components/responses/500'
+        """
+        try:
+            datasources = CacheInvalidationRequestSchema().load(request.json)
+        except KeyError:
+            return self.response_400(message="Request is incorrect")
+        except ValidationError as error:
+            return self.response_400(message=error.message)
+        datasource_uids = set(datasources.get("datasource_uids", []))
+        for ds in datasources.get("datasources", []):
+            ds_obj = ConnectorRegistry.get_datasource_by_name(
+                session=db.session,
+                datasource_type=ds.get("datasource_type"),
+                datasource_name=ds.get("datasource_name"),
+                schema=ds.get("schema"),
+                database_name=ds.get("database_name"),
+            )
+            if ds_obj:
+                datasource_uids.add(ds_obj.uid)
+
+        cache_keys = (
+            db.session.query(CacheKey)
+            .filter(CacheKey.datasource_uid.in_(datasource_uids))
+            .all()
+        )
+        if cache_keys:
+            cache_manager.cache.delete_many(*[c.cache_key for c in cache_keys])

Review comment:
       delete_many doesn't fail, only returns true / false if the keys were deleted. e.g. if the key was not found it will return false. Updated CacheKey logic to delete is a single statement & rollback if fails -> that would be more robust.
   
   deleting case & keeping records is not too bad as there still a need to garbage collect those and cache is not a persistent store.

##########
File path: superset/cachekeys/schemas.py
##########
@@ -0,0 +1,45 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+# RISON/JSON schemas for query parameters
+from marshmallow import fields, Schema, validate
+
+from superset.charts.schemas import (
+    datasource_name_description,
+    datasource_type_description,
+    datasource_uid_description,
+)
+
+
+class Datasource(Schema):
+    database_name = fields.String(description="Datasource name",)
+    datasource_name = fields.String(description=datasource_name_description,)
+    schema = fields.String(description="Datasource schema",)
+    datasource_type = fields.String(
+        description=datasource_type_description,
+        validate=validate.OneOf(choices=("druid", "table", "view")),
+        allow_none=True,

Review comment:
       done

##########
File path: superset/cachekeys/api.py
##########
@@ -0,0 +1,98 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from flask import request, Response
+from flask_appbuilder import expose
+from flask_appbuilder.api import safe
+from flask_appbuilder.models.sqla.interface import SQLAInterface
+from flask_appbuilder.security.decorators import protect
+from jsonschema import ValidationError
+
+from superset.cachekeys.schemas import CacheInvalidationRequestSchema
+from superset.connectors.connector_registry import ConnectorRegistry
+from superset.extensions import cache_manager, db, event_logger
+from superset.models.cache import CacheKey
+from superset.views.base_api import BaseSupersetModelRestApi, statsd_metrics
+
+
+class CacheRestApi(BaseSupersetModelRestApi):
+    datamodel = SQLAInterface(CacheKey)
+    resource_name = "cache"
+    allow_browser_login = True
+    class_permission_name = "CacheRestApi"
+
+    openapi_spec_component_schemas = (CacheInvalidationRequestSchema,)
+
+    @expose("/invalidate", methods=["POST"])
+    @event_logger.log_this
+    @protect()
+    @safe
+    @statsd_metrics
+    def invalidate(self) -> Response:
+        """
+        Takes a list of datasources, finds the associated cache records and
+        invalidates them and removes the database records
+
+        ---
+        post:
+          description: >-
+            Takes a list of datasources, finds the associated cache records and
+            invalidates them and removes the database records
+          requestBody:
+            description: >-
+              A list of datasources uuid or the tuples of database and datasource names
+            required: true
+            content:
+              application/json:
+                schema:
+                  $ref: "#/components/schemas/CacheInvalidationRequestSchema"
+          responses:
+            201:
+              description: cache was successfully invalidated
+            400:
+              $ref: '#/components/responses/400'
+            500:
+              $ref: '#/components/responses/500'
+        """
+        try:
+            datasources = CacheInvalidationRequestSchema().load(request.json)
+        except KeyError:
+            return self.response_400(message="Request is incorrect")
+        except ValidationError as error:
+            return self.response_400(message=error.message)
+        datasource_uids = set(datasources.get("datasource_uids", []))
+        for ds in datasources.get("datasources", []):
+            ds_obj = ConnectorRegistry.get_datasource_by_name(
+                session=db.session,
+                datasource_type=ds.get("datasource_type"),
+                datasource_name=ds.get("datasource_name"),
+                schema=ds.get("schema"),
+                database_name=ds.get("database_name"),
+            )
+            if ds_obj:
+                datasource_uids.add(ds_obj.uid)
+
+        cache_keys = (
+            db.session.query(CacheKey)
+            .filter(CacheKey.datasource_uid.in_(datasource_uids))
+            .all()
+        )
+        if cache_keys:
+            cache_manager.cache.delete_many(*[c.cache_key for c in cache_keys])

Review comment:
       delete_many doesn't fail, only returns true / false if the keys were deleted. e.g. if the key was not found it will return false. Updated CacheKey logic to delete is a single statement & rollback if fails -> that would be more robust.
   
   deleting case & keeping records is not too bad as there still a need to garbage collect those and cache is not a persistent store.

##########
File path: superset/cachekeys/schemas.py
##########
@@ -0,0 +1,45 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+# RISON/JSON schemas for query parameters
+from marshmallow import fields, Schema, validate
+
+from superset.charts.schemas import (
+    datasource_name_description,
+    datasource_type_description,
+    datasource_uid_description,
+)
+
+
+class Datasource(Schema):
+    database_name = fields.String(description="Datasource name",)
+    datasource_name = fields.String(description=datasource_name_description,)
+    schema = fields.String(description="Datasource schema",)
+    datasource_type = fields.String(
+        description=datasource_type_description,
+        validate=validate.OneOf(choices=("druid", "table", "view")),
+        allow_none=True,

Review comment:
       done

##########
File path: superset/cachekeys/api.py
##########
@@ -0,0 +1,98 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from flask import request, Response
+from flask_appbuilder import expose
+from flask_appbuilder.api import safe
+from flask_appbuilder.models.sqla.interface import SQLAInterface
+from flask_appbuilder.security.decorators import protect
+from jsonschema import ValidationError
+
+from superset.cachekeys.schemas import CacheInvalidationRequestSchema
+from superset.connectors.connector_registry import ConnectorRegistry
+from superset.extensions import cache_manager, db, event_logger
+from superset.models.cache import CacheKey
+from superset.views.base_api import BaseSupersetModelRestApi, statsd_metrics
+
+
+class CacheRestApi(BaseSupersetModelRestApi):
+    datamodel = SQLAInterface(CacheKey)
+    resource_name = "cache"
+    allow_browser_login = True
+    class_permission_name = "CacheRestApi"
+
+    openapi_spec_component_schemas = (CacheInvalidationRequestSchema,)
+
+    @expose("/invalidate", methods=["POST"])
+    @event_logger.log_this
+    @protect()
+    @safe
+    @statsd_metrics
+    def invalidate(self) -> Response:
+        """
+        Takes a list of datasources, finds the associated cache records and
+        invalidates them and removes the database records
+
+        ---
+        post:
+          description: >-
+            Takes a list of datasources, finds the associated cache records and
+            invalidates them and removes the database records
+          requestBody:
+            description: >-
+              A list of datasources uuid or the tuples of database and datasource names
+            required: true
+            content:
+              application/json:
+                schema:
+                  $ref: "#/components/schemas/CacheInvalidationRequestSchema"
+          responses:
+            201:
+              description: cache was successfully invalidated
+            400:
+              $ref: '#/components/responses/400'
+            500:
+              $ref: '#/components/responses/500'
+        """
+        try:
+            datasources = CacheInvalidationRequestSchema().load(request.json)
+        except KeyError:
+            return self.response_400(message="Request is incorrect")
+        except ValidationError as error:
+            return self.response_400(message=error.message)
+        datasource_uids = set(datasources.get("datasource_uids", []))
+        for ds in datasources.get("datasources", []):
+            ds_obj = ConnectorRegistry.get_datasource_by_name(
+                session=db.session,
+                datasource_type=ds.get("datasource_type"),
+                datasource_name=ds.get("datasource_name"),
+                schema=ds.get("schema"),
+                database_name=ds.get("database_name"),
+            )
+            if ds_obj:
+                datasource_uids.add(ds_obj.uid)
+
+        cache_keys = (
+            db.session.query(CacheKey)
+            .filter(CacheKey.datasource_uid.in_(datasource_uids))
+            .all()
+        )
+        if cache_keys:
+            cache_manager.cache.delete_many(*[c.cache_key for c in cache_keys])

Review comment:
       delete_many doesn't fail, only returns true / false if the keys were deleted. e.g. if the key was not found it will return false. Updated CacheKey logic to delete is a single statement & rollback if fails -> that would be more robust.
   
   deleting case & keeping records is not too bad as there still a need to garbage collect those and cache is not a persistent store.

##########
File path: superset/cachekeys/schemas.py
##########
@@ -0,0 +1,45 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+# RISON/JSON schemas for query parameters
+from marshmallow import fields, Schema, validate
+
+from superset.charts.schemas import (
+    datasource_name_description,
+    datasource_type_description,
+    datasource_uid_description,
+)
+
+
+class Datasource(Schema):
+    database_name = fields.String(description="Datasource name",)
+    datasource_name = fields.String(description=datasource_name_description,)
+    schema = fields.String(description="Datasource schema",)
+    datasource_type = fields.String(
+        description=datasource_type_description,
+        validate=validate.OneOf(choices=("druid", "table", "view")),
+        allow_none=True,

Review comment:
       done

##########
File path: superset/cachekeys/api.py
##########
@@ -0,0 +1,98 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from flask import request, Response
+from flask_appbuilder import expose
+from flask_appbuilder.api import safe
+from flask_appbuilder.models.sqla.interface import SQLAInterface
+from flask_appbuilder.security.decorators import protect
+from jsonschema import ValidationError
+
+from superset.cachekeys.schemas import CacheInvalidationRequestSchema
+from superset.connectors.connector_registry import ConnectorRegistry
+from superset.extensions import cache_manager, db, event_logger
+from superset.models.cache import CacheKey
+from superset.views.base_api import BaseSupersetModelRestApi, statsd_metrics
+
+
+class CacheRestApi(BaseSupersetModelRestApi):
+    datamodel = SQLAInterface(CacheKey)
+    resource_name = "cache"
+    allow_browser_login = True
+    class_permission_name = "CacheRestApi"
+
+    openapi_spec_component_schemas = (CacheInvalidationRequestSchema,)
+
+    @expose("/invalidate", methods=["POST"])
+    @event_logger.log_this
+    @protect()
+    @safe
+    @statsd_metrics
+    def invalidate(self) -> Response:
+        """
+        Takes a list of datasources, finds the associated cache records and
+        invalidates them and removes the database records
+
+        ---
+        post:
+          description: >-
+            Takes a list of datasources, finds the associated cache records and
+            invalidates them and removes the database records
+          requestBody:
+            description: >-
+              A list of datasources uuid or the tuples of database and datasource names
+            required: true
+            content:
+              application/json:
+                schema:
+                  $ref: "#/components/schemas/CacheInvalidationRequestSchema"
+          responses:
+            201:
+              description: cache was successfully invalidated
+            400:
+              $ref: '#/components/responses/400'
+            500:
+              $ref: '#/components/responses/500'
+        """
+        try:
+            datasources = CacheInvalidationRequestSchema().load(request.json)
+        except KeyError:
+            return self.response_400(message="Request is incorrect")
+        except ValidationError as error:
+            return self.response_400(message=error.message)
+        datasource_uids = set(datasources.get("datasource_uids", []))
+        for ds in datasources.get("datasources", []):
+            ds_obj = ConnectorRegistry.get_datasource_by_name(
+                session=db.session,
+                datasource_type=ds.get("datasource_type"),
+                datasource_name=ds.get("datasource_name"),
+                schema=ds.get("schema"),
+                database_name=ds.get("database_name"),
+            )
+            if ds_obj:
+                datasource_uids.add(ds_obj.uid)
+
+        cache_keys = (
+            db.session.query(CacheKey)
+            .filter(CacheKey.datasource_uid.in_(datasource_uids))
+            .all()
+        )
+        if cache_keys:
+            cache_manager.cache.delete_many(*[c.cache_key for c in cache_keys])

Review comment:
       delete_many doesn't fail, only returns true / false if the keys were deleted. e.g. if the key was not found it will return false. Updated CacheKey logic to delete is a single statement & rollback if fails -> that would be more robust.
   
   deleting case & keeping records is not too bad as there still a need to garbage collect those and cache is not a persistent store.

##########
File path: superset/cachekeys/schemas.py
##########
@@ -0,0 +1,45 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+# RISON/JSON schemas for query parameters
+from marshmallow import fields, Schema, validate
+
+from superset.charts.schemas import (
+    datasource_name_description,
+    datasource_type_description,
+    datasource_uid_description,
+)
+
+
+class Datasource(Schema):
+    database_name = fields.String(description="Datasource name",)
+    datasource_name = fields.String(description=datasource_name_description,)
+    schema = fields.String(description="Datasource schema",)
+    datasource_type = fields.String(
+        description=datasource_type_description,
+        validate=validate.OneOf(choices=("druid", "table", "view")),
+        allow_none=True,

Review comment:
       done

##########
File path: superset/cachekeys/api.py
##########
@@ -0,0 +1,98 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from flask import request, Response
+from flask_appbuilder import expose
+from flask_appbuilder.api import safe
+from flask_appbuilder.models.sqla.interface import SQLAInterface
+from flask_appbuilder.security.decorators import protect
+from jsonschema import ValidationError
+
+from superset.cachekeys.schemas import CacheInvalidationRequestSchema
+from superset.connectors.connector_registry import ConnectorRegistry
+from superset.extensions import cache_manager, db, event_logger
+from superset.models.cache import CacheKey
+from superset.views.base_api import BaseSupersetModelRestApi, statsd_metrics
+
+
+class CacheRestApi(BaseSupersetModelRestApi):
+    datamodel = SQLAInterface(CacheKey)
+    resource_name = "cache"
+    allow_browser_login = True
+    class_permission_name = "CacheRestApi"
+
+    openapi_spec_component_schemas = (CacheInvalidationRequestSchema,)
+
+    @expose("/invalidate", methods=["POST"])
+    @event_logger.log_this
+    @protect()
+    @safe
+    @statsd_metrics
+    def invalidate(self) -> Response:
+        """
+        Takes a list of datasources, finds the associated cache records and
+        invalidates them and removes the database records
+
+        ---
+        post:
+          description: >-
+            Takes a list of datasources, finds the associated cache records and
+            invalidates them and removes the database records
+          requestBody:
+            description: >-
+              A list of datasources uuid or the tuples of database and datasource names
+            required: true
+            content:
+              application/json:
+                schema:
+                  $ref: "#/components/schemas/CacheInvalidationRequestSchema"
+          responses:
+            201:
+              description: cache was successfully invalidated
+            400:
+              $ref: '#/components/responses/400'
+            500:
+              $ref: '#/components/responses/500'
+        """
+        try:
+            datasources = CacheInvalidationRequestSchema().load(request.json)
+        except KeyError:
+            return self.response_400(message="Request is incorrect")
+        except ValidationError as error:
+            return self.response_400(message=error.message)
+        datasource_uids = set(datasources.get("datasource_uids", []))
+        for ds in datasources.get("datasources", []):
+            ds_obj = ConnectorRegistry.get_datasource_by_name(
+                session=db.session,
+                datasource_type=ds.get("datasource_type"),
+                datasource_name=ds.get("datasource_name"),
+                schema=ds.get("schema"),
+                database_name=ds.get("database_name"),
+            )
+            if ds_obj:
+                datasource_uids.add(ds_obj.uid)
+
+        cache_keys = (
+            db.session.query(CacheKey)
+            .filter(CacheKey.datasource_uid.in_(datasource_uids))
+            .all()
+        )
+        if cache_keys:
+            cache_manager.cache.delete_many(*[c.cache_key for c in cache_keys])

Review comment:
       delete_many doesn't fail, only returns true / false if the keys were deleted. e.g. if the key was not found it will return false. Updated CacheKey logic to delete is a single statement & rollback if fails -> that would be more robust.
   
   deleting case & keeping records is not too bad as there still a need to garbage collect those and cache is not a persistent store.

##########
File path: superset/cachekeys/schemas.py
##########
@@ -0,0 +1,45 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+# RISON/JSON schemas for query parameters
+from marshmallow import fields, Schema, validate
+
+from superset.charts.schemas import (
+    datasource_name_description,
+    datasource_type_description,
+    datasource_uid_description,
+)
+
+
+class Datasource(Schema):
+    database_name = fields.String(description="Datasource name",)
+    datasource_name = fields.String(description=datasource_name_description,)
+    schema = fields.String(description="Datasource schema",)
+    datasource_type = fields.String(
+        description=datasource_type_description,
+        validate=validate.OneOf(choices=("druid", "table", "view")),
+        allow_none=True,

Review comment:
       done

##########
File path: superset/cachekeys/api.py
##########
@@ -0,0 +1,98 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from flask import request, Response
+from flask_appbuilder import expose
+from flask_appbuilder.api import safe
+from flask_appbuilder.models.sqla.interface import SQLAInterface
+from flask_appbuilder.security.decorators import protect
+from jsonschema import ValidationError
+
+from superset.cachekeys.schemas import CacheInvalidationRequestSchema
+from superset.connectors.connector_registry import ConnectorRegistry
+from superset.extensions import cache_manager, db, event_logger
+from superset.models.cache import CacheKey
+from superset.views.base_api import BaseSupersetModelRestApi, statsd_metrics
+
+
+class CacheRestApi(BaseSupersetModelRestApi):
+    datamodel = SQLAInterface(CacheKey)
+    resource_name = "cache"
+    allow_browser_login = True
+    class_permission_name = "CacheRestApi"
+
+    openapi_spec_component_schemas = (CacheInvalidationRequestSchema,)
+
+    @expose("/invalidate", methods=["POST"])
+    @event_logger.log_this
+    @protect()
+    @safe
+    @statsd_metrics
+    def invalidate(self) -> Response:
+        """
+        Takes a list of datasources, finds the associated cache records and
+        invalidates them and removes the database records
+
+        ---
+        post:
+          description: >-
+            Takes a list of datasources, finds the associated cache records and
+            invalidates them and removes the database records
+          requestBody:
+            description: >-
+              A list of datasources uuid or the tuples of database and datasource names
+            required: true
+            content:
+              application/json:
+                schema:
+                  $ref: "#/components/schemas/CacheInvalidationRequestSchema"
+          responses:
+            201:
+              description: cache was successfully invalidated
+            400:
+              $ref: '#/components/responses/400'
+            500:
+              $ref: '#/components/responses/500'
+        """
+        try:
+            datasources = CacheInvalidationRequestSchema().load(request.json)
+        except KeyError:
+            return self.response_400(message="Request is incorrect")
+        except ValidationError as error:
+            return self.response_400(message=error.message)
+        datasource_uids = set(datasources.get("datasource_uids", []))
+        for ds in datasources.get("datasources", []):
+            ds_obj = ConnectorRegistry.get_datasource_by_name(
+                session=db.session,
+                datasource_type=ds.get("datasource_type"),
+                datasource_name=ds.get("datasource_name"),
+                schema=ds.get("schema"),
+                database_name=ds.get("database_name"),
+            )
+            if ds_obj:
+                datasource_uids.add(ds_obj.uid)
+
+        cache_keys = (
+            db.session.query(CacheKey)
+            .filter(CacheKey.datasource_uid.in_(datasource_uids))
+            .all()
+        )
+        if cache_keys:
+            cache_manager.cache.delete_many(*[c.cache_key for c in cache_keys])

Review comment:
       delete_many doesn't fail, only returns true / false if the keys were deleted. e.g. if the key was not found it will return false. Updated CacheKey logic to delete is a single statement & rollback if fails -> that would be more robust.
   
   deleting case & keeping records is not too bad as there still a need to garbage collect those and cache is not a persistent store.

##########
File path: superset/cachekeys/schemas.py
##########
@@ -0,0 +1,45 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+# RISON/JSON schemas for query parameters
+from marshmallow import fields, Schema, validate
+
+from superset.charts.schemas import (
+    datasource_name_description,
+    datasource_type_description,
+    datasource_uid_description,
+)
+
+
+class Datasource(Schema):
+    database_name = fields.String(description="Datasource name",)
+    datasource_name = fields.String(description=datasource_name_description,)
+    schema = fields.String(description="Datasource schema",)
+    datasource_type = fields.String(
+        description=datasource_type_description,
+        validate=validate.OneOf(choices=("druid", "table", "view")),
+        allow_none=True,

Review comment:
       done

##########
File path: superset/cachekeys/api.py
##########
@@ -0,0 +1,98 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from flask import request, Response
+from flask_appbuilder import expose
+from flask_appbuilder.api import safe
+from flask_appbuilder.models.sqla.interface import SQLAInterface
+from flask_appbuilder.security.decorators import protect
+from jsonschema import ValidationError
+
+from superset.cachekeys.schemas import CacheInvalidationRequestSchema
+from superset.connectors.connector_registry import ConnectorRegistry
+from superset.extensions import cache_manager, db, event_logger
+from superset.models.cache import CacheKey
+from superset.views.base_api import BaseSupersetModelRestApi, statsd_metrics
+
+
+class CacheRestApi(BaseSupersetModelRestApi):
+    datamodel = SQLAInterface(CacheKey)
+    resource_name = "cache"
+    allow_browser_login = True
+    class_permission_name = "CacheRestApi"
+
+    openapi_spec_component_schemas = (CacheInvalidationRequestSchema,)
+
+    @expose("/invalidate", methods=["POST"])
+    @event_logger.log_this
+    @protect()
+    @safe
+    @statsd_metrics
+    def invalidate(self) -> Response:
+        """
+        Takes a list of datasources, finds the associated cache records and
+        invalidates them and removes the database records
+
+        ---
+        post:
+          description: >-
+            Takes a list of datasources, finds the associated cache records and
+            invalidates them and removes the database records
+          requestBody:
+            description: >-
+              A list of datasources uuid or the tuples of database and datasource names
+            required: true
+            content:
+              application/json:
+                schema:
+                  $ref: "#/components/schemas/CacheInvalidationRequestSchema"
+          responses:
+            201:
+              description: cache was successfully invalidated
+            400:
+              $ref: '#/components/responses/400'
+            500:
+              $ref: '#/components/responses/500'
+        """
+        try:
+            datasources = CacheInvalidationRequestSchema().load(request.json)
+        except KeyError:
+            return self.response_400(message="Request is incorrect")
+        except ValidationError as error:
+            return self.response_400(message=error.message)
+        datasource_uids = set(datasources.get("datasource_uids", []))
+        for ds in datasources.get("datasources", []):
+            ds_obj = ConnectorRegistry.get_datasource_by_name(
+                session=db.session,
+                datasource_type=ds.get("datasource_type"),
+                datasource_name=ds.get("datasource_name"),
+                schema=ds.get("schema"),
+                database_name=ds.get("database_name"),
+            )
+            if ds_obj:
+                datasource_uids.add(ds_obj.uid)
+
+        cache_keys = (
+            db.session.query(CacheKey)
+            .filter(CacheKey.datasource_uid.in_(datasource_uids))
+            .all()
+        )
+        if cache_keys:
+            cache_manager.cache.delete_many(*[c.cache_key for c in cache_keys])

Review comment:
       delete_many doesn't fail, only returns true / false if the keys were deleted. e.g. if the key was not found it will return false. Updated CacheKey logic to delete is a single statement & rollback if fails -> that would be more robust.
   
   deleting case & keeping records is not too bad as there still a need to garbage collect those and cache is not a persistent store.

##########
File path: superset/cachekeys/schemas.py
##########
@@ -0,0 +1,45 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+# RISON/JSON schemas for query parameters
+from marshmallow import fields, Schema, validate
+
+from superset.charts.schemas import (
+    datasource_name_description,
+    datasource_type_description,
+    datasource_uid_description,
+)
+
+
+class Datasource(Schema):
+    database_name = fields.String(description="Datasource name",)
+    datasource_name = fields.String(description=datasource_name_description,)
+    schema = fields.String(description="Datasource schema",)
+    datasource_type = fields.String(
+        description=datasource_type_description,
+        validate=validate.OneOf(choices=("druid", "table", "view")),
+        allow_none=True,

Review comment:
       done

##########
File path: superset/cachekeys/api.py
##########
@@ -0,0 +1,98 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from flask import request, Response
+from flask_appbuilder import expose
+from flask_appbuilder.api import safe
+from flask_appbuilder.models.sqla.interface import SQLAInterface
+from flask_appbuilder.security.decorators import protect
+from jsonschema import ValidationError
+
+from superset.cachekeys.schemas import CacheInvalidationRequestSchema
+from superset.connectors.connector_registry import ConnectorRegistry
+from superset.extensions import cache_manager, db, event_logger
+from superset.models.cache import CacheKey
+from superset.views.base_api import BaseSupersetModelRestApi, statsd_metrics
+
+
+class CacheRestApi(BaseSupersetModelRestApi):
+    datamodel = SQLAInterface(CacheKey)
+    resource_name = "cache"
+    allow_browser_login = True
+    class_permission_name = "CacheRestApi"
+
+    openapi_spec_component_schemas = (CacheInvalidationRequestSchema,)
+
+    @expose("/invalidate", methods=["POST"])
+    @event_logger.log_this
+    @protect()
+    @safe
+    @statsd_metrics
+    def invalidate(self) -> Response:
+        """
+        Takes a list of datasources, finds the associated cache records and
+        invalidates them and removes the database records
+
+        ---
+        post:
+          description: >-
+            Takes a list of datasources, finds the associated cache records and
+            invalidates them and removes the database records
+          requestBody:
+            description: >-
+              A list of datasources uuid or the tuples of database and datasource names
+            required: true
+            content:
+              application/json:
+                schema:
+                  $ref: "#/components/schemas/CacheInvalidationRequestSchema"
+          responses:
+            201:
+              description: cache was successfully invalidated
+            400:
+              $ref: '#/components/responses/400'
+            500:
+              $ref: '#/components/responses/500'
+        """
+        try:
+            datasources = CacheInvalidationRequestSchema().load(request.json)
+        except KeyError:
+            return self.response_400(message="Request is incorrect")
+        except ValidationError as error:
+            return self.response_400(message=error.message)
+        datasource_uids = set(datasources.get("datasource_uids", []))
+        for ds in datasources.get("datasources", []):
+            ds_obj = ConnectorRegistry.get_datasource_by_name(
+                session=db.session,
+                datasource_type=ds.get("datasource_type"),
+                datasource_name=ds.get("datasource_name"),
+                schema=ds.get("schema"),
+                database_name=ds.get("database_name"),
+            )
+            if ds_obj:
+                datasource_uids.add(ds_obj.uid)
+
+        cache_keys = (
+            db.session.query(CacheKey)
+            .filter(CacheKey.datasource_uid.in_(datasource_uids))
+            .all()
+        )
+        if cache_keys:
+            cache_manager.cache.delete_many(*[c.cache_key for c in cache_keys])

Review comment:
       delete_many doesn't fail, only returns true / false if the keys were deleted. e.g. if the key was not found it will return false. Updated CacheKey logic to delete is a single statement & rollback if fails -> that would be more robust.
   
   deleting case & keeping records is not too bad as there still a need to garbage collect those and cache is not a persistent store.

##########
File path: superset/cachekeys/schemas.py
##########
@@ -0,0 +1,45 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+# RISON/JSON schemas for query parameters
+from marshmallow import fields, Schema, validate
+
+from superset.charts.schemas import (
+    datasource_name_description,
+    datasource_type_description,
+    datasource_uid_description,
+)
+
+
+class Datasource(Schema):
+    database_name = fields.String(description="Datasource name",)
+    datasource_name = fields.String(description=datasource_name_description,)
+    schema = fields.String(description="Datasource schema",)
+    datasource_type = fields.String(
+        description=datasource_type_description,
+        validate=validate.OneOf(choices=("druid", "table", "view")),
+        allow_none=True,

Review comment:
       done




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


[GitHub] [incubator-superset] codecov-commenter edited a comment on pull request #10761: feat: [WIP] implement cache invalidation api

Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #10761:
URL: https://github.com/apache/incubator-superset/pull/10761#issuecomment-686252702


   # [Codecov](https://codecov.io/gh/apache/incubator-superset/pull/10761?src=pr&el=h1) Report
   > Merging [#10761](https://codecov.io/gh/apache/incubator-superset/pull/10761?src=pr&el=desc) into [master](https://codecov.io/gh/apache/incubator-superset/commit/1ee87cc4d1810163bd00788636291588ecbfcbdf?el=desc) will **increase** coverage by `2.36%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/incubator-superset/pull/10761/graphs/tree.svg?width=650&height=150&src=pr&token=KsB0fHcx6l)](https://codecov.io/gh/apache/incubator-superset/pull/10761?src=pr&el=tree)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master   #10761      +/-   ##
   ==========================================
   + Coverage   59.23%   61.59%   +2.36%     
   ==========================================
     Files         768      429     -339     
     Lines       36651    13971   -22680     
     Branches     3302     3555     +253     
   ==========================================
   - Hits        21710     8606   -13104     
   + Misses      14759     5179    -9580     
   - Partials      182      186       +4     
   ```
   
   | Flag | Coverage Δ | |
   |---|---|---|
   | #cypress | `?` | |
   | #javascript | `61.59% <ø> (?)` | |
   | #python | `?` | |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   | [Impacted Files](https://codecov.io/gh/apache/incubator-superset/pull/10761?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [superset-frontend/src/views/App.tsx](https://codecov.io/gh/apache/incubator-superset/pull/10761/diff?src=pr&el=tree#diff-c3VwZXJzZXQtZnJvbnRlbmQvc3JjL3ZpZXdzL0FwcC50c3g=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | [superset-frontend/src/SqlLab/App.jsx](https://codecov.io/gh/apache/incubator-superset/pull/10761/diff?src=pr&el=tree#diff-c3VwZXJzZXQtZnJvbnRlbmQvc3JjL1NxbExhYi9BcHAuanN4) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | [superset-frontend/src/views/menu.tsx](https://codecov.io/gh/apache/incubator-superset/pull/10761/diff?src=pr&el=tree#diff-c3VwZXJzZXQtZnJvbnRlbmQvc3JjL3ZpZXdzL21lbnUudHN4) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | [superset-frontend/src/explore/App.jsx](https://codecov.io/gh/apache/incubator-superset/pull/10761/diff?src=pr&el=tree#diff-c3VwZXJzZXQtZnJvbnRlbmQvc3JjL2V4cGxvcmUvQXBwLmpzeA==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | [superset-frontend/src/views/index.tsx](https://codecov.io/gh/apache/incubator-superset/pull/10761/diff?src=pr&el=tree#diff-c3VwZXJzZXQtZnJvbnRlbmQvc3JjL3ZpZXdzL2luZGV4LnRzeA==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | [superset-frontend/src/SqlLab/index.tsx](https://codecov.io/gh/apache/incubator-superset/pull/10761/diff?src=pr&el=tree#diff-c3VwZXJzZXQtZnJvbnRlbmQvc3JjL1NxbExhYi9pbmRleC50c3g=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | [superset-frontend/src/dashboard/App.jsx](https://codecov.io/gh/apache/incubator-superset/pull/10761/diff?src=pr&el=tree#diff-c3VwZXJzZXQtZnJvbnRlbmQvc3JjL2Rhc2hib2FyZC9BcHAuanN4) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | [superset-frontend/src/explore/index.jsx](https://codecov.io/gh/apache/incubator-superset/pull/10761/diff?src=pr&el=tree#diff-c3VwZXJzZXQtZnJvbnRlbmQvc3JjL2V4cGxvcmUvaW5kZXguanN4) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | [superset-frontend/src/dashboard/index.jsx](https://codecov.io/gh/apache/incubator-superset/pull/10761/diff?src=pr&el=tree#diff-c3VwZXJzZXQtZnJvbnRlbmQvc3JjL2Rhc2hib2FyZC9pbmRleC5qc3g=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | [superset-frontend/src/setup/setupColors.js](https://codecov.io/gh/apache/incubator-superset/pull/10761/diff?src=pr&el=tree#diff-c3VwZXJzZXQtZnJvbnRlbmQvc3JjL3NldHVwL3NldHVwQ29sb3JzLmpz) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | ... and [691 more](https://codecov.io/gh/apache/incubator-superset/pull/10761/diff?src=pr&el=tree-more) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-superset/pull/10761?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/incubator-superset/pull/10761?src=pr&el=footer). Last update [1ee87cc...2dc4543](https://codecov.io/gh/apache/incubator-superset/pull/10761?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


[GitHub] [incubator-superset] benceorlai commented on pull request #10761: feat: implement cache invalidation api

Posted by GitBox <gi...@apache.org>.
benceorlai commented on pull request #10761:
URL: https://github.com/apache/incubator-superset/pull/10761#issuecomment-735502767


   Roadmap: https://github.com/apache-superset/superset-roadmap/issues/74


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


[GitHub] [incubator-superset] codecov-commenter commented on pull request #10761: feat: [WIP] implement cache invalidation api

Posted by GitBox <gi...@apache.org>.
codecov-commenter commented on pull request #10761:
URL: https://github.com/apache/incubator-superset/pull/10761#issuecomment-686252702


   # [Codecov](https://codecov.io/gh/apache/incubator-superset/pull/10761?src=pr&el=h1) Report
   > Merging [#10761](https://codecov.io/gh/apache/incubator-superset/pull/10761?src=pr&el=desc) into [master](https://codecov.io/gh/apache/incubator-superset/commit/e0ad9935c145493b5cb6048f1aeab9018edd363a?el=desc) will **decrease** coverage by `3.46%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/incubator-superset/pull/10761/graphs/tree.svg?width=650&height=150&src=pr&token=KsB0fHcx6l)](https://codecov.io/gh/apache/incubator-superset/pull/10761?src=pr&el=tree)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master   #10761      +/-   ##
   ==========================================
   - Coverage   65.06%   61.59%   -3.47%     
   ==========================================
     Files         795      429     -366     
     Lines       37403    13971   -23432     
     Branches     3555     3555              
   ==========================================
   - Hits        24337     8606   -15731     
   + Misses      12959     5179    -7780     
   - Partials      107      186      +79     
   ```
   
   | Flag | Coverage Δ | |
   |---|---|---|
   | #cypress | `?` | |
   | #javascript | `61.59% <ø> (+0.10%)` | :arrow_up: |
   | #python | `?` | |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   | [Impacted Files](https://codecov.io/gh/apache/incubator-superset/pull/10761?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [superset-frontend/src/SqlLab/App.jsx](https://codecov.io/gh/apache/incubator-superset/pull/10761/diff?src=pr&el=tree#diff-c3VwZXJzZXQtZnJvbnRlbmQvc3JjL1NxbExhYi9BcHAuanN4) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | [superset-frontend/src/explore/App.jsx](https://codecov.io/gh/apache/incubator-superset/pull/10761/diff?src=pr&el=tree#diff-c3VwZXJzZXQtZnJvbnRlbmQvc3JjL2V4cGxvcmUvQXBwLmpzeA==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | [superset-frontend/src/dashboard/App.jsx](https://codecov.io/gh/apache/incubator-superset/pull/10761/diff?src=pr&el=tree#diff-c3VwZXJzZXQtZnJvbnRlbmQvc3JjL2Rhc2hib2FyZC9BcHAuanN4) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | [superset-frontend/src/explore/index.jsx](https://codecov.io/gh/apache/incubator-superset/pull/10761/diff?src=pr&el=tree#diff-c3VwZXJzZXQtZnJvbnRlbmQvc3JjL2V4cGxvcmUvaW5kZXguanN4) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | [superset-frontend/src/dashboard/index.jsx](https://codecov.io/gh/apache/incubator-superset/pull/10761/diff?src=pr&el=tree#diff-c3VwZXJzZXQtZnJvbnRlbmQvc3JjL2Rhc2hib2FyZC9pbmRleC5qc3g=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | [superset-frontend/src/setup/setupColors.js](https://codecov.io/gh/apache/incubator-superset/pull/10761/diff?src=pr&el=tree#diff-c3VwZXJzZXQtZnJvbnRlbmQvc3JjL3NldHVwL3NldHVwQ29sb3JzLmpz) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | [superset-frontend/src/chart/ChartContainer.jsx](https://codecov.io/gh/apache/incubator-superset/pull/10761/diff?src=pr&el=tree#diff-c3VwZXJzZXQtZnJvbnRlbmQvc3JjL2NoYXJ0L0NoYXJ0Q29udGFpbmVyLmpzeA==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | [superset-frontend/src/setup/setupFormatters.js](https://codecov.io/gh/apache/incubator-superset/pull/10761/diff?src=pr&el=tree#diff-c3VwZXJzZXQtZnJvbnRlbmQvc3JjL3NldHVwL3NldHVwRm9ybWF0dGVycy5qcw==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | [superset-frontend/src/explore/reducers/index.js](https://codecov.io/gh/apache/incubator-superset/pull/10761/diff?src=pr&el=tree#diff-c3VwZXJzZXQtZnJvbnRlbmQvc3JjL2V4cGxvcmUvcmVkdWNlcnMvaW5kZXguanM=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | [superset-frontend/src/setup/setupPluginsExtra.js](https://codecov.io/gh/apache/incubator-superset/pull/10761/diff?src=pr&el=tree#diff-c3VwZXJzZXQtZnJvbnRlbmQvc3JjL3NldHVwL3NldHVwUGx1Z2luc0V4dHJhLmpz) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | ... and [524 more](https://codecov.io/gh/apache/incubator-superset/pull/10761/diff?src=pr&el=tree-more) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-superset/pull/10761?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/incubator-superset/pull/10761?src=pr&el=footer). Last update [e0ad993...2028348](https://codecov.io/gh/apache/incubator-superset/pull/10761?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


[GitHub] [incubator-superset] bkyryliuk commented on pull request #10761: feat: [WIP] implement cache invalidation api

Posted by GitBox <gi...@apache.org>.
bkyryliuk commented on pull request #10761:
URL: https://github.com/apache/incubator-superset/pull/10761#issuecomment-687282114


   @willbarrett , @dpgaspar - ready for review


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


[GitHub] [incubator-superset] bkyryliuk merged pull request #10761: feat: implement cache invalidation api

Posted by GitBox <gi...@apache.org>.
bkyryliuk merged pull request #10761:
URL: https://github.com/apache/incubator-superset/pull/10761


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


[GitHub] [incubator-superset] mistercrunch commented on a change in pull request #10761: feat: implement cache invalidation api

Posted by GitBox <gi...@apache.org>.
mistercrunch commented on a change in pull request #10761:
URL: https://github.com/apache/incubator-superset/pull/10761#discussion_r484002542



##########
File path: superset/cachekeys/api.py
##########
@@ -0,0 +1,98 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from flask import request, Response
+from flask_appbuilder import expose
+from flask_appbuilder.api import safe
+from flask_appbuilder.models.sqla.interface import SQLAInterface
+from flask_appbuilder.security.decorators import protect
+from jsonschema import ValidationError
+
+from superset.cachekeys.schemas import CacheInvalidationRequestSchema
+from superset.connectors.connector_registry import ConnectorRegistry
+from superset.extensions import cache_manager, db, event_logger
+from superset.models.cache import CacheKey
+from superset.views.base_api import BaseSupersetModelRestApi, statsd_metrics
+
+
+class CacheRestApi(BaseSupersetModelRestApi):
+    datamodel = SQLAInterface(CacheKey)
+    resource_name = "cache"
+    allow_browser_login = True
+    class_permission_name = "CacheRestApi"
+
+    openapi_spec_component_schemas = (CacheInvalidationRequestSchema,)
+
+    @expose("/invalidate", methods=["POST"])
+    @event_logger.log_this
+    @protect()
+    @safe
+    @statsd_metrics
+    def invalidate(self) -> Response:
+        """
+        Takes a list of datasources, finds the associated cache records and

Review comment:
       Side note (outside the scope of this PR), we'll be renaming datasource to dataset. I think it's mostly done in the UI layer, but has yet to be done in the code.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


[GitHub] [incubator-superset] dpgaspar commented on a change in pull request #10761: feat: [WIP] implement cache invalidation api

Posted by GitBox <gi...@apache.org>.
dpgaspar commented on a change in pull request #10761:
URL: https://github.com/apache/incubator-superset/pull/10761#discussion_r482152879



##########
File path: superset/cache/api.py
##########
@@ -0,0 +1,91 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from flask import request, Response
+from flask_appbuilder import expose
+from flask_appbuilder.api import safe
+from flask_appbuilder.models.sqla.interface import SQLAInterface
+from flask_appbuilder.security.decorators import protect
+from flask_babel import gettext as _
+from jsonschema import ValidationError
+
+from superset.cache.schemas import CacheInvalidationRequestSchema
+from superset.connectors.connector_registry import ConnectorRegistry
+from superset.extensions import cache_manager, db, event_logger
+from superset.models.cache import CacheKey
+from superset.views.base_api import BaseSupersetModelRestApi, statsd_metrics
+
+
+class CacheRestApi(BaseSupersetModelRestApi):
+    datamodel = SQLAInterface(CacheKey)
+    resource_name = "cache"
+    allow_browser_login = True
+    class_permission_name = "CacheRestApi"
+

Review comment:
       use `openapi_spec_component_schemas = CacheInvalidateRequestSchema` to add the schema to the OpenAPI spec




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


[GitHub] [incubator-superset] bkyryliuk commented on pull request #10761: feat: implement cache invalidation api

Posted by GitBox <gi...@apache.org>.
bkyryliuk commented on pull request #10761:
URL: https://github.com/apache/incubator-superset/pull/10761#issuecomment-692213931


   > Do you want all the `ModelRestApi` route methods to be available? If not include `include_route_methods = {"invalidate",}` also note this will create a new specific permission `can invalidate on Cache key`.
   > Check `https://localhost:8080/swagger/v1`
   
   addressed, thanks


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


[GitHub] [incubator-superset] willbarrett commented on a change in pull request #10761: feat: implement cache invalidation api

Posted by GitBox <gi...@apache.org>.
willbarrett commented on a change in pull request #10761:
URL: https://github.com/apache/incubator-superset/pull/10761#discussion_r488349155



##########
File path: superset/cachekeys/api.py
##########
@@ -0,0 +1,98 @@
+# Licensed to the Apache Software Foundation (ASF) under one

Review comment:
       How about `DELETE /api/v1/cachekey/<cachekey>`?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


[GitHub] [incubator-superset] bkyryliuk commented on a change in pull request #10761: feat: implement cache invalidation api

Posted by GitBox <gi...@apache.org>.
bkyryliuk commented on a change in pull request #10761:
URL: https://github.com/apache/incubator-superset/pull/10761#discussion_r487139156



##########
File path: superset/cachekeys/api.py
##########
@@ -0,0 +1,98 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from flask import request, Response
+from flask_appbuilder import expose
+from flask_appbuilder.api import safe
+from flask_appbuilder.models.sqla.interface import SQLAInterface
+from flask_appbuilder.security.decorators import protect
+from jsonschema import ValidationError
+
+from superset.cachekeys.schemas import CacheInvalidationRequestSchema
+from superset.connectors.connector_registry import ConnectorRegistry
+from superset.extensions import cache_manager, db, event_logger
+from superset.models.cache import CacheKey
+from superset.views.base_api import BaseSupersetModelRestApi, statsd_metrics
+
+
+class CacheRestApi(BaseSupersetModelRestApi):
+    datamodel = SQLAInterface(CacheKey)
+    resource_name = "cache"
+    allow_browser_login = True
+    class_permission_name = "CacheRestApi"
+
+    openapi_spec_component_schemas = (CacheInvalidationRequestSchema,)
+
+    @expose("/invalidate", methods=["POST"])
+    @event_logger.log_this
+    @protect()
+    @safe
+    @statsd_metrics
+    def invalidate(self) -> Response:
+        """
+        Takes a list of datasources, finds the associated cache records and
+        invalidates them and removes the database records
+
+        ---
+        post:
+          description: >-
+            Takes a list of datasources, finds the associated cache records and
+            invalidates them and removes the database records
+          requestBody:
+            description: >-
+              A list of datasources uuid or the tuples of database and datasource names
+            required: true
+            content:
+              application/json:
+                schema:
+                  $ref: "#/components/schemas/CacheInvalidationRequestSchema"
+          responses:
+            201:
+              description: cache was successfully invalidated
+            400:
+              $ref: '#/components/responses/400'
+            500:
+              $ref: '#/components/responses/500'
+        """
+        try:
+            datasources = CacheInvalidationRequestSchema().load(request.json)
+        except KeyError:
+            return self.response_400(message="Request is incorrect")
+        except ValidationError as error:
+            return self.response_400(message=error.message)
+        datasource_uids = set(datasources.get("datasource_uids", []))
+        for ds in datasources.get("datasources", []):
+            ds_obj = ConnectorRegistry.get_datasource_by_name(
+                session=db.session,
+                datasource_type=ds.get("datasource_type"),
+                datasource_name=ds.get("datasource_name"),
+                schema=ds.get("schema"),
+                database_name=ds.get("database_name"),
+            )
+            if ds_obj:
+                datasource_uids.add(ds_obj.uid)
+
+        cache_keys = (
+            db.session.query(CacheKey)
+            .filter(CacheKey.datasource_uid.in_(datasource_uids))
+            .all()
+        )
+        if cache_keys:
+            cache_manager.cache.delete_many(*[c.cache_key for c in cache_keys])

Review comment:
       delete_many doesn't fail, only returns true / false if the keys were deleted. e.g. if the key was not found it will return false. Updated CacheKey logic to delete is a single statement & rollback if fails -> that would be more robust.
   
   deleting case & keeping records is not too bad as there still a need to garbage collect those and cache is not a persistent store.

##########
File path: superset/cachekeys/schemas.py
##########
@@ -0,0 +1,45 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+# RISON/JSON schemas for query parameters
+from marshmallow import fields, Schema, validate
+
+from superset.charts.schemas import (
+    datasource_name_description,
+    datasource_type_description,
+    datasource_uid_description,
+)
+
+
+class Datasource(Schema):
+    database_name = fields.String(description="Datasource name",)
+    datasource_name = fields.String(description=datasource_name_description,)
+    schema = fields.String(description="Datasource schema",)
+    datasource_type = fields.String(
+        description=datasource_type_description,
+        validate=validate.OneOf(choices=("druid", "table", "view")),
+        allow_none=True,

Review comment:
       done




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


[GitHub] [incubator-superset] bkyryliuk commented on a change in pull request #10761: feat: implement cache invalidation api

Posted by GitBox <gi...@apache.org>.
bkyryliuk commented on a change in pull request #10761:
URL: https://github.com/apache/incubator-superset/pull/10761#discussion_r486693571



##########
File path: superset/cachekeys/api.py
##########
@@ -0,0 +1,98 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from flask import request, Response
+from flask_appbuilder import expose
+from flask_appbuilder.api import safe
+from flask_appbuilder.models.sqla.interface import SQLAInterface
+from flask_appbuilder.security.decorators import protect
+from jsonschema import ValidationError
+
+from superset.cachekeys.schemas import CacheInvalidationRequestSchema
+from superset.connectors.connector_registry import ConnectorRegistry
+from superset.extensions import cache_manager, db, event_logger
+from superset.models.cache import CacheKey
+from superset.views.base_api import BaseSupersetModelRestApi, statsd_metrics
+
+
+class CacheRestApi(BaseSupersetModelRestApi):
+    datamodel = SQLAInterface(CacheKey)
+    resource_name = "cache"

Review comment:
       renamed to cachekey, I am happy with either way naming it `cachekey` or `cache`
   I was thinking about moving it into separate class from the cache key, but most of the instrumentation lives in BaseSupersetModelRestApi that requires to model.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


[GitHub] [incubator-superset] bkyryliuk commented on a change in pull request #10761: feat: [WIP] implement cache invalidation api

Posted by GitBox <gi...@apache.org>.
bkyryliuk commented on a change in pull request #10761:
URL: https://github.com/apache/incubator-superset/pull/10761#discussion_r483758459



##########
File path: tests/superset_test_config.py
##########
@@ -76,6 +76,14 @@ def GET_FEATURE_FLAGS_FUNC(ff):
 REDIS_PORT = os.environ.get("REDIS_PORT", "6379")
 REDIS_CELERY_DB = os.environ.get("REDIS_CELERY_DB", 2)
 REDIS_RESULTS_DB = os.environ.get("REDIS_RESULTS_DB", 3)
+REDIS_CACHE_DB = os.environ.get("REDIS_CACHE_DB", 4)
+
+CACHE_CONFIG = {

Review comment:
       needed for cache to work on CI




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


[GitHub] [incubator-superset] bkyryliuk commented on a change in pull request #10761: feat: implement cache invalidation api

Posted by GitBox <gi...@apache.org>.
bkyryliuk commented on a change in pull request #10761:
URL: https://github.com/apache/incubator-superset/pull/10761#discussion_r487139156



##########
File path: superset/cachekeys/api.py
##########
@@ -0,0 +1,98 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from flask import request, Response
+from flask_appbuilder import expose
+from flask_appbuilder.api import safe
+from flask_appbuilder.models.sqla.interface import SQLAInterface
+from flask_appbuilder.security.decorators import protect
+from jsonschema import ValidationError
+
+from superset.cachekeys.schemas import CacheInvalidationRequestSchema
+from superset.connectors.connector_registry import ConnectorRegistry
+from superset.extensions import cache_manager, db, event_logger
+from superset.models.cache import CacheKey
+from superset.views.base_api import BaseSupersetModelRestApi, statsd_metrics
+
+
+class CacheRestApi(BaseSupersetModelRestApi):
+    datamodel = SQLAInterface(CacheKey)
+    resource_name = "cache"
+    allow_browser_login = True
+    class_permission_name = "CacheRestApi"
+
+    openapi_spec_component_schemas = (CacheInvalidationRequestSchema,)
+
+    @expose("/invalidate", methods=["POST"])
+    @event_logger.log_this
+    @protect()
+    @safe
+    @statsd_metrics
+    def invalidate(self) -> Response:
+        """
+        Takes a list of datasources, finds the associated cache records and
+        invalidates them and removes the database records
+
+        ---
+        post:
+          description: >-
+            Takes a list of datasources, finds the associated cache records and
+            invalidates them and removes the database records
+          requestBody:
+            description: >-
+              A list of datasources uuid or the tuples of database and datasource names
+            required: true
+            content:
+              application/json:
+                schema:
+                  $ref: "#/components/schemas/CacheInvalidationRequestSchema"
+          responses:
+            201:
+              description: cache was successfully invalidated
+            400:
+              $ref: '#/components/responses/400'
+            500:
+              $ref: '#/components/responses/500'
+        """
+        try:
+            datasources = CacheInvalidationRequestSchema().load(request.json)
+        except KeyError:
+            return self.response_400(message="Request is incorrect")
+        except ValidationError as error:
+            return self.response_400(message=error.message)
+        datasource_uids = set(datasources.get("datasource_uids", []))
+        for ds in datasources.get("datasources", []):
+            ds_obj = ConnectorRegistry.get_datasource_by_name(
+                session=db.session,
+                datasource_type=ds.get("datasource_type"),
+                datasource_name=ds.get("datasource_name"),
+                schema=ds.get("schema"),
+                database_name=ds.get("database_name"),
+            )
+            if ds_obj:
+                datasource_uids.add(ds_obj.uid)
+
+        cache_keys = (
+            db.session.query(CacheKey)
+            .filter(CacheKey.datasource_uid.in_(datasource_uids))
+            .all()
+        )
+        if cache_keys:
+            cache_manager.cache.delete_many(*[c.cache_key for c in cache_keys])

Review comment:
       delete_many doesn't fail, only returns true / false if the keys were deleted. e.g. if the key was not found it will return false. Updated CacheKey logic to delete is a single statement & rollback if fails -> that would be more robust.
   
   deleting case & keeping records is not too bad as there still a need to garbage collect those and cache is not a persistent store.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


[GitHub] [incubator-superset] codecov-commenter edited a comment on pull request #10761: feat: [WIP] implement cache invalidation api

Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #10761:
URL: https://github.com/apache/incubator-superset/pull/10761#issuecomment-686252702


   # [Codecov](https://codecov.io/gh/apache/incubator-superset/pull/10761?src=pr&el=h1) Report
   > Merging [#10761](https://codecov.io/gh/apache/incubator-superset/pull/10761?src=pr&el=desc) into [master](https://codecov.io/gh/apache/incubator-superset/commit/e0ad9935c145493b5cb6048f1aeab9018edd363a?el=desc) will **increase** coverage by `7.55%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/incubator-superset/pull/10761/graphs/tree.svg?width=650&height=150&src=pr&token=KsB0fHcx6l)](https://codecov.io/gh/apache/incubator-superset/pull/10761?src=pr&el=tree)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master   #10761      +/-   ##
   ==========================================
   + Coverage   65.06%   72.61%   +7.55%     
   ==========================================
     Files         795      429     -366     
     Lines       37403    13973   -23430     
     Branches     3555     3555              
   ==========================================
   - Hits        24337    10147   -14190     
   + Misses      12959     3717    -9242     
   - Partials      107      109       +2     
   ```
   
   | Flag | Coverage Δ | |
   |---|---|---|
   | #cypress | `55.36% <ø> (-0.70%)` | :arrow_down: |
   | #javascript | `61.59% <ø> (+0.10%)` | :arrow_up: |
   | #python | `?` | |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   | [Impacted Files](https://codecov.io/gh/apache/incubator-superset/pull/10761?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [...et-frontend/src/SqlLab/reducers/getInitialState.js](https://codecov.io/gh/apache/incubator-superset/pull/10761/diff?src=pr&el=tree#diff-c3VwZXJzZXQtZnJvbnRlbmQvc3JjL1NxbExhYi9yZWR1Y2Vycy9nZXRJbml0aWFsU3RhdGUuanM=) | `33.33% <0.00%> (-16.67%)` | :arrow_down: |
   | [...rontend/src/visualizations/FilterBox/FilterBox.jsx](https://codecov.io/gh/apache/incubator-superset/pull/10761/diff?src=pr&el=tree#diff-c3VwZXJzZXQtZnJvbnRlbmQvc3JjL3Zpc3VhbGl6YXRpb25zL0ZpbHRlckJveC9GaWx0ZXJCb3guanN4) | `56.52% <0.00%> (-8.70%)` | :arrow_down: |
   | [...rontend/src/SqlLab/components/QueryAutoRefresh.jsx](https://codecov.io/gh/apache/incubator-superset/pull/10761/diff?src=pr&el=tree#diff-c3VwZXJzZXQtZnJvbnRlbmQvc3JjL1NxbExhYi9jb21wb25lbnRzL1F1ZXJ5QXV0b1JlZnJlc2guanN4) | `65.90% <0.00%> (-6.82%)` | :arrow_down: |
   | [...rontend/src/SqlLab/components/TabbedSqlEditors.jsx](https://codecov.io/gh/apache/incubator-superset/pull/10761/diff?src=pr&el=tree#diff-c3VwZXJzZXQtZnJvbnRlbmQvc3JjL1NxbExhYi9jb21wb25lbnRzL1RhYmJlZFNxbEVkaXRvcnMuanN4) | `77.27% <0.00%> (-5.20%)` | :arrow_down: |
   | [...rontend/src/SqlLab/components/SqlEditorLeftBar.jsx](https://codecov.io/gh/apache/incubator-superset/pull/10761/diff?src=pr&el=tree#diff-c3VwZXJzZXQtZnJvbnRlbmQvc3JjL1NxbExhYi9jb21wb25lbnRzL1NxbEVkaXRvckxlZnRCYXIuanN4) | `44.00% <0.00%> (-4.00%)` | :arrow_down: |
   | [...et-frontend/src/components/ConfirmStatusChange.tsx](https://codecov.io/gh/apache/incubator-superset/pull/10761/diff?src=pr&el=tree#diff-c3VwZXJzZXQtZnJvbnRlbmQvc3JjL2NvbXBvbmVudHMvQ29uZmlybVN0YXR1c0NoYW5nZS50c3g=) | `96.15% <0.00%> (-3.85%)` | :arrow_down: |
   | [...end/src/SqlLab/components/TemplateParamsEditor.jsx](https://codecov.io/gh/apache/incubator-superset/pull/10761/diff?src=pr&el=tree#diff-c3VwZXJzZXQtZnJvbnRlbmQvc3JjL1NxbExhYi9jb21wb25lbnRzL1RlbXBsYXRlUGFyYW1zRWRpdG9yLmpzeA==) | `88.57% <0.00%> (-2.86%)` | :arrow_down: |
   | [...ontend/src/views/CRUD/data/dataset/DatasetList.tsx](https://codecov.io/gh/apache/incubator-superset/pull/10761/diff?src=pr&el=tree#diff-c3VwZXJzZXQtZnJvbnRlbmQvc3JjL3ZpZXdzL0NSVUQvZGF0YS9kYXRhc2V0L0RhdGFzZXRMaXN0LnRzeA==) | `72.58% <0.00%> (-2.78%)` | :arrow_down: |
   | [superset-frontend/src/SqlLab/actions/sqlLab.js](https://codecov.io/gh/apache/incubator-superset/pull/10761/diff?src=pr&el=tree#diff-c3VwZXJzZXQtZnJvbnRlbmQvc3JjL1NxbExhYi9hY3Rpb25zL3NxbExhYi5qcw==) | `60.25% <0.00%> (-2.36%)` | :arrow_down: |
   | [.../src/dashboard/components/gridComponents/Chart.jsx](https://codecov.io/gh/apache/incubator-superset/pull/10761/diff?src=pr&el=tree#diff-c3VwZXJzZXQtZnJvbnRlbmQvc3JjL2Rhc2hib2FyZC9jb21wb25lbnRzL2dyaWRDb21wb25lbnRzL0NoYXJ0LmpzeA==) | `83.67% <0.00%> (-2.05%)` | :arrow_down: |
   | ... and [374 more](https://codecov.io/gh/apache/incubator-superset/pull/10761/diff?src=pr&el=tree-more) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-superset/pull/10761?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/incubator-superset/pull/10761?src=pr&el=footer). Last update [e0ad993...2028348](https://codecov.io/gh/apache/incubator-superset/pull/10761?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


[GitHub] [incubator-superset] bkyryliuk commented on a change in pull request #10761: feat: implement cache invalidation api

Posted by GitBox <gi...@apache.org>.
bkyryliuk commented on a change in pull request #10761:
URL: https://github.com/apache/incubator-superset/pull/10761#discussion_r486413655



##########
File path: superset/cachekeys/api.py
##########
@@ -0,0 +1,98 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from flask import request, Response
+from flask_appbuilder import expose
+from flask_appbuilder.api import safe
+from flask_appbuilder.models.sqla.interface import SQLAInterface
+from flask_appbuilder.security.decorators import protect
+from jsonschema import ValidationError
+
+from superset.cachekeys.schemas import CacheInvalidationRequestSchema
+from superset.connectors.connector_registry import ConnectorRegistry
+from superset.extensions import cache_manager, db, event_logger
+from superset.models.cache import CacheKey
+from superset.views.base_api import BaseSupersetModelRestApi, statsd_metrics
+
+
+class CacheRestApi(BaseSupersetModelRestApi):
+    datamodel = SQLAInterface(CacheKey)
+    resource_name = "cache"
+    allow_browser_login = True
+    class_permission_name = "CacheRestApi"
+
+    openapi_spec_component_schemas = (CacheInvalidationRequestSchema,)
+
+    @expose("/invalidate", methods=["POST"])
+    @event_logger.log_this
+    @protect()
+    @safe
+    @statsd_metrics
+    def invalidate(self) -> Response:
+        """
+        Takes a list of datasources, finds the associated cache records and

Review comment:
       good catch, I can do it in the separate PR if that works for you, would rename it in API & CacheKey model for consistency.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


[GitHub] [incubator-superset] dpgaspar commented on a change in pull request #10761: feat: [WIP] implement cache invalidation api

Posted by GitBox <gi...@apache.org>.
dpgaspar commented on a change in pull request #10761:
URL: https://github.com/apache/incubator-superset/pull/10761#discussion_r482152879



##########
File path: superset/cache/api.py
##########
@@ -0,0 +1,91 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from flask import request, Response
+from flask_appbuilder import expose
+from flask_appbuilder.api import safe
+from flask_appbuilder.models.sqla.interface import SQLAInterface
+from flask_appbuilder.security.decorators import protect
+from flask_babel import gettext as _
+from jsonschema import ValidationError
+
+from superset.cache.schemas import CacheInvalidationRequestSchema
+from superset.connectors.connector_registry import ConnectorRegistry
+from superset.extensions import cache_manager, db, event_logger
+from superset.models.cache import CacheKey
+from superset.views.base_api import BaseSupersetModelRestApi, statsd_metrics
+
+
+class CacheRestApi(BaseSupersetModelRestApi):
+    datamodel = SQLAInterface(CacheKey)
+    resource_name = "cache"
+    allow_browser_login = True
+    class_permission_name = "CacheRestApi"
+

Review comment:
       use `openapi_spec_component_schemas = (CacheInvalidateRequestSchema,)` to add the schema to the OpenAPI spec




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


[GitHub] [incubator-superset] bkyryliuk commented on a change in pull request #10761: feat: implement cache invalidation api

Posted by GitBox <gi...@apache.org>.
bkyryliuk commented on a change in pull request #10761:
URL: https://github.com/apache/incubator-superset/pull/10761#discussion_r486613012



##########
File path: tests/base_tests.py
##########
@@ -69,6 +71,39 @@ def get_resp(
     return resp.data.decode("utf-8")
 
 
+def post_assert_metric(

Review comment:
       yes same code, moved it into this function for reuse in the pytests




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


[GitHub] [incubator-superset] bkyryliuk commented on pull request #10761: feat: implement cache invalidation api

Posted by GitBox <gi...@apache.org>.
bkyryliuk commented on pull request #10761:
URL: https://github.com/apache/incubator-superset/pull/10761#issuecomment-692216023


   ![image](https://user-images.githubusercontent.com/5727938/93120888-dfd4e000-f678-11ea-87b5-621c4b410224.png)
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


[GitHub] [incubator-superset] bkyryliuk commented on a change in pull request #10761: feat: implement cache invalidation api

Posted by GitBox <gi...@apache.org>.
bkyryliuk commented on a change in pull request #10761:
URL: https://github.com/apache/incubator-superset/pull/10761#discussion_r488870421



##########
File path: superset/cachekeys/api.py
##########
@@ -0,0 +1,98 @@
+# Licensed to the Apache Software Foundation (ASF) under one

Review comment:
       it would be too much struggle to fetch cache keys though the API and then issue deletes for those.
   I think we could expose GET / DELETE later once those use cases will arise




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


[GitHub] [incubator-superset] dpgaspar commented on a change in pull request #10761: feat: [WIP] implement cache invalidation api

Posted by GitBox <gi...@apache.org>.
dpgaspar commented on a change in pull request #10761:
URL: https://github.com/apache/incubator-superset/pull/10761#discussion_r482154191



##########
File path: superset/cache/api.py
##########
@@ -0,0 +1,91 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from flask import request, Response
+from flask_appbuilder import expose
+from flask_appbuilder.api import safe
+from flask_appbuilder.models.sqla.interface import SQLAInterface
+from flask_appbuilder.security.decorators import protect
+from flask_babel import gettext as _
+from jsonschema import ValidationError
+
+from superset.cache.schemas import CacheInvalidationRequestSchema
+from superset.connectors.connector_registry import ConnectorRegistry
+from superset.extensions import cache_manager, db, event_logger
+from superset.models.cache import CacheKey
+from superset.views.base_api import BaseSupersetModelRestApi, statsd_metrics
+
+
+class CacheRestApi(BaseSupersetModelRestApi):
+    datamodel = SQLAInterface(CacheKey)
+    resource_name = "cache"
+    allow_browser_login = True
+    class_permission_name = "CacheRestApi"
+
+    @expose("/invalidate", methods=["POST"])
+    @event_logger.log_this
+    @protect()
+    @safe
+    @statsd_metrics
+    def invalidate(self) -> Response:
+        """
+        Takes a list of datasources, finds the associated cache records and invalidates them.
+
+        ---
+        post:
+          description: >-
+            Takes a list of datasources, finds the associated cache records and invalidates them.
+          requestBody:
+            description: >-
+              A list of datasources uuid or the tuples of the database and datasource names
+            required: true
+            content:
+              application/json:
+                schema:
+                  $ref: "#/components/schemas/CacheInvalidateRequestSchema"
+          responses:
+            201:
+              $ref: '#/components/responses/201'
+            400:
+              $ref: '#/components/responses/400'
+            500:
+              $ref: '#/components/responses/500'
+        """
+        try:
+            datasources = CacheInvalidationRequestSchema().load(request.json)
+        except KeyError:
+            return self.response_400(message="Request is incorrect")
+        except ValidationError as error:
+            return self.response_400(
+                message=_("Request is incorrect: %(error)s", error=error.messages)

Review comment:
       nit: Just use: `return self.response_400(message=error.messages)`




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


[GitHub] [incubator-superset] mistercrunch commented on a change in pull request #10761: feat: implement cache invalidation api

Posted by GitBox <gi...@apache.org>.
mistercrunch commented on a change in pull request #10761:
URL: https://github.com/apache/incubator-superset/pull/10761#discussion_r484002444



##########
File path: superset/cachekeys/api.py
##########
@@ -0,0 +1,98 @@
+# Licensed to the Apache Software Foundation (ASF) under one

Review comment:
       I think the route `/api/v1/cachekey/invalidate` is not ideal. I'm struggling to find something good though




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


[GitHub] [incubator-superset] bkyryliuk commented on a change in pull request #10761: feat: implement cache invalidation api

Posted by GitBox <gi...@apache.org>.
bkyryliuk commented on a change in pull request #10761:
URL: https://github.com/apache/incubator-superset/pull/10761#discussion_r486693694



##########
File path: tests/cachekeys/api_tests.py
##########
@@ -0,0 +1,131 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+# isort:skip_file
+"""Unit tests for Superset"""
+from tests.test_app import app  # noqa
+
+from superset.extensions import cache_manager, db
+from superset.models.cache import CacheKey
+from tests.base_tests import (
+    SupersetTestCase,
+    post_assert_metric,
+    test_client,
+    logged_in_admin,
+)  # noqa
+
+
+def test_invalidate_cache(logged_in_admin):
+    rv = post_assert_metric(
+        test_client,
+        "api/v1/cache/invalidate",
+        {"datasource_uids": ["3__table"]},
+        "invalidate",
+    )
+    assert rv.status_code == 201
+
+
+def test_invalidate_existing_cache(logged_in_admin):
+    db.session.add(CacheKey(cache_key="cache_key", datasource_uid="3__table"))
+    db.session.commit()
+    cache_manager.cache.set("cache_key", "value")
+
+    rv = post_assert_metric(
+        test_client,
+        "api/v1/cache/invalidate",
+        {"datasource_uids": ["3__table"]},
+        "invalidate",
+    )
+
+    assert rv.status_code == 201
+    assert cache_manager.cache.get("cache_key") == None
+    assert (
+        not db.session.query(CacheKey).filter(CacheKey.cache_key == "cache_key").first()
+    )
+
+
+def test_invalidate_existing_caches(logged_in_admin):
+    bn = SupersetTestCase.get_birth_names_dataset()
+
+    db.session.add(CacheKey(cache_key="cache_key1", datasource_uid="3__druid"))
+    db.session.add(CacheKey(cache_key="cache_key2", datasource_uid="3__druid"))
+    db.session.add(CacheKey(cache_key="cache_key4", datasource_uid=f"{bn.id}__table"))
+    db.session.add(CacheKey(cache_key="cache_keyX", datasource_uid="X__table"))
+    db.session.commit()
+
+    cache_manager.cache.set("cache_key1", "value")
+    cache_manager.cache.set("cache_key2", "value")
+    cache_manager.cache.set("cache_key4", "value")
+    cache_manager.cache.set("cache_keyX", "value")
+
+    rv = post_assert_metric(
+        test_client,
+        "api/v1/cache/invalidate",
+        {
+            "datasource_uids": ["3__druid", "4__druid"],

Review comment:
       done, it caught the bug by the way :) thanks !




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


[GitHub] [incubator-superset] codecov-commenter edited a comment on pull request #10761: feat: implement cache invalidation api

Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #10761:
URL: https://github.com/apache/incubator-superset/pull/10761#issuecomment-686252702


   # [Codecov](https://codecov.io/gh/apache/incubator-superset/pull/10761?src=pr&el=h1) Report
   > Merging [#10761](https://codecov.io/gh/apache/incubator-superset/pull/10761?src=pr&el=desc) into [master](https://codecov.io/gh/apache/incubator-superset/commit/8a9ae811d006c886fa43383cf3c351ac3be3010b?el=desc) will **decrease** coverage by `0.32%`.
   > The diff coverage is `96.77%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/incubator-superset/pull/10761/graphs/tree.svg?width=650&height=150&src=pr&token=KsB0fHcx6l)](https://codecov.io/gh/apache/incubator-superset/pull/10761?src=pr&el=tree)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master   #10761      +/-   ##
   ==========================================
   - Coverage   61.32%   60.99%   -0.33%     
   ==========================================
     Files         803      806       +3     
     Lines       37927    38002      +75     
     Branches     3561     3561              
   ==========================================
   - Hits        23258    23180      -78     
   - Misses      14483    14636     +153     
     Partials      186      186              
   ```
   
   | Flag | Coverage Δ | |
   |---|---|---|
   | #javascript | `61.59% <ø> (+<0.01%)` | :arrow_up: |
   | #python | `60.65% <96.77%> (-0.52%)` | :arrow_down: |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   | [Impacted Files](https://codecov.io/gh/apache/incubator-superset/pull/10761?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [superset/cachekeys/api.py](https://codecov.io/gh/apache/incubator-superset/pull/10761/diff?src=pr&el=tree#diff-c3VwZXJzZXQvY2FjaGVrZXlzL2FwaS5weQ==) | `95.91% <95.91%> (ø)` | |
   | [superset/app.py](https://codecov.io/gh/apache/incubator-superset/pull/10761/diff?src=pr&el=tree#diff-c3VwZXJzZXQvYXBwLnB5) | `80.30% <100.00%> (+0.15%)` | :arrow_up: |
   | [superset/cachekeys/schemas.py](https://codecov.io/gh/apache/incubator-superset/pull/10761/diff?src=pr&el=tree#diff-c3VwZXJzZXQvY2FjaGVrZXlzL3NjaGVtYXMucHk=) | `100.00% <100.00%> (ø)` | |
   | [superset/charts/schemas.py](https://codecov.io/gh/apache/incubator-superset/pull/10761/diff?src=pr&el=tree#diff-c3VwZXJzZXQvY2hhcnRzL3NjaGVtYXMucHk=) | `100.00% <100.00%> (ø)` | |
   | [superset/db\_engines/hive.py](https://codecov.io/gh/apache/incubator-superset/pull/10761/diff?src=pr&el=tree#diff-c3VwZXJzZXQvZGJfZW5naW5lcy9oaXZlLnB5) | `0.00% <0.00%> (-85.72%)` | :arrow_down: |
   | [superset/db\_engine\_specs/hive.py](https://codecov.io/gh/apache/incubator-superset/pull/10761/diff?src=pr&el=tree#diff-c3VwZXJzZXQvZGJfZW5naW5lX3NwZWNzL2hpdmUucHk=) | `53.90% <0.00%> (-30.08%)` | :arrow_down: |
   | [superset/db\_engine\_specs/presto.py](https://codecov.io/gh/apache/incubator-superset/pull/10761/diff?src=pr&el=tree#diff-c3VwZXJzZXQvZGJfZW5naW5lX3NwZWNzL3ByZXN0by5weQ==) | `70.85% <0.00%> (-10.77%)` | :arrow_down: |
   | [superset/connectors/base/views.py](https://codecov.io/gh/apache/incubator-superset/pull/10761/diff?src=pr&el=tree#diff-c3VwZXJzZXQvY29ubmVjdG9ycy9iYXNlL3ZpZXdzLnB5) | `71.42% <0.00%> (-3.58%)` | :arrow_down: |
   | [superset/examples/world\_bank.py](https://codecov.io/gh/apache/incubator-superset/pull/10761/diff?src=pr&el=tree#diff-c3VwZXJzZXQvZXhhbXBsZXMvd29ybGRfYmFuay5weQ==) | `97.10% <0.00%> (-2.90%)` | :arrow_down: |
   | [superset/examples/birth\_names.py](https://codecov.io/gh/apache/incubator-superset/pull/10761/diff?src=pr&el=tree#diff-c3VwZXJzZXQvZXhhbXBsZXMvYmlydGhfbmFtZXMucHk=) | `97.36% <0.00%> (-2.64%)` | :arrow_down: |
   | ... and [15 more](https://codecov.io/gh/apache/incubator-superset/pull/10761/diff?src=pr&el=tree-more) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-superset/pull/10761?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/incubator-superset/pull/10761?src=pr&el=footer). Last update [8a9ae81...115de31](https://codecov.io/gh/apache/incubator-superset/pull/10761?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


[GitHub] [incubator-superset] dpgaspar commented on a change in pull request #10761: feat: implement cache invalidation api

Posted by GitBox <gi...@apache.org>.
dpgaspar commented on a change in pull request #10761:
URL: https://github.com/apache/incubator-superset/pull/10761#discussion_r486469842



##########
File path: superset/cachekeys/schemas.py
##########
@@ -0,0 +1,45 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+# RISON/JSON schemas for query parameters
+from marshmallow import fields, Schema, validate
+
+from superset.charts.schemas import (
+    datasource_name_description,
+    datasource_type_description,
+    datasource_uid_description,
+)
+
+
+class Datasource(Schema):
+    database_name = fields.String(description="Datasource name",)
+    datasource_name = fields.String(description=datasource_name_description,)
+    schema = fields.String(description="Datasource schema",)
+    datasource_type = fields.String(
+        description=datasource_type_description,
+        validate=validate.OneOf(choices=("druid", "table", "view")),
+        allow_none=True,

Review comment:
       I think that `get_datasource_by_name` will not handle well a `None` here, replace by `required=True`.
   

##########
File path: superset/cachekeys/api.py
##########
@@ -0,0 +1,98 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from flask import request, Response
+from flask_appbuilder import expose
+from flask_appbuilder.api import safe
+from flask_appbuilder.models.sqla.interface import SQLAInterface
+from flask_appbuilder.security.decorators import protect
+from jsonschema import ValidationError
+
+from superset.cachekeys.schemas import CacheInvalidationRequestSchema
+from superset.connectors.connector_registry import ConnectorRegistry
+from superset.extensions import cache_manager, db, event_logger
+from superset.models.cache import CacheKey
+from superset.views.base_api import BaseSupersetModelRestApi, statsd_metrics
+
+
+class CacheRestApi(BaseSupersetModelRestApi):
+    datamodel = SQLAInterface(CacheKey)
+    resource_name = "cache"

Review comment:
       nit: since this was renamed to cache, evaluate if placing everything on `cachekeys` still makes sense

##########
File path: tests/base_tests.py
##########
@@ -69,6 +71,39 @@ def get_resp(
     return resp.data.decode("utf-8")
 
 
+def post_assert_metric(

Review comment:
       Is this the same but on a different place?

##########
File path: tests/cachekeys/api_tests.py
##########
@@ -0,0 +1,131 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+# isort:skip_file
+"""Unit tests for Superset"""
+from tests.test_app import app  # noqa
+
+from superset.extensions import cache_manager, db
+from superset.models.cache import CacheKey
+from tests.base_tests import (
+    SupersetTestCase,
+    post_assert_metric,
+    test_client,
+    logged_in_admin,
+)  # noqa
+
+
+def test_invalidate_cache(logged_in_admin):
+    rv = post_assert_metric(
+        test_client,
+        "api/v1/cache/invalidate",
+        {"datasource_uids": ["3__table"]},
+        "invalidate",
+    )
+    assert rv.status_code == 201
+
+
+def test_invalidate_existing_cache(logged_in_admin):
+    db.session.add(CacheKey(cache_key="cache_key", datasource_uid="3__table"))
+    db.session.commit()
+    cache_manager.cache.set("cache_key", "value")
+
+    rv = post_assert_metric(
+        test_client,
+        "api/v1/cache/invalidate",
+        {"datasource_uids": ["3__table"]},
+        "invalidate",
+    )
+
+    assert rv.status_code == 201
+    assert cache_manager.cache.get("cache_key") == None
+    assert (
+        not db.session.query(CacheKey).filter(CacheKey.cache_key == "cache_key").first()
+    )
+
+
+def test_invalidate_existing_caches(logged_in_admin):
+    bn = SupersetTestCase.get_birth_names_dataset()
+
+    db.session.add(CacheKey(cache_key="cache_key1", datasource_uid="3__druid"))
+    db.session.add(CacheKey(cache_key="cache_key2", datasource_uid="3__druid"))
+    db.session.add(CacheKey(cache_key="cache_key4", datasource_uid=f"{bn.id}__table"))
+    db.session.add(CacheKey(cache_key="cache_keyX", datasource_uid="X__table"))
+    db.session.commit()
+
+    cache_manager.cache.set("cache_key1", "value")
+    cache_manager.cache.set("cache_key2", "value")
+    cache_manager.cache.set("cache_key4", "value")
+    cache_manager.cache.set("cache_keyX", "value")
+
+    rv = post_assert_metric(
+        test_client,
+        "api/v1/cache/invalidate",
+        {
+            "datasource_uids": ["3__druid", "4__druid"],

Review comment:
       can we add a test with edge case POST's: `{ "datasource_uids": [], "datasources": []}`
   `{ "datasource_uids": [], "datasources": [{"datasource_name": "", "datasource_type": null}]}`
   
   etc

##########
File path: superset/cachekeys/api.py
##########
@@ -0,0 +1,98 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from flask import request, Response
+from flask_appbuilder import expose
+from flask_appbuilder.api import safe
+from flask_appbuilder.models.sqla.interface import SQLAInterface
+from flask_appbuilder.security.decorators import protect
+from jsonschema import ValidationError
+
+from superset.cachekeys.schemas import CacheInvalidationRequestSchema
+from superset.connectors.connector_registry import ConnectorRegistry
+from superset.extensions import cache_manager, db, event_logger
+from superset.models.cache import CacheKey
+from superset.views.base_api import BaseSupersetModelRestApi, statsd_metrics
+
+
+class CacheRestApi(BaseSupersetModelRestApi):
+    datamodel = SQLAInterface(CacheKey)
+    resource_name = "cache"
+    allow_browser_login = True
+    class_permission_name = "CacheRestApi"
+
+    openapi_spec_component_schemas = (CacheInvalidationRequestSchema,)
+
+    @expose("/invalidate", methods=["POST"])
+    @event_logger.log_this
+    @protect()
+    @safe
+    @statsd_metrics
+    def invalidate(self) -> Response:
+        """
+        Takes a list of datasources, finds the associated cache records and
+        invalidates them and removes the database records
+
+        ---
+        post:
+          description: >-
+            Takes a list of datasources, finds the associated cache records and
+            invalidates them and removes the database records
+          requestBody:
+            description: >-
+              A list of datasources uuid or the tuples of database and datasource names
+            required: true
+            content:
+              application/json:
+                schema:
+                  $ref: "#/components/schemas/CacheInvalidationRequestSchema"
+          responses:
+            201:
+              description: cache was successfully invalidated
+            400:
+              $ref: '#/components/responses/400'
+            500:
+              $ref: '#/components/responses/500'
+        """
+        try:
+            datasources = CacheInvalidationRequestSchema().load(request.json)
+        except KeyError:
+            return self.response_400(message="Request is incorrect")
+        except ValidationError as error:
+            return self.response_400(message=error.message)
+        datasource_uids = set(datasources.get("datasource_uids", []))
+        for ds in datasources.get("datasources", []):
+            ds_obj = ConnectorRegistry.get_datasource_by_name(
+                session=db.session,
+                datasource_type=ds.get("datasource_type"),
+                datasource_name=ds.get("datasource_name"),
+                schema=ds.get("schema"),
+                database_name=ds.get("database_name"),
+            )
+            if ds_obj:
+                datasource_uids.add(ds_obj.uid)
+
+        cache_keys = (
+            db.session.query(CacheKey)
+            .filter(CacheKey.datasource_uid.in_(datasource_uids))
+            .all()
+        )
+        if cache_keys:
+            cache_manager.cache.delete_many(*[c.cache_key for c in cache_keys])

Review comment:
       A couple of questions:
   - if a cache `delete_many` fails, we may end up with delete keys on the cache that still exist on `CacheKey` will a next run of `delete_many` fail if it can't find a key?
   - the same applies if a delete `CacheKey` fails, would also be safer to try/except and rollback
   




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org