You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@superset.apache.org by "villebro (via GitHub)" <gi...@apache.org> on 2024/06/18 07:51:10 UTC

[PR] fix(key-value): use flush instead of commit [superset]

villebro opened a new pull request, #29286:
URL: https://github.com/apache/superset/pull/29286

   ### SUMMARY
   In the key value commands, we are often returning the keys from newly generated ORM objects, which are persisted in the `key_value` table. As we're currently using `commit()` to persist the objects prior to reading the keys, the session is reset, requiring a new session for fetching the keys. This can cause issues when distributing reads to read replicas, as the data may sometimes not be available:
   
   ```
   sqlalchemy.orm.exc.ObjectDeletedError: Instance '<KeyValueEntry at 0x7fe508293280>' has been deleted, or its row is otherwise not present.
   ```
   
   Furthermore, since we often delete expired records before inserting new ones, collisions can occur if the commands aren't handled in the same session. For this reason we should avoid committing when flushing is sufficient, as stated in [SIP-99A](#25107):
   
   > It is worth noting that objects within the session, in either pending* or persisted state, can be queried albeit not having been committed to the database. We have a tendency to over commit (be that in the code or tests) resulting in fractured/partial atomic units. Typically working with objects in a persisted state is sufficient.
   
   While fixing these, I also noticed a case where we're purging expired entries in the Metastore cache's `add` method _after_ inserting a new entry. This has the potential of causing a key conflict, as a previous entry with the same key may already be present but expired (note that in the `get` key value command we're filtering out expired entries: https://github.com/apache/superset/blob/c7b8ae9013dcbffb3bb0dda4dc1d57cc9c18048c/superset/commands/key_value/get.py#L70-L72)
   
   Finally, as we're replacing `commit()` with `flush()`, we need to ensure that transactions are scoped correctly. Usually we want to encapsulate many commands in a single transaction. For example, for the create/upsert commands, we usually want to do the following in a single transaction:
   1. delete expired entries
   2. insert new entry
   
   However, there are instances where we explicitly need for commits to happen during execution flow. An good example of this is the Key Value Distributed Lock, which needs to commit the lock to the metastore so other workers can observe it. For this reason we're adding explicit `db.session.commit()` calls both in the Metastore Cache and Lock context manager to ensure the values are persisted in the db during execution flow.
   
   ### BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF
   <!--- Skip this if not applicable -->
   
   ### TESTING INSTRUCTIONS
   <!--- Required! What steps can be taken to manually verify the changes? -->
   
   ### ADDITIONAL INFORMATION
   <!--- Check any relevant boxes with "x" -->
   <!--- HINT: Include "Fixes #nnn" if you are fixing an existing issue -->
   - [ ] Has associated issue:
   - [ ] Required feature flags:
   - [ ] Changes UI
   - [ ] Includes DB Migration (follow approval process in [SIP-59](https://github.com/apache/superset/issues/13351))
     - [ ] Migration is atomic, supports rollback & is backwards-compatible
     - [ ] Confirm DB migration upgrade and downgrade tested
     - [ ] Runtime estimates and downtime expectations provided
   - [ ] Introduces new feature or API
   - [ ] Removes existing feature or API
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [PR] fix(key-value): use flush instead of commit [superset]

Posted by "john-bodley (via GitHub)" <gi...@apache.org>.
john-bodley commented on code in PR #29286:
URL: https://github.com/apache/superset/pull/29286#discussion_r1644818384


##########
superset/commands/dashboard/permalink/create.py:
##########
@@ -62,6 +63,7 @@ def run(self) -> str:
                 codec=self.codec,
             ).run()
             assert key.id  # for type checks
+            db.session.commit()

Review Comment:
   Does the `UpsertKeyValueCommand` not commit? Note that _hopefully_ (if merged) that https://github.com/apache/superset/pull/24969 should remove the need to have to commit in various places given it violates the "unit of work" philosophy.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [PR] fix(key-value): use flush instead of commit [superset]

Posted by "michael-s-molina (via GitHub)" <gi...@apache.org>.
michael-s-molina commented on code in PR #29286:
URL: https://github.com/apache/superset/pull/29286#discussion_r1647489486


##########
superset/commands/dashboard/permalink/create.py:
##########
@@ -62,6 +63,7 @@ def run(self) -> str:
                 codec=self.codec,
             ).run()
             assert key.id  # for type checks
+            db.session.commit()

Review Comment:
   > This would make it possible to use all existing commands both as full units of work, or as part of a bigger chain, with minimal code duplication
   
   We'll achieve that with nested transactions using [being_nested](https://docs.sqlalchemy.org/en/20/orm/session_api.html#sqlalchemy.orm.Session.begin_nested). Check https://github.com/apache/superset/pull/24969 for reference.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [PR] fix(key-value): use flush instead of commit [superset]

Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #29286:
URL: https://github.com/apache/superset/pull/29286#issuecomment-2175518503

   ## [Codecov](https://app.codecov.io/gh/apache/superset/pull/29286?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
   Attention: Patch coverage is `65.21739%` with `8 lines` in your changes missing coverage. Please review.
   > Project coverage is 64.73%. Comparing base [(`76d897e`)](https://app.codecov.io/gh/apache/superset/commit/76d897eaa2f9e137102bc194c2e3109c29d0348f?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) to head [(`966e2e4`)](https://app.codecov.io/gh/apache/superset/commit/966e2e44cb8d34add96ed8db6231e4e9649bf96f?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
   > Report is 329 commits behind head on master.
   
   | [Files](https://app.codecov.io/gh/apache/superset/pull/29286?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
   |---|---|---|
   | [superset/extensions/metastore\_cache.py](https://app.codecov.io/gh/apache/superset/pull/29286?src=pr&el=tree&filepath=superset%2Fextensions%2Fmetastore_cache.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c3VwZXJzZXQvZXh0ZW5zaW9ucy9tZXRhc3RvcmVfY2FjaGUucHk=) | 14.28% | [6 Missing :warning: ](https://app.codecov.io/gh/apache/superset/pull/29286?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   | [superset/commands/key\_value/update.py](https://app.codecov.io/gh/apache/superset/pull/29286?src=pr&el=tree&filepath=superset%2Fcommands%2Fkey_value%2Fupdate.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c3VwZXJzZXQvY29tbWFuZHMva2V5X3ZhbHVlL3VwZGF0ZS5weQ==) | 0.00% | [1 Missing :warning: ](https://app.codecov.io/gh/apache/superset/pull/29286?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   | [superset/commands/key\_value/upsert.py](https://app.codecov.io/gh/apache/superset/pull/29286?src=pr&el=tree&filepath=superset%2Fcommands%2Fkey_value%2Fupsert.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c3VwZXJzZXQvY29tbWFuZHMva2V5X3ZhbHVlL3Vwc2VydC5weQ==) | 0.00% | [1 Missing :warning: ](https://app.codecov.io/gh/apache/superset/pull/29286?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   
   <details><summary>Additional details and impacted files</summary>
   
   
   ```diff
   @@            Coverage Diff             @@
   ##           master   #29286      +/-   ##
   ==========================================
   + Coverage   60.48%   64.73%   +4.24%     
   ==========================================
     Files        1931      518    -1413     
     Lines       76236    37561   -38675     
     Branches     8568        0    -8568     
   ==========================================
   - Hits        46114    24316   -21798     
   + Misses      28017    13245   -14772     
   + Partials     2105        0    -2105     
   ```
   
   | [Flag](https://app.codecov.io/gh/apache/superset/pull/29286/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
   |---|---|---|
   | [hive](https://app.codecov.io/gh/apache/superset/pull/29286/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.93% <21.73%> (-0.24%)` | :arrow_down: |
   | [javascript](https://app.codecov.io/gh/apache/superset/pull/29286/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [presto](https://app.codecov.io/gh/apache/superset/pull/29286/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `53.54% <21.73%> (-0.26%)` | :arrow_down: |
   | [python](https://app.codecov.io/gh/apache/superset/pull/29286/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `64.73% <65.21%> (+1.24%)` | :arrow_up: |
   | [unit](https://app.codecov.io/gh/apache/superset/pull/29286/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `59.28% <65.21%> (+1.65%)` | :arrow_up: |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   
   </details>
   
   [:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/superset/pull/29286?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).   
   :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [PR] fix(key-value): use flush instead of commit [superset]

Posted by "villebro (via GitHub)" <gi...@apache.org>.
villebro commented on code in PR #29286:
URL: https://github.com/apache/superset/pull/29286#discussion_r1645008173


##########
superset/commands/dashboard/permalink/create.py:
##########
@@ -62,6 +63,7 @@ def run(self) -> str:
                 codec=self.codec,
             ).run()
             assert key.id  # for type checks
+            db.session.commit()

Review Comment:
   While this has previously been considered an antipattern (=having `commit: bool` flag in these types of methods or similar), I'm personally also kind of leaning in that direction. This would make it possible to use all existing commands both as full units of work, or as part of a bigger chain, with minimal code duplication. Thoughts @john-bodley ? Also pinging @michael-s-molina as you've worked on these components in the past.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [PR] fix(key-value): use flush instead of commit [superset]

Posted by "villebro (via GitHub)" <gi...@apache.org>.
villebro commented on code in PR #29286:
URL: https://github.com/apache/superset/pull/29286#discussion_r1644437052


##########
superset/utils/lock.py:
##########
@@ -77,21 +82,23 @@ def KeyValueDistributedLock(  # pylint: disable=invalid-name
     from superset.commands.key_value.delete import DeleteKeyValueCommand
     from superset.commands.key_value.delete_expired import DeleteExpiredKeyValueCommand
 
-    key = uuid.uuid5(uuid.uuid5(uuid.NAMESPACE_DNS, namespace), serialize(kwargs))
+    key = get_key(namespace, **kwargs)
     logger.debug("Acquiring lock on namespace %s for key %s", namespace, key)
     try:
         DeleteExpiredKeyValueCommand(resource=KeyValueResource.LOCK).run()
         CreateKeyValueCommand(
             resource=KeyValueResource.LOCK,
-            codec=PickleKeyValueCodec(),
+            codec=JsonKeyValueCodec(),

Review Comment:
   This is fixed in a separate PR (I had to pull in the change here as the test needed to be updated): #29285



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [PR] fix(key-value): use flush instead of commit [superset]

Posted by "villebro (via GitHub)" <gi...@apache.org>.
villebro commented on code in PR #29286:
URL: https://github.com/apache/superset/pull/29286#discussion_r1644836178


##########
superset/commands/dashboard/permalink/create.py:
##########
@@ -62,6 +63,7 @@ def run(self) -> str:
                 codec=self.codec,
             ).run()
             assert key.id  # for type checks
+            db.session.commit()

Review Comment:
   Here I'm proposing to not commit in these commands, as we have cases where we want to chain multiple commands together. But if we consider each Key Value command a unit of work, then we should naturally commit there.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [PR] fix(key-value): use flush instead of commit [superset]

Posted by "villebro (via GitHub)" <gi...@apache.org>.
villebro commented on code in PR #29286:
URL: https://github.com/apache/superset/pull/29286#discussion_r1647528724


##########
superset/commands/dashboard/permalink/create.py:
##########
@@ -62,6 +63,7 @@ def run(self) -> str:
                 codec=self.codec,
             ).run()
             assert key.id  # for type checks
+            db.session.commit()

Review Comment:
   > We'll achieve that with nested transactions using [being_nested](https://docs.sqlalchemy.org/en/20/orm/session_api.html#sqlalchemy.orm.Session.begin_nested). Check https://github.com/apache/superset/pull/24969 for reference.
   
   @michael-s-molina I read up on the docs I could find, and I agree, this should be an elegant solution to this dual use case.
   
   @john-bodley it would be great if you could add a description to #24969 so we can start reviewing/testing it. I'm keen on getting this important refactor in, as it will have a profound impact on the general quality and perormance of the backend.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [PR] fix(key-value): use flush instead of commit [superset]

Posted by "nytai (via GitHub)" <gi...@apache.org>.
nytai commented on code in PR #29286:
URL: https://github.com/apache/superset/pull/29286#discussion_r1644954794


##########
superset/commands/dashboard/permalink/create.py:
##########
@@ -62,6 +63,7 @@ def run(self) -> str:
                 codec=self.codec,
             ).run()
             assert key.id  # for type checks
+            db.session.commit()

Review Comment:
   i think there's a case to be made for both. Maybe we can have default behavior be to commit but provide an option to skip for use cases where there's a command chaining multiple sub-commands? It would be inconsistent for some commands to commit while others requiring the caller to commit. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org