You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@superset.apache.org by "villebro (via GitHub)" <gi...@apache.org> on 2024/06/18 07:51:10 UTC
[PR] fix(key-value): use flush instead of commit [superset]
villebro opened a new pull request, #29286:
URL: https://github.com/apache/superset/pull/29286
### SUMMARY
In the key value commands, we are often returning the keys from newly generated ORM objects, which are persisted in the `key_value` table. As we're currently using `commit()` to persist the objects prior to reading the keys, the session is reset, requiring a new session for fetching the keys. This can cause issues when distributing reads to read replicas, as the data may sometimes not be available:
```
sqlalchemy.orm.exc.ObjectDeletedError: Instance '<KeyValueEntry at 0x7fe508293280>' has been deleted, or its row is otherwise not present.
```
Furthermore, since we often delete expired records before inserting new ones, collisions can occur if the commands aren't handled in the same session. For this reason we should avoid committing when flushing is sufficient, as stated in [SIP-99A](#25107):
> It is worth noting that objects within the session, in either pending* or persisted state, can be queried albeit not having been committed to the database. We have a tendency to over commit (be that in the code or tests) resulting in fractured/partial atomic units. Typically working with objects in a persisted state is sufficient.
While fixing these, I also noticed a case where we're purging expired entries in the Metastore cache's `add` method _after_ inserting a new entry. This has the potential of causing a key conflict, as a previous entry with the same key may already be present but expired (note that in the `get` key value command we're filtering out expired entries: https://github.com/apache/superset/blob/c7b8ae9013dcbffb3bb0dda4dc1d57cc9c18048c/superset/commands/key_value/get.py#L70-L72)
Finally, as we're replacing `commit()` with `flush()`, we need to ensure that transactions are scoped correctly. Usually we want to encapsulate many commands in a single transaction. For example, for the create/upsert commands, we usually want to do the following in a single transaction:
1. delete expired entries
2. insert new entry
However, there are instances where we explicitly need for commits to happen during execution flow. An good example of this is the Key Value Distributed Lock, which needs to commit the lock to the metastore so other workers can observe it. For this reason we're adding explicit `db.session.commit()` calls both in the Metastore Cache and Lock context manager to ensure the values are persisted in the db during execution flow.
### BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF
<!--- Skip this if not applicable -->
### TESTING INSTRUCTIONS
<!--- Required! What steps can be taken to manually verify the changes? -->
### ADDITIONAL INFORMATION
<!--- Check any relevant boxes with "x" -->
<!--- HINT: Include "Fixes #nnn" if you are fixing an existing issue -->
- [ ] Has associated issue:
- [ ] Required feature flags:
- [ ] Changes UI
- [ ] Includes DB Migration (follow approval process in [SIP-59](https://github.com/apache/superset/issues/13351))
- [ ] Migration is atomic, supports rollback & is backwards-compatible
- [ ] Confirm DB migration upgrade and downgrade tested
- [ ] Runtime estimates and downtime expectations provided
- [ ] Introduces new feature or API
- [ ] Removes existing feature or API
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(key-value): use flush instead of commit [superset]
Posted by "john-bodley (via GitHub)" <gi...@apache.org>.
john-bodley commented on code in PR #29286:
URL: https://github.com/apache/superset/pull/29286#discussion_r1644818384
##########
superset/commands/dashboard/permalink/create.py:
##########
@@ -62,6 +63,7 @@ def run(self) -> str:
codec=self.codec,
).run()
assert key.id # for type checks
+ db.session.commit()
Review Comment:
Does the `UpsertKeyValueCommand` not commit? Note that _hopefully_ (if merged) that https://github.com/apache/superset/pull/24969 should remove the need to have to commit in various places given it violates the "unit of work" philosophy.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(key-value): use flush instead of commit [superset]
Posted by "michael-s-molina (via GitHub)" <gi...@apache.org>.
michael-s-molina commented on code in PR #29286:
URL: https://github.com/apache/superset/pull/29286#discussion_r1647489486
##########
superset/commands/dashboard/permalink/create.py:
##########
@@ -62,6 +63,7 @@ def run(self) -> str:
codec=self.codec,
).run()
assert key.id # for type checks
+ db.session.commit()
Review Comment:
> This would make it possible to use all existing commands both as full units of work, or as part of a bigger chain, with minimal code duplication
We'll achieve that with nested transactions using [being_nested](https://docs.sqlalchemy.org/en/20/orm/session_api.html#sqlalchemy.orm.Session.begin_nested). Check https://github.com/apache/superset/pull/24969 for reference.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(key-value): use flush instead of commit [superset]
Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #29286:
URL: https://github.com/apache/superset/pull/29286#issuecomment-2175518503
## [Codecov](https://app.codecov.io/gh/apache/superset/pull/29286?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
Attention: Patch coverage is `65.21739%` with `8 lines` in your changes missing coverage. Please review.
> Project coverage is 64.73%. Comparing base [(`76d897e`)](https://app.codecov.io/gh/apache/superset/commit/76d897eaa2f9e137102bc194c2e3109c29d0348f?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) to head [(`966e2e4`)](https://app.codecov.io/gh/apache/superset/commit/966e2e44cb8d34add96ed8db6231e4e9649bf96f?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
> Report is 329 commits behind head on master.
| [Files](https://app.codecov.io/gh/apache/superset/pull/29286?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [superset/extensions/metastore\_cache.py](https://app.codecov.io/gh/apache/superset/pull/29286?src=pr&el=tree&filepath=superset%2Fextensions%2Fmetastore_cache.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c3VwZXJzZXQvZXh0ZW5zaW9ucy9tZXRhc3RvcmVfY2FjaGUucHk=) | 14.28% | [6 Missing :warning: ](https://app.codecov.io/gh/apache/superset/pull/29286?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [superset/commands/key\_value/update.py](https://app.codecov.io/gh/apache/superset/pull/29286?src=pr&el=tree&filepath=superset%2Fcommands%2Fkey_value%2Fupdate.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c3VwZXJzZXQvY29tbWFuZHMva2V5X3ZhbHVlL3VwZGF0ZS5weQ==) | 0.00% | [1 Missing :warning: ](https://app.codecov.io/gh/apache/superset/pull/29286?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [superset/commands/key\_value/upsert.py](https://app.codecov.io/gh/apache/superset/pull/29286?src=pr&el=tree&filepath=superset%2Fcommands%2Fkey_value%2Fupsert.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c3VwZXJzZXQvY29tbWFuZHMva2V5X3ZhbHVlL3Vwc2VydC5weQ==) | 0.00% | [1 Missing :warning: ](https://app.codecov.io/gh/apache/superset/pull/29286?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #29286 +/- ##
==========================================
+ Coverage 60.48% 64.73% +4.24%
==========================================
Files 1931 518 -1413
Lines 76236 37561 -38675
Branches 8568 0 -8568
==========================================
- Hits 46114 24316 -21798
+ Misses 28017 13245 -14772
+ Partials 2105 0 -2105
```
| [Flag](https://app.codecov.io/gh/apache/superset/pull/29286/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [hive](https://app.codecov.io/gh/apache/superset/pull/29286/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.93% <21.73%> (-0.24%)` | :arrow_down: |
| [javascript](https://app.codecov.io/gh/apache/superset/pull/29286/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [presto](https://app.codecov.io/gh/apache/superset/pull/29286/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `53.54% <21.73%> (-0.26%)` | :arrow_down: |
| [python](https://app.codecov.io/gh/apache/superset/pull/29286/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `64.73% <65.21%> (+1.24%)` | :arrow_up: |
| [unit](https://app.codecov.io/gh/apache/superset/pull/29286/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `59.28% <65.21%> (+1.65%)` | :arrow_up: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
</details>
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/superset/pull/29286?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(key-value): use flush instead of commit [superset]
Posted by "villebro (via GitHub)" <gi...@apache.org>.
villebro commented on code in PR #29286:
URL: https://github.com/apache/superset/pull/29286#discussion_r1645008173
##########
superset/commands/dashboard/permalink/create.py:
##########
@@ -62,6 +63,7 @@ def run(self) -> str:
codec=self.codec,
).run()
assert key.id # for type checks
+ db.session.commit()
Review Comment:
While this has previously been considered an antipattern (=having `commit: bool` flag in these types of methods or similar), I'm personally also kind of leaning in that direction. This would make it possible to use all existing commands both as full units of work, or as part of a bigger chain, with minimal code duplication. Thoughts @john-bodley ? Also pinging @michael-s-molina as you've worked on these components in the past.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(key-value): use flush instead of commit [superset]
Posted by "villebro (via GitHub)" <gi...@apache.org>.
villebro commented on code in PR #29286:
URL: https://github.com/apache/superset/pull/29286#discussion_r1644437052
##########
superset/utils/lock.py:
##########
@@ -77,21 +82,23 @@ def KeyValueDistributedLock( # pylint: disable=invalid-name
from superset.commands.key_value.delete import DeleteKeyValueCommand
from superset.commands.key_value.delete_expired import DeleteExpiredKeyValueCommand
- key = uuid.uuid5(uuid.uuid5(uuid.NAMESPACE_DNS, namespace), serialize(kwargs))
+ key = get_key(namespace, **kwargs)
logger.debug("Acquiring lock on namespace %s for key %s", namespace, key)
try:
DeleteExpiredKeyValueCommand(resource=KeyValueResource.LOCK).run()
CreateKeyValueCommand(
resource=KeyValueResource.LOCK,
- codec=PickleKeyValueCodec(),
+ codec=JsonKeyValueCodec(),
Review Comment:
This is fixed in a separate PR (I had to pull in the change here as the test needed to be updated): #29285
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(key-value): use flush instead of commit [superset]
Posted by "villebro (via GitHub)" <gi...@apache.org>.
villebro commented on code in PR #29286:
URL: https://github.com/apache/superset/pull/29286#discussion_r1644836178
##########
superset/commands/dashboard/permalink/create.py:
##########
@@ -62,6 +63,7 @@ def run(self) -> str:
codec=self.codec,
).run()
assert key.id # for type checks
+ db.session.commit()
Review Comment:
Here I'm proposing to not commit in these commands, as we have cases where we want to chain multiple commands together. But if we consider each Key Value command a unit of work, then we should naturally commit there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(key-value): use flush instead of commit [superset]
Posted by "villebro (via GitHub)" <gi...@apache.org>.
villebro commented on code in PR #29286:
URL: https://github.com/apache/superset/pull/29286#discussion_r1647528724
##########
superset/commands/dashboard/permalink/create.py:
##########
@@ -62,6 +63,7 @@ def run(self) -> str:
codec=self.codec,
).run()
assert key.id # for type checks
+ db.session.commit()
Review Comment:
> We'll achieve that with nested transactions using [being_nested](https://docs.sqlalchemy.org/en/20/orm/session_api.html#sqlalchemy.orm.Session.begin_nested). Check https://github.com/apache/superset/pull/24969 for reference.
@michael-s-molina I read up on the docs I could find, and I agree, this should be an elegant solution to this dual use case.
@john-bodley it would be great if you could add a description to #24969 so we can start reviewing/testing it. I'm keen on getting this important refactor in, as it will have a profound impact on the general quality and perormance of the backend.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(key-value): use flush instead of commit [superset]
Posted by "nytai (via GitHub)" <gi...@apache.org>.
nytai commented on code in PR #29286:
URL: https://github.com/apache/superset/pull/29286#discussion_r1644954794
##########
superset/commands/dashboard/permalink/create.py:
##########
@@ -62,6 +63,7 @@ def run(self) -> str:
codec=self.codec,
).run()
assert key.id # for type checks
+ db.session.commit()
Review Comment:
i think there's a case to be made for both. Maybe we can have default behavior be to commit but provide an option to skip for use cases where there's a command chaining multiple sub-commands? It would be inconsistent for some commands to commit while others requiring the caller to commit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org