You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@superset.apache.org by "nsivarajan (via GitHub)" <gi...@apache.org> on 2024/08/10 17:58:52 UTC
[PR] Feat (GAQ) : Add Redis Sentinel Support for Global Async Queries [superset]
nsivarajan opened a new pull request, #29912:
URL: https://github.com/apache/superset/pull/29912
<!---
Please write the PR title following the conventions at https://www.conventionalcommits.org/en/v1.0.0/
Example:
fix(dashboard): load charts correctly
-->
### SUMMARY
<!--- Describe the change below, including rationale and design decisions -->
This PR introduces a feature that allows the Async Query Manager to use a configured cache backend through the new **GLOBAL_ASYNC_QUERIES_CACHE_BACKEND** setting. To maintain backward compatibility, the existing **GLOBAL_ASYNC_QUERIES_REDIS_CONFIG** setting is still supported but will be deprecated in the future. Additionally, this update _introduces support for Redis Sentinel caching_, which eliminates the single point of failure associated with the previous standalone Redis configuration.
References taken from this Idea https://github.com/apache/superset/discussions/28839
### TESTING INSTRUCTIONS
<!--- Required! What steps can be taken to manually verify the changes? -->
Configure **GLOBAL_ASYNC_QUERIES_CACHE_BACKEND** in _config.py_ or _superset_config.py_ with the appropriate properties:
Set CACHE_TYPE to "RedisCache" for Redis cache backend.
Set CACHE_TYPE to "RedisSentinelCache" for Redis Sentinel cache backend.
Set CACHE_TYPE to None to fall back on the previous GLOBAL_ASYNC_QUERIES_REDIS_CONFIG.
### ADDITIONAL INFORMATION
<!--- Check any relevant boxes with "x" -->
<!--- HINT: Include "Fixes #nnn" if you are fixing an existing issue -->
- [ ] Has associated issue:
- [ ] Required feature flags:
- [ ] Changes UI
- [ ] Includes DB Migration (follow approval process in [SIP-59](https://github.com/apache/superset/issues/13351))
- [ ] Migration is atomic, supports rollback & is backwards-compatible
- [ ] Confirm DB migration upgrade and downgrade tested
- [ ] Runtime estimates and downtime expectations provided
- [ ] Introduces new feature or API
- [ ] Removes existing feature or API
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(GAQ): Add Redis Sentinel Support for Global Async Queries [superset]
Posted by "nsivarajan (via GitHub)" <gi...@apache.org>.
nsivarajan commented on code in PR #29912:
URL: https://github.com/apache/superset/pull/29912#discussion_r1734682927
##########
superset/async_events/async_query_manager.py:
##########
@@ -16,18 +16,27 @@
# under the License.
import logging
import uuid
-from typing import Any, Literal, Optional
+from typing import Any, Dict, Literal, Optional, Union
Review Comment:
Thank you for pointing that out. I'll make sure to update the code accordingly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(GAQ): Add Redis Sentinel Support for Global Async Queries [superset]
Posted by "villebro (via GitHub)" <gi...@apache.org>.
villebro commented on code in PR #29912:
URL: https://github.com/apache/superset/pull/29912#discussion_r1714434237
##########
superset/async_events/async_query_manager.py:
##########
@@ -16,18 +16,27 @@
# under the License.
import logging
import uuid
-from typing import Any, Literal, Optional
+from typing import Any, Dict, Literal, Optional, Union
Review Comment:
nit: we can now use `dict` and `|` in type annotations as long as we do the following import at the beginning:
```python
from __future__ iimport annotations
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(GAQ): Add Redis Sentinel Support for Global Async Queries [superset]
Posted by "villebro (via GitHub)" <gi...@apache.org>.
villebro commented on code in PR #29912:
URL: https://github.com/apache/superset/pull/29912#discussion_r1736740946
##########
superset/async_events/async_query_manager.py:
##########
@@ -230,14 +257,35 @@ def submit_chart_data_job(
def read_events(
self, channel: str, last_id: Optional[str]
) -> list[Optional[dict[str, Any]]]:
+ if not self._cache:
+ raise CacheBackendNotInitialized("Cache backend not initialized")
+
stream_name = f"{self._stream_prefix}{channel}"
start_id = increment_id(last_id) if last_id else "-"
- results = self._redis.xrange(stream_name, start_id, "+", self.MAX_EVENT_COUNT)
+ results = self._cache.xrange(stream_name, start_id, "+", self.MAX_EVENT_COUNT)
+ # Decode bytes to strings, decode_responses is not supported at RedisCache and RedisSentinelCache
+ if isinstance(self._cache, (RedisSentinelCacheBackend, RedisCacheBackend)):
+ decoded_results = [
+ (
+ event_id.decode("utf-8"),
+ {
+ key.decode("utf-8"): value.decode("utf-8")
+ for key, value in event_data.items()
+ },
+ )
+ for event_id, event_data in results
+ ]
+ return (
+ [] if not decoded_results else list(map(parse_event, decoded_results))
+ )
Review Comment:
Got it, I hadn't noticed we were using `redis.Redis`, I thought we were using `RedisCache` from `flask_caching` here. Thanks for clarifying.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(GAQ): Add Redis Sentinel Support for Global Async Queries [superset]
Posted by "villebro (via GitHub)" <gi...@apache.org>.
villebro commented on PR #29912:
URL: https://github.com/apache/superset/pull/29912#issuecomment-2322211871
Thanks @nsivarajan, I really appreciate the help! Here's the project card: https://github.com/orgs/apache/projects/345?pane=issue&itemId=78081192
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(GAQ): Add Redis Sentinel Support for Global Async Queries [superset]
Posted by "villebro (via GitHub)" <gi...@apache.org>.
villebro commented on code in PR #29912:
URL: https://github.com/apache/superset/pull/29912#discussion_r1736706207
##########
tests/integration_tests/tasks/async_queries_tests.py:
##########
@@ -37,18 +43,37 @@
)
from tests.integration_tests.test_app import app
+# Define the cache backends once
+cache_backends = {
+ "RedisCacheBackend": mock.Mock(spec=RedisCacheBackend),
+ "RedisSentinelCacheBackend": mock.Mock(spec=RedisSentinelCacheBackend),
+ "redis.Redis": mock.Mock(spec=redis.Redis),
+}
+
Review Comment:
I believe these are now redundant as we have the parametrized tests?
```suggestion
# Define the cache backends once
cache_backends = {
"RedisCacheBackend": mock.Mock(spec=RedisCacheBackend),
"RedisSentinelCacheBackend": mock.Mock(spec=RedisSentinelCacheBackend),
"redis.Redis": mock.Mock(spec=redis.Redis),
}
```
##########
tests/unit_tests/async_events/async_query_manager_tests.py:
##########
@@ -17,26 +17,37 @@
from unittest import mock
from unittest.mock import ANY, Mock
+import redis
from flask import g
from jwt import encode
-from pytest import fixture, raises
+from pytest import fixture, mark, raises
from superset import security_manager
from superset.async_events.async_query_manager import (
AsyncQueryManager,
AsyncQueryTokenException,
)
+from superset.async_events.cache_backend import (
+ RedisCacheBackend,
+ RedisSentinelCacheBackend,
+)
JWT_TOKEN_SECRET = "some_secret"
JWT_TOKEN_COOKIE_NAME = "superset_async_jwt"
+# Define the cache backends once as mocks
+# cache_backends = {
+# "RedisCacheBackend": mock.Mock(spec=RedisCacheBackend),
+# "RedisSentinelCacheBackend": mock.Mock(spec=RedisSentinelCacheBackend),
+# "redis.Redis": mock.Mock(spec=redis.Redis),
+# }
+
Review Comment:
Same here:
```suggestion
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(GAQ): Add Redis Sentinel Support for Global Async Queries [superset]
Posted by "nsivarajan (via GitHub)" <gi...@apache.org>.
nsivarajan commented on PR #29912:
URL: https://github.com/apache/superset/pull/29912#issuecomment-2322096562
Thanks, @villebro, for the feedback and type annotation tips!
I appreciate your guidance throughout this process. I'll go ahead and open a new PR to remove the old config option and will reference the task once it's created on the project board, so it’ll be ready when the 5.0 breaking window opens up. Looking forward to wrapping this up and moving on to the next steps!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(GAQ): Add Redis Sentinel Support for Global Async Queries [superset]
Posted by "nsivarajan (via GitHub)" <gi...@apache.org>.
nsivarajan commented on code in PR #29912:
URL: https://github.com/apache/superset/pull/29912#discussion_r1736614424
##########
tests/unit_tests/async_events/async_query_manager_tests.py:
##########
@@ -26,17 +27,27 @@
AsyncQueryManager,
AsyncQueryTokenException,
)
+from superset.async_events.cache_backend import (
+ RedisCacheBackend,
+ RedisSentinelCacheBackend,
+)
JWT_TOKEN_SECRET = "some_secret"
JWT_TOKEN_COOKIE_NAME = "superset_async_jwt"
+# Define the cache backends once as mocks
+cache_backends = {
+ "RedisCacheBackend": mock.Mock(spec=RedisCacheBackend),
+ "RedisSentinelCacheBackend": mock.Mock(spec=RedisSentinelCacheBackend),
+ "redis.Redis": mock.Mock(spec=redis.Redis),
+}
+
Review Comment:
Using `pytest.mark.parametrize` with `mock` from `unittest` caused compatibility issues. Switching to `parameterized.expand` resolved these issues and successfully covered the test cases, leading to a successful CI run. I’ll revisit this case later for further investigation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(GAQ): Add Redis Sentinel Support for Global Async Queries [superset]
Posted by "villebro (via GitHub)" <gi...@apache.org>.
villebro commented on code in PR #29912:
URL: https://github.com/apache/superset/pull/29912#discussion_r1736697593
##########
tests/unit_tests/async_events/async_query_manager_tests.py:
##########
@@ -26,17 +27,27 @@
AsyncQueryManager,
AsyncQueryTokenException,
)
+from superset.async_events.cache_backend import (
+ RedisCacheBackend,
+ RedisSentinelCacheBackend,
+)
JWT_TOKEN_SECRET = "some_secret"
JWT_TOKEN_COOKIE_NAME = "superset_async_jwt"
+# Define the cache backends once as mocks
+cache_backends = {
+ "RedisCacheBackend": mock.Mock(spec=RedisCacheBackend),
+ "RedisSentinelCacheBackend": mock.Mock(spec=RedisSentinelCacheBackend),
+ "redis.Redis": mock.Mock(spec=redis.Redis),
+}
+
Review Comment:
@nsivarajan I believe it's generally suggested to use `pytest_mock` with `pytest` test cases. I don't want to drag out this review process more than necessary, but I think the cleanest solution would be one with `pytest.mark.parametrize` and `pytest_mock`. LMKWYT, but in the meantime let me re-review the functional parts.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(GAQ): Add Redis Sentinel Support for Global Async Queries [superset]
Posted by "nsivarajan (via GitHub)" <gi...@apache.org>.
nsivarajan commented on code in PR #29912:
URL: https://github.com/apache/superset/pull/29912#discussion_r1738361124
##########
tests/unit_tests/async_events/async_query_manager_tests.py:
##########
@@ -17,26 +17,37 @@
from unittest import mock
from unittest.mock import ANY, Mock
+import redis
from flask import g
from jwt import encode
-from pytest import fixture, raises
+from pytest import fixture, mark, raises
from superset import security_manager
from superset.async_events.async_query_manager import (
AsyncQueryManager,
AsyncQueryTokenException,
)
+from superset.async_events.cache_backend import (
+ RedisCacheBackend,
+ RedisSentinelCacheBackend,
+)
JWT_TOKEN_SECRET = "some_secret"
JWT_TOKEN_COOKIE_NAME = "superset_async_jwt"
+# Define the cache backends once as mocks
+# cache_backends = {
+# "RedisCacheBackend": mock.Mock(spec=RedisCacheBackend),
+# "RedisSentinelCacheBackend": mock.Mock(spec=RedisSentinelCacheBackend),
+# "redis.Redis": mock.Mock(spec=redis.Redis),
+# }
+
Review Comment:
Updated in last commit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(GAQ): Add Redis Sentinel Support for Global Async Queries [superset]
Posted by "nsivarajan (via GitHub)" <gi...@apache.org>.
nsivarajan commented on code in PR #29912:
URL: https://github.com/apache/superset/pull/29912#discussion_r1738361612
##########
tests/integration_tests/tasks/async_queries_tests.py:
##########
@@ -37,18 +43,37 @@
)
from tests.integration_tests.test_app import app
+# Define the cache backends once
+cache_backends = {
+ "RedisCacheBackend": mock.Mock(spec=RedisCacheBackend),
+ "RedisSentinelCacheBackend": mock.Mock(spec=RedisSentinelCacheBackend),
+ "redis.Redis": mock.Mock(spec=redis.Redis),
+}
+
Review Comment:
Handled in last commit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(GAQ): Add Redis Sentinel Support for Global Async Queries [superset]
Posted by "nsivarajan (via GitHub)" <gi...@apache.org>.
nsivarajan commented on PR #29912:
URL: https://github.com/apache/superset/pull/29912#issuecomment-2290288475
@villebro Thanks for the feedback! Looking forward to your review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(GAQ): Add Redis Sentinel Support for Global Async Queries [superset]
Posted by "villebro (via GitHub)" <gi...@apache.org>.
villebro commented on PR #29912:
URL: https://github.com/apache/superset/pull/29912#issuecomment-2284511433
@nsivarajan thanks for the PR - I think this PR is welcomed even if there's ongoing work to modernize the async task framework. Looks generally good, but let me do a proper review shortly (either today or tomorrow)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(GAQ): Add Redis Sentinel Support for Global Async Queries [superset]
Posted by "nsivarajan (via GitHub)" <gi...@apache.org>.
nsivarajan commented on code in PR #29912:
URL: https://github.com/apache/superset/pull/29912#discussion_r1738375979
##########
tests/unit_tests/async_events/async_query_manager_tests.py:
##########
@@ -26,17 +27,27 @@
AsyncQueryManager,
AsyncQueryTokenException,
)
+from superset.async_events.cache_backend import (
+ RedisCacheBackend,
+ RedisSentinelCacheBackend,
+)
JWT_TOKEN_SECRET = "some_secret"
JWT_TOKEN_COOKIE_NAME = "superset_async_jwt"
+# Define the cache backends once as mocks
+cache_backends = {
+ "RedisCacheBackend": mock.Mock(spec=RedisCacheBackend),
+ "RedisSentinelCacheBackend": mock.Mock(spec=RedisSentinelCacheBackend),
+ "redis.Redis": mock.Mock(spec=redis.Redis),
+}
+
Review Comment:
Thanks, @villebro , for the review. I agree that using the standard approach is ideal. However, I've encountered continuous failures when combining `pytest.mark.parametrize` with `mock`, which suggests there might be compatibility issues. After some research, I found references that indicate potential conflicts when using pytest features with unittest.TestCase subclasses and mock (see [pytest documentation](https://docs.pytest.org/en/stable/how-to/unittest.html#pytest-features-in-unittest-testcase-subclasses), [SeleniumBase issue](https://github.com/seleniumbase/SeleniumBase/issues/395), and [pytest issue](https://github.com/pytest-dev/pytest/issues/541#issuecomment-609956951)).
I noticed that our repository already uses `parameterized.expand` in similar cases, so I opted for it here to ensure the tests run smoothly.
Would it be possible to proceed with this PR using `parameterized.expand`? I'll continue investigating the issues with `pytest.mark.parametrize` and `mock` and plan to submit a follow-up PR if I find a viable solution.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(GAQ): Add Redis Sentinel Support for Global Async Queries [superset]
Posted by "rusackas (via GitHub)" <gi...@apache.org>.
rusackas commented on PR #29912:
URL: https://github.com/apache/superset/pull/29912#issuecomment-2305483174
@nsivarajan thank you for your patience in the meantime... good to see that CI is passing and there's no conflicts still, at least! :D
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(GAQ): Add Redis Sentinel Support for Global Async Queries [superset]
Posted by "villebro (via GitHub)" <gi...@apache.org>.
villebro commented on code in PR #29912:
URL: https://github.com/apache/superset/pull/29912#discussion_r1738836361
##########
superset/async_events/async_query_manager.py:
##########
@@ -55,13 +66,32 @@ def parse_event(event_data: tuple[str, dict[str, Any]]) -> dict[str, Any]:
return {"id": event_id, **json.loads(event_payload)}
-def increment_id(redis_id: str) -> str:
+def increment_id(entry_id: str) -> str:
# redis stream IDs are in this format: '1607477697866-0'
try:
- prefix, last = redis_id[:-1], int(redis_id[-1])
+ prefix, last = entry_id[:-1], int(entry_id[-1])
return prefix + str(last + 1)
except Exception: # pylint: disable=broad-except
- return redis_id
+ return entry_id
+
+
+def get_cache_backend(
+ config: dict[str, Any],
+) -> Union[RedisCacheBackend, RedisSentinelCacheBackend, redis.Redis]: # type: ignore
Review Comment:
nit: the new type annotations support this simpler format:
```suggestion
) -> RedisCacheBackend | RedisSentinelCacheBackend | redis.Redis: # type: ignore
```
##########
superset/async_events/async_query_manager.py:
##########
@@ -73,7 +103,7 @@ class AsyncQueryManager:
def __init__(self) -> None:
super().__init__()
- self._redis: redis.Redis # type: ignore
+ self._cache: Optional[BaseCache] = None
Review Comment:
Same here:
```suggestion
self._cache: BaseCache | None = None
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(GAQ): Add Redis Sentinel Support for Global Async Queries [superset]
Posted by "villebro (via GitHub)" <gi...@apache.org>.
villebro commented on code in PR #29912:
URL: https://github.com/apache/superset/pull/29912#discussion_r1738828593
##########
tests/unit_tests/async_events/async_query_manager_tests.py:
##########
@@ -26,17 +27,27 @@
AsyncQueryManager,
AsyncQueryTokenException,
)
+from superset.async_events.cache_backend import (
+ RedisCacheBackend,
+ RedisSentinelCacheBackend,
+)
JWT_TOKEN_SECRET = "some_secret"
JWT_TOKEN_COOKIE_NAME = "superset_async_jwt"
+# Define the cache backends once as mocks
+cache_backends = {
+ "RedisCacheBackend": mock.Mock(spec=RedisCacheBackend),
+ "RedisSentinelCacheBackend": mock.Mock(spec=RedisSentinelCacheBackend),
+ "redis.Redis": mock.Mock(spec=redis.Redis),
+}
+
Review Comment:
Thanks for putting in the extra effort on these tests @nsivarajan ! Let's not let this derail your PR - I can also take a stab at migrating the remaining components to `pytest` (seems like a fun coffee break challenge).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(GAQ): Add Redis Sentinel Support for Global Async Queries [superset]
Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #29912:
URL: https://github.com/apache/superset/pull/29912#issuecomment-2282269987
## [Codecov](https://app.codecov.io/gh/apache/superset/pull/29912?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
Attention: Patch coverage is `58.33333%` with `30 lines` in your changes missing coverage. Please review.
> Project coverage is 65.05%. Comparing base [(`76d897e`)](https://app.codecov.io/gh/apache/superset/commit/76d897eaa2f9e137102bc194c2e3109c29d0348f?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) to head [(`fa85476`)](https://app.codecov.io/gh/apache/superset/commit/fa854767d6aa2f1f8a1f03e208bd502e60b4a445?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
> Report is 579 commits behind head on master.
| [Files](https://app.codecov.io/gh/apache/superset/pull/29912?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [superset/async\_events/async\_query\_manager.py](https://app.codecov.io/gh/apache/superset/pull/29912?src=pr&el=tree&filepath=superset%2Fasync_events%2Fasync_query_manager.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c3VwZXJzZXQvYXN5bmNfZXZlbnRzL2FzeW5jX3F1ZXJ5X21hbmFnZXIucHk=) | 53.12% | [15 Missing :warning: ](https://app.codecov.io/gh/apache/superset/pull/29912?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [superset/async\_events/cache\_backend.py](https://app.codecov.io/gh/apache/superset/pull/29912?src=pr&el=tree&filepath=superset%2Fasync_events%2Fcache_backend.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c3VwZXJzZXQvYXN5bmNfZXZlbnRzL2NhY2hlX2JhY2tlbmQucHk=) | 59.45% | [15 Missing :warning: ](https://app.codecov.io/gh/apache/superset/pull/29912?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #29912 +/- ##
==========================================
+ Coverage 60.48% 65.05% +4.56%
==========================================
Files 1931 528 -1403
Lines 76236 38112 -38124
Branches 8568 0 -8568
==========================================
- Hits 46114 24793 -21321
+ Misses 28017 13319 -14698
+ Partials 2105 0 -2105
```
| [Flag](https://app.codecov.io/gh/apache/superset/pull/29912/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [hive](https://app.codecov.io/gh/apache/superset/pull/29912/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `49.00% <38.88%> (-0.16%)` | :arrow_down: |
| [javascript](https://app.codecov.io/gh/apache/superset/pull/29912/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [presto](https://app.codecov.io/gh/apache/superset/pull/29912/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `53.57% <58.33%> (-0.23%)` | :arrow_down: |
| [python](https://app.codecov.io/gh/apache/superset/pull/29912/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `65.05% <58.33%> (+1.56%)` | :arrow_up: |
| [unit](https://app.codecov.io/gh/apache/superset/pull/29912/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `60.33% <40.27%> (+2.70%)` | :arrow_up: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
</details>
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/superset/pull/29912?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(GAQ): Add Redis Sentinel Support for Global Async Queries [superset]
Posted by "nsivarajan (via GitHub)" <gi...@apache.org>.
nsivarajan commented on PR #29912:
URL: https://github.com/apache/superset/pull/29912#issuecomment-2284151332
HI @villebro , Could you please review this PR? It was created based on the idea discussed https://github.com/apache/superset/discussions/28839
However, I recently noticed that there is a new SIP in the design phase to modernize the framework around the same topic: [#29839](https://github.com/apache/superset/issues/29839) . I'm not sure if this PR will still add value before the new proposal is rolled out, but I would appreciate your feedback.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(GAQ): Add Redis Sentinel Support for Global Async Queries [superset]
Posted by "nsivarajan (via GitHub)" <gi...@apache.org>.
nsivarajan commented on PR #29912:
URL: https://github.com/apache/superset/pull/29912#issuecomment-2305400022
Hi @villebro,
I wanted to follow up on this PR. I understand you're busy, but I was hoping to get your review when you have a moment. Please let me know if there's any additional information I can provide or if there are any specific areas you'd like me to address.
Thanks again for your time and feedback!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(GAQ): Add Redis Sentinel Support for Global Async Queries [superset]
Posted by "nsivarajan (via GitHub)" <gi...@apache.org>.
nsivarajan commented on code in PR #29912:
URL: https://github.com/apache/superset/pull/29912#discussion_r1734680325
##########
superset/async_events/async_query_manager.py:
##########
@@ -230,14 +257,35 @@ def submit_chart_data_job(
def read_events(
self, channel: str, last_id: Optional[str]
) -> list[Optional[dict[str, Any]]]:
+ if not self._cache:
+ raise CacheBackendNotInitialized("Cache backend not initialized")
+
stream_name = f"{self._stream_prefix}{channel}"
start_id = increment_id(last_id) if last_id else "-"
- results = self._redis.xrange(stream_name, start_id, "+", self.MAX_EVENT_COUNT)
+ results = self._cache.xrange(stream_name, start_id, "+", self.MAX_EVENT_COUNT)
+ # Decode bytes to strings, decode_responses is not supported at RedisCache and RedisSentinelCache
+ if isinstance(self._cache, (RedisSentinelCacheBackend, RedisCacheBackend)):
+ decoded_results = [
+ (
+ event_id.decode("utf-8"),
+ {
+ key.decode("utf-8"): value.decode("utf-8")
+ for key, value in event_data.items()
+ },
+ )
+ for event_id, event_data in results
+ ]
+ return (
+ [] if not decoded_results else list(map(parse_event, decoded_results))
+ )
Review Comment:
RedisCache and RedisSentinelCache typically don't support built-in decoding like decode_responses. They return data in bytes, which needs to be manually decoded into the desired formats (e.g., strings).
https://github.com/pallets-eco/flask-caching/blob/master/src/flask_caching/backends/rediscache.py#L155
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(GAQ): Add Redis Sentinel Support for Global Async Queries [superset]
Posted by "nsivarajan (via GitHub)" <gi...@apache.org>.
nsivarajan commented on code in PR #29912:
URL: https://github.com/apache/superset/pull/29912#discussion_r1734685526
##########
tests/unit_tests/async_events/async_query_manager_tests.py:
##########
@@ -26,17 +27,27 @@
AsyncQueryManager,
AsyncQueryTokenException,
)
+from superset.async_events.cache_backend import (
+ RedisCacheBackend,
+ RedisSentinelCacheBackend,
+)
JWT_TOKEN_SECRET = "some_secret"
JWT_TOKEN_COOKIE_NAME = "superset_async_jwt"
+# Define the cache backends once as mocks
+cache_backends = {
+ "RedisCacheBackend": mock.Mock(spec=RedisCacheBackend),
+ "RedisSentinelCacheBackend": mock.Mock(spec=RedisSentinelCacheBackend),
+ "redis.Redis": mock.Mock(spec=redis.Redis),
+}
+
Review Comment:
Yes, that's a great suggestion. I'll update the code with `pytest.mark.parametrize` and commit the changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(GAQ): Add Redis Sentinel Support for Global Async Queries [superset]
Posted by "villebro (via GitHub)" <gi...@apache.org>.
villebro merged PR #29912:
URL: https://github.com/apache/superset/pull/29912
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(GAQ): Add Redis Sentinel Support for Global Async Queries [superset]
Posted by "nsivarajan (via GitHub)" <gi...@apache.org>.
nsivarajan commented on code in PR #29912:
URL: https://github.com/apache/superset/pull/29912#discussion_r1738035374
##########
tests/unit_tests/async_events/async_query_manager_tests.py:
##########
@@ -17,26 +17,37 @@
from unittest import mock
from unittest.mock import ANY, Mock
+import redis
from flask import g
from jwt import encode
-from pytest import fixture, raises
+from pytest import fixture, mark, raises
from superset import security_manager
from superset.async_events.async_query_manager import (
AsyncQueryManager,
AsyncQueryTokenException,
)
+from superset.async_events.cache_backend import (
+ RedisCacheBackend,
+ RedisSentinelCacheBackend,
+)
JWT_TOKEN_SECRET = "some_secret"
JWT_TOKEN_COOKIE_NAME = "superset_async_jwt"
+# Define the cache backends once as mocks
+# cache_backends = {
+# "RedisCacheBackend": mock.Mock(spec=RedisCacheBackend),
+# "RedisSentinelCacheBackend": mock.Mock(spec=RedisSentinelCacheBackend),
+# "redis.Redis": mock.Mock(spec=redis.Redis),
+# }
+
Review Comment:
Thanks, I'll get this removed.
##########
tests/integration_tests/tasks/async_queries_tests.py:
##########
@@ -37,18 +43,37 @@
)
from tests.integration_tests.test_app import app
+# Define the cache backends once
+cache_backends = {
+ "RedisCacheBackend": mock.Mock(spec=RedisCacheBackend),
+ "RedisSentinelCacheBackend": mock.Mock(spec=RedisSentinelCacheBackend),
+ "redis.Redis": mock.Mock(spec=redis.Redis),
+}
+
Review Comment:
Thanks, I'll get this removed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(GAQ): Add Redis Sentinel Support for Global Async Queries [superset]
Posted by "villebro (via GitHub)" <gi...@apache.org>.
villebro commented on code in PR #29912:
URL: https://github.com/apache/superset/pull/29912#discussion_r1733477617
##########
superset/async_events/async_query_manager.py:
##########
@@ -230,14 +257,35 @@ def submit_chart_data_job(
def read_events(
self, channel: str, last_id: Optional[str]
) -> list[Optional[dict[str, Any]]]:
+ if not self._cache:
+ raise CacheBackendNotInitialized("Cache backend not initialized")
+
stream_name = f"{self._stream_prefix}{channel}"
start_id = increment_id(last_id) if last_id else "-"
- results = self._redis.xrange(stream_name, start_id, "+", self.MAX_EVENT_COUNT)
+ results = self._cache.xrange(stream_name, start_id, "+", self.MAX_EVENT_COUNT)
+ # Decode bytes to strings, decode_responses is not supported at RedisCache and RedisSentinelCache
+ if isinstance(self._cache, (RedisSentinelCacheBackend, RedisCacheBackend)):
+ decoded_results = [
+ (
+ event_id.decode("utf-8"),
+ {
+ key.decode("utf-8"): value.decode("utf-8")
+ for key, value in event_data.items()
+ },
+ )
+ for event_id, event_data in results
+ ]
+ return (
+ [] if not decoded_results else list(map(parse_event, decoded_results))
+ )
Review Comment:
I'm not sure this part was mentioned in the description. Can you elaborate on the need for this change?
##########
superset/async_events/async_query_manager.py:
##########
@@ -16,18 +16,27 @@
# under the License.
import logging
import uuid
-from typing import Any, Literal, Optional
+from typing import Any, Dict, Literal, Optional, Union
Review Comment:
nit: we can now use `dict` in type annotations as long as we do the following import at the beginning:
```python
from __future__ iimport annotations
```
##########
tests/unit_tests/async_events/async_query_manager_tests.py:
##########
@@ -26,17 +27,27 @@
AsyncQueryManager,
AsyncQueryTokenException,
)
+from superset.async_events.cache_backend import (
+ RedisCacheBackend,
+ RedisSentinelCacheBackend,
+)
JWT_TOKEN_SECRET = "some_secret"
JWT_TOKEN_COOKIE_NAME = "superset_async_jwt"
+# Define the cache backends once as mocks
+cache_backends = {
+ "RedisCacheBackend": mock.Mock(spec=RedisCacheBackend),
+ "RedisSentinelCacheBackend": mock.Mock(spec=RedisSentinelCacheBackend),
+ "redis.Redis": mock.Mock(spec=redis.Redis),
+}
+
Review Comment:
Could this loop be replaced by `pytest.mark.parametrize`? It feels more idiomatic (unless I'm missing some reason why it needs to be implemented like this..)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org