You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@superset.apache.org by "saha23412 (via GitHub)" <gi...@apache.org> on 2025/12/02 18:05:42 UTC
[D] How the WebSocket server interacts with Redis ? [superset]
GitHub user saha23412 created a discussion: How the WebSocket server interacts with Redis ?
I have a couple of questions about how WebSocket server works in conjunction with Redis.
1. How query parallelism, backpressure, TTL in Redis, key clearing works, how it can be configured using the superset-websocket module?
2. How to set up auto-closing/TTL keys in Redis, handling stale data, backpressure and payload size limitation?
3. What protocols and formats does it use, and how much can it be reused to implement front-end subscription to Redis
(WebSocket)?
4. What cookies/headers are required for authorization, how are subscriptions/channels handled?
5. What cookies/headers are required for authorization, how are subscriptions/channels handled?
6. What security settings can be used when interacting with superset-websocket and Redis?
GitHub link: https://github.com/apache/superset/discussions/36383
----
This is an automatically sent email for notifications@superset.apache.org.
To unsubscribe, please send an email to: notifications-unsubscribe@superset.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [D] How the WebSocket server interacts with Redis ? [superset]
Posted by "dosubot[bot] (via GitHub)" <gi...@apache.org>.
GitHub user dosubot[bot] edited a comment on the discussion: How the WebSocket server interacts with Redis ?
<!-- Answer -->
The superset-websocket server interacts with Redis by reading async event data from Redis Streams and forwarding it to browser clients over WebSocket. Events are published to both a global stream and user-specific streams by the Superset Flask app. The WebSocket server does not use Redis consumer groups; instead, each instance reads the full stream so all clients get all relevant events ([source](https://github.com/apache/superset/blob/829e4d92d91ceae4b43b1ed3b063ffe45377799c/superset-websocket/README.md#L19-L115)).
Query parallelism and backpressure are managed by configuring how many messages are read from Redis in parallel (`redisStreamReadCount`) and the blocking time for reads (`redisStreamReadBlockMs`). TTL and key clearing for Redis Streams are handled by limiting the number of events in each stream (`GLOBAL_ASYNC_QUERIES_REDIS_STREAM_LIMIT` and `GLOBAL_ASYNC_QUERIES_REDIS_STREAM_LIMIT_FIREHOSE`), not by Redis key expiration ([source](https://github.com/apache/superset/blob/829e4d92d91ceae4b43b1ed3b063ffe45377799c/superset/async_events/async_query_manager.py#L66-L308)). Stale data and inactive connections are cleaned up by periodic garbage collection (`gcChannelsIntervalMs`) and ping/pong health checks (`pingSocketsIntervalMs`, `socketResponseTimeoutMs`) ([source](https://github.com/apache/superset/blob/829e4d92d91ceae4b43b1ed3b063ffe45377799c/superset-websocket/src/index.ts#L23-L463)).
Payloads sent over WebSocket are JSON-encoded event data. There is no explicit payload size limit enforced in the current implementation. The protocol is a custom JSON-over-WebSocket scheme, so reusing it for direct front-end subscriptions to Redis would require implementing the same JWT-based authentication and channel management as the superset-websocket server ([source](https://github.com/apache/superset/blob/829e4d92d91ceae4b43b1ed3b063ffe45377799c/superset-websocket/README.md#L19-L115)).
Authorization uses JWT tokens stored in cookies (`jwtCookieName`), which must match between the Flask app and the WebSocket server. The JWT contains a channel ID for routing events. Subscriptions/channels are managed in-memory by tracking sockets per channel; all sockets for a channel receive all events for that channel ([source](https://github.com/apache/superset/blob/829e4d92d91ceae4b43b1ed3b063ffe45377799c/superset-websocket/src/index.ts#L23-L463)).
Security settings include JWT verification (with configurable secret and algorithms), optional Redis TLS/SSL, and Content Security Policy (CSP) configuration to restrict WebSocket origins. The WebSocket server must run on the same hostname as the Flask app for cookies to be shared for authentication ([source](https://github.com/apache/superset/discussions/34703)).
Let me know if you need config examples or details on a specific part of the setup.
<!-- Dosu Comment Footer -->
*To reply, just mention [@dosu](https://go.dosu.dev/dosubot?utm_source=github&utm_medium=bot-comment&utm_campaign=github-comment-footer-20260415&utm_content=reply-with-mention&utm_term=apache%2Fsuperset).*
---
Share context across your team and agents. Try [Dosu](https://dosu.dev?utm_source=github&utm_medium=bot-comment&utm_campaign=github-comment-footer-20260415&utm_content=knowledge-infrastructure-tagline&utm_term=apache%2Fsuperset).
[](https://app.dosu.dev/response-feedback?utm_source=github&utm_medium=bot-comment&utm_campaign=github-comment-footer-20260415&utm_content=knowledge-infrastructure-feedback&utm_term=apache%2Fsuperset&message_id=ca3bc12f-4b1d-46dd-8d67-e6cb043ede18) [](https://github.dosu.com/apache/superset?utm_source=github&utm_medium=bot-comment&utm_campaign=github-comment-footer-20260415&utm_content=knowledge-infrastructure-learn-repo&utm_term=apache%2Fsuperset) [](https://app.dosu.dev/signup?referrer=openSource&source=github-footer&utm_source=github&utm_medium=bot-comment&utm_campaign=github-comment-footer-20260415&utm_content=knowledge-infrastructure-add-team&utm_term=apache%2Fsuperset)
GitHub link: https://github.com/apache/superset/discussions/36383#discussioncomment-15138664
----
This is an automatically sent email for notifications@superset.apache.org.
To unsubscribe, please send an email to: notifications-unsubscribe@superset.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [D] How the WebSocket server interacts with Redis ? [superset]
Posted by "saha23412 (via GitHub)" <gi...@apache.org>.
GitHub user saha23412 closed the discussion with a comment: How the WebSocket server interacts with Redis ?
What is "db": 2 in GLOBAL_ASYNC_QUERIES_REDIS_CONFIG
GitHub link: https://github.com/apache/superset/discussions/36383#discussioncomment-15207906
----
This is an automatically sent email for notifications@superset.apache.org.
To unsubscribe, please send an email to: notifications-unsubscribe@superset.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [D] How the WebSocket server interacts with Redis ? [superset]
Posted by "saha23412 (via GitHub)" <gi...@apache.org>.
GitHub user saha23412 closed a discussion: How the WebSocket server interacts with Redis ?
I have a couple of questions about how WebSocket server works in conjunction with Redis.
1. How query parallelism, backpressure, TTL in Redis, key clearing works, how it can be configured using the superset-websocket module?
2. How to set up auto-closing/TTL keys in Redis, handling stale data, backpressure and payload size limitation?
3. What protocols and formats does it use, and how much can it be reused to implement front-end subscription to Redis
(WebSocket)?
4. What cookies/headers are required for authorization, how are subscriptions/channels handled?
5. What cookies/headers are required for authorization, how are subscriptions/channels handled?
6. What security settings can be used when interacting with superset-websocket and Redis?
GitHub link: https://github.com/apache/superset/discussions/36383
----
This is an automatically sent email for notifications@superset.apache.org.
To unsubscribe, please send an email to: notifications-unsubscribe@superset.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [D] How the WebSocket server interacts with Redis ? [superset]
Posted by "dosubot[bot] (via GitHub)" <gi...@apache.org>.
GitHub user dosubot[bot] added a comment to the discussion: How the WebSocket server interacts with Redis ?
<!-- Answer -->
The superset-websocket server interacts with Redis by reading async event data from Redis Streams and forwarding it to browser clients over WebSocket. Events are published to both a global stream and user-specific streams by the Superset Flask app. The WebSocket server does not use Redis consumer groups; instead, each instance reads the full stream so all clients get all relevant events ([source](https://github.com/apache/superset/blob/829e4d92d91ceae4b43b1ed3b063ffe45377799c/superset-websocket/README.md#L19-L115)).
Query parallelism and backpressure are managed by configuring how many messages are read from Redis in parallel (`redisStreamReadCount`) and the blocking time for reads (`redisStreamReadBlockMs`). TTL and key clearing for Redis Streams are handled by limiting the number of events in each stream (`GLOBAL_ASYNC_QUERIES_REDIS_STREAM_LIMIT` and `GLOBAL_ASYNC_QUERIES_REDIS_STREAM_LIMIT_FIREHOSE`), not by Redis key expiration ([source](https://github.com/apache/superset/blob/829e4d92d91ceae4b43b1ed3b063ffe45377799c/superset/async_events/async_query_manager.py#L66-L308)). Stale data and inactive connections are cleaned up by periodic garbage collection (`gcChannelsIntervalMs`) and ping/pong health checks (`pingSocketsIntervalMs`, `socketResponseTimeoutMs`) ([source](https://github.com/apache/superset/blob/829e4d92d91ceae4b43b1ed3b063ffe45377799c/superset-websocket/src/index.ts#L23-L463)).
Payloads sent over WebSocket are JSON-encoded event data. There is no explicit payload size limit enforced in the current implementation. The protocol is a custom JSON-over-WebSocket scheme, so reusing it for direct front-end subscriptions to Redis would require implementing the same JWT-based authentication and channel management as the superset-websocket server ([source](https://github.com/apache/superset/blob/829e4d92d91ceae4b43b1ed3b063ffe45377799c/superset-websocket/README.md#L19-L115)).
Authorization uses JWT tokens stored in cookies (`jwtCookieName`), which must match between the Flask app and the WebSocket server. The JWT contains a channel ID for routing events. Subscriptions/channels are managed in-memory by tracking sockets per channel; all sockets for a channel receive all events for that channel ([source](https://github.com/apache/superset/blob/829e4d92d91ceae4b43b1ed3b063ffe45377799c/superset-websocket/src/index.ts#L23-L463)).
Security settings include JWT verification (with configurable secret and algorithms), optional Redis TLS/SSL, and Content Security Policy (CSP) configuration to restrict WebSocket origins. The WebSocket server must run on the same hostname as the Flask app for cookies to be shared for authentication ([source](https://github.com/apache/superset/discussions/34703)).
Let me know if you need config examples or details on a specific part of the setup.
<!-- Dosu Comment Footer -->
*To reply, just mention [@dosu](https://go.dosu.dev/dosubot).*
---
<sup>How did I do? [Good](https://app.dosu.dev/response-feedback/ca3bc12f-4b1d-46dd-8d67-e6cb043ede18?feedback_type=great_response) | [Irrelevant](https://app.dosu.dev/response-feedback/ca3bc12f-4b1d-46dd-8d67-e6cb043ede18?feedback_type=irrelevant_answer) | [Incorrect](https://app.dosu.dev/response-feedback/ca3bc12f-4b1d-46dd-8d67-e6cb043ede18?feedback_type=incorrect_sources) | [Verbose](https://app.dosu.dev/response-feedback/ca3bc12f-4b1d-46dd-8d67-e6cb043ede18?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/ca3bc12f-4b1d-46dd-8d67-e6cb043ede18?feedback_type=hallucination) | [Report 🐛](https://app.dosu.dev/response-feedback/ca3bc12f-4b1d-46dd-8d67-e6cb043ede18?feedback_type=bug_report) | [Other](https://app.dosu.dev/response-feedback/ca3bc12f-4b1d-46dd-8d67-e6cb043ede18?feedback_type=other)</sup>
[](https://app.dosu.dev/a28d3c7e-a9d3-459e-9fb6-3a6f9ff4f357/ask?utm_source=github) [](https://cursor.com/link/prompt?text=The%20superset-websocket%20server%20interacts%20with%20Redis%20by%20reading%20async%20event%20data%20from%20Redis%20Streams%20and%20forwarding%20it%20to%20browser%20clients%20over%20WebSocket.%20Events%20are%20published%20to%20both%20a%20global%20stream%20and%20user-specific%20streams%20by%20the%20Superset%20Flask%20app.%20The%20WebSocket%20server%20does%20not%20use%20Redis%20consumer%20groups%3B%20instead%2C%20each%20instance%20reads%20the%20full%20stream%20so%20all%20clients%20get%20all%20relevant%20events%C2%A0%28%5Bsource%5D%28https%3A//github.com/apache/superset/blob/829e4d92d91ceae4b43b1ed3b063ffe45377799c/superset-websocket/README.md%23L19-L115%29%29.%0A%0AQuery%20parallelism%20and%20backpressure%20are%20managed%20by%20configuring%20how%20many%
20messages%20are%20read%20from%20Redis%20in%20parallel%20%28%60redisStreamReadCount%60%29%20and%20the%20blocking%20time%20for%20reads%20%28%60redisStreamReadBlockMs%60%29.%20TTL%20and%20key%20clearing%20for%20Redis%20Streams%20are%20handled%20by%20limiting%20the%20number%20of%20events%20in%20each%20stream%20%28%60GLOBAL_ASYNC_QUERIES_REDIS_STREAM_LIMIT%60%20and%20%60GLOBAL_ASYNC_QUERIES_REDIS_STREAM_LIMIT_FIREHOSE%60%29%2C%20not%20by%20Redis%20key%20expiration%C2%A0%28%5Bsource%5D%28https%3A//github.com/apache/superset/blob/829e4d92d91ceae4b43b1ed3b063ffe45377799c/superset/async_events/async_query_manager.py%23L66-L308%29%29.%20Stale%20data%20and%20inactive%20connections%20are%20cleaned%20up%20by%20periodic%20garbage%20collection%20%28%60gcChannelsIntervalMs%60%29%20and%20ping/pong%20health%20checks%20%28%60pingSocketsIntervalMs%60%2C%20%60socketResponseTimeoutMs%60%29%C2%A0%28%5Bsource%5D%28https%3A//github.com/apache/superset/blob/829e4d92d91ceae4b43b1ed3b063ffe45377799c/superset-
websocket/src/index.ts%23L23-L463%29%29.%0A%0APayloads%20sent%20over%20WebSocket%20are%20JSON-encoded%20event%20data.%20There%20is%20no%20explicit%20payload%20size%20limit%20enforced%20in%20the%20current%20implementation.%20The%20protocol%20is%20a%20custom%20JSON-over-WebSocket%20scheme%2C%20so%20reusing%20it%20for%20direct%20front-end%20subscriptions%20to%20Redis%20would%20require%20implementing%20the%20same%20JWT-based%20authentication%20and%20channel%20management%20as%20the%20superset-websocket%20server%C2%A0%28%5Bsource%5D%28https%3A//github.com/apache/superset/blob/829e4d92d91ceae4b43b1ed3b063ffe45377799c/superset-websocket/README.md%23L19-L115%29%29.%0A%0AAuthorization%20uses%20JWT%20tokens%20stored%20in%20cookies%20%28%60jwtCookieName%60%29%2C%20which%20must%20match%20between%20the%20Flask%20app%20and%20the%20WebSocket%20server.%20The%20JWT%20contains%20a%20channel%20ID%20for%20routing%20events.%20Subscriptions/channels%20are%20managed%20in-memory%20by%20tracking%20sockets%20
per%20channel%3B%20all%20sockets%20for%20a%20channel%20receive%20all%20events%20for%20that%20channel%C2%A0%28%5Bsource%5D%28https%3A//github.com/apache/superset/blob/829e4d92d91ceae4b43b1ed3b063ffe45377799c/superset-websocket/src/index.ts%23L23-L463%29%29.%0A%0ASecurity%20settings%20include%20JWT%20verification%20%28with%20configurable%20secret%20and%20algorithms%29%2C%20optional%20Redis%20TLS/SSL%2C%20and%20Content%20Security%20Policy%20%28CSP%29%20configuration%20to%20restrict%20WebSocket%20origins.%20The%20WebSocket%20server%20must%20run%20on%20the%20same%20hostname%20as%20the%20Flask%20app%20for%20cookies%20to%20be%20shared%20for%20authentication%C2%A0%28%5Bsource%5D%28https%3A//github.com/apache/superset/discussions/34703%29%29.%20%0A%0ALet%20me%20know%20if%20you%20need%20config%20examples%20or%20details%20on%20a%20specific%20part%20of%20the%20setup.) [](https://go.dosu.dev/discord
-bot) [](https://twitter.com/intent/tweet?text=%40dosu_ai%20helped%20me%20solve%20this%20issue!&url=https%3A//github.com/apache/superset/discussions/36383)
GitHub link: https://github.com/apache/superset/discussions/36383#discussioncomment-15138664
----
This is an automatically sent email for notifications@superset.apache.org.
To unsubscribe, please send an email to: notifications-unsubscribe@superset.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [D] How the WebSocket server interacts with Redis ? [superset]
Posted by "dosubot[bot] (via GitHub)" <gi...@apache.org>.
GitHub user dosubot[bot] edited a comment on the discussion: How the WebSocket server interacts with Redis ?
<!-- Answer -->
The frontend connects to the superset-websocket server using a WebSocket client, authenticating via a JWT token stored in a cookie (set by the Flask backend). The WebSocket URL is typically configured via `GLOBAL_ASYNC_QUERIES_WEBSOCKET_URL`, and the frontend appends a `last_id` query parameter to support reconnection and missed event delivery. Incoming messages are JSON objects containing async event data, including any custom fields you add in the backend payload (like `request_id`, `type`, `axis`, etc.) [example and details](https://github.com/apache/superset/blob/master/superset-websocket/README.md).
Superset’s frontend manages listeners for async events keyed by `job_id`, so components can react to updates for specific jobs. If a listener isn’t registered when an event arrives, the event is retried a few times before logging a warning. If the WebSocket connection fails, the frontend falls back to polling via a REST API [see asyncEvent.ts](https://github.com/apache/superset/blob/master/superset-frontend/src/middleware/asyncEvent.ts).
**Example: Minimal WebSocket Client (from Superset frontend)**
```typescript
// Simplified from superset-frontend/src/middleware/asyncEvent.ts
let ws: WebSocket | null = null;
let lastReceivedEventId: string | null = null;
function wsConnect() {
let url = config.GLOBAL_ASYNC_QUERIES_WEBSOCKET_URL;
if (lastReceivedEventId) url += `?last_id=${lastReceivedEventId}`;
ws = new WebSocket(url);
ws.addEventListener('open', () => {
// Connected
});
ws.addEventListener('message', event => {
const asyncEvent = JSON.parse(event.data);
// Handle event, e.g. dispatch to listeners keyed by asyncEvent.job_id
});
ws.addEventListener('close', () => {
// Optionally reconnect or fall back to polling
});
}
```
**Example: Utility Client App (from superset-websocket utils)**
```javascript
// Simplified from superset-websocket/utils/client-ws-app/public/javascripts/app.js
const socket = new WebSocket('ws://127.0.0.1:8080?last_id=...');
socket.addEventListener('open', () => {
socket.send('Hello Server!');
});
socket.addEventListener('message', event => {
const data = JSON.parse(event.data);
// Process async event data
});
```
[See full example](https://github.com/apache/superset/blob/master/superset-websocket/utils/client-ws-app/public/javascripts/app.js)
**Listener Management Pattern**
Superset’s frontend keeps a dictionary of listeners by `job_id`:
```typescript
const listenersByJobId: { [jobId: string]: (event: AsyncEvent) => void } = {};
function addListener(jobId: string, fn: (event: AsyncEvent) => void) {
listenersByJobId[jobId] = fn;
}
function removeListener(jobId: string) {
delete listenersByJobId[jobId];
}
```
When a message arrives, it’s dispatched to the appropriate listener based on `job_id` [see asyncEvent.ts](https://github.com/apache/superset/blob/master/superset-frontend/src/middleware/asyncEvent.ts).
**Best Practices**
- Make sure your backend includes all required custom fields in the event payload.
- The frontend expects JSON messages and can access any custom fields you add.
- Use the listener pattern to handle updates for specific jobs.
- Handle reconnection and fallback to polling for reliability.
Let me know if you want a more detailed React hook or integration example!
<!-- Dosu Comment Footer -->
*To reply, just mention [@dosu](https://go.dosu.dev/dosubot?utm_source=github&utm_medium=bot-comment&utm_campaign=github-comment-footer-20260415&utm_content=reply-with-mention&utm_term=apache%2Fsuperset).*
---
Docs are dead. Just use [Dosu](https://dosu.dev?utm_source=github&utm_medium=bot-comment&utm_campaign=github-comment-footer-20260415&utm_content=static-docs-tagline&utm_term=apache%2Fsuperset).
[](https://app.dosu.dev/response-feedback?utm_source=github&utm_medium=bot-comment&utm_campaign=github-comment-footer-20260415&utm_content=static-docs-feedback&utm_term=apache%2Fsuperset&message_id=4b93ade1-562b-4c45-a0d3-56c391040c99) [](https://github.dosu.com/apache/superset?utm_source=github&utm_medium=bot-comment&utm_campaign=github-comment-footer-20260415&utm_content=static-docs-ask-repo&utm_term=apache%2Fsuperset) [](https://app.dosu.dev/signup?referrer=openSource&source=github-footer&utm_source=github&utm_medium=bot-comment&utm_campaign=github-comment-footer-20260415&utm_content=static-docs-share-team&utm_term=apache%2Fsuperset)
GitHub link: https://github.com/apache/superset/discussions/36383#discussioncomment-15148250
----
This is an automatically sent email for notifications@superset.apache.org.
To unsubscribe, please send an email to: notifications-unsubscribe@superset.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [D] How the WebSocket server interacts with Redis ? [superset]
Posted by "dosubot[bot] (via GitHub)" <gi...@apache.org>.
GitHub user dosubot[bot] added a comment to the discussion: How the WebSocket server interacts with Redis ?
<!-- Answer -->
Here’s how you can configure superset-websocket and Superset for async event messaging, and implement a custom message format with fields like request_id, type, axis, level, offset, limit, payload, metrics_meta, and error:
**Configuration Examples**
In `superset_config.py` (Superset backend):
```python
GLOBAL_ASYNC_QUERIES = True
GLOBAL_ASYNC_QUERIES_TRANSPORT = "ws"
GLOBAL_ASYNC_QUERIES_WEBSOCKET_URL = "ws://localhost:8080/"
GLOBAL_ASYNC_QUERIES_REDIS_CONFIG = {
"host": "127.0.0.1",
"port": 6379,
"db": 2,
"ssl": False
}
GLOBAL_ASYNC_QUERIES_JWT_SECRET = "your-secret-key"
GLOBAL_ASYNC_QUERIES_JWT_COOKIE_NAME = "async-token"
GLOBAL_ASYNC_QUERIES_REDIS_STREAM_PREFIX = "async-events-"
```
In `superset-websocket/config.json`:
```json
{
"port": 8080,
"redis": {
"port": 6379,
"host": "127.0.0.1",
"db": 2,
"ssl": false
},
"redisStreamPrefix": "async-events-",
"jwtSecret": "your-secret-key",
"jwtCookieName": "async-token"
}
```
Make sure the JWT secret, cookie name, Redis details, and stream prefix match in both configs. The hostname must match how you access Superset in your browser for cookies to work. Both the Flask server and superset-websocket server must be running simultaneously for async features to work [source](https://github.com/apache/superset/issues/36069).
**Custom Message Format Implementation**
Superset’s async event messages are built in the backend (see `async_query_manager.py`). The event data dict is constructed using `build_job_metadata` and updated via `update_job`. You can add custom fields (like request_id, type, axis, etc.) by passing them as keyword arguments to `update_job`. These fields will be included in the JSON payload written to Redis and forwarded by superset-websocket to clients [source](https://github.com/apache/superset/blob/829e4d92d91ceae4b43b1ed3b063ffe45377799c/superset/async_events/async_query_manager.py#L53-L308).
**Example: Custom Event Payload**
Here’s what a custom event payload might look like:
```json
{
"id": "1615426152415-0", // Redis stream ID
"channel_id": "abc123",
"job_id": "job-uuid",
"user_id": "1",
"request_id": "req-uuid",
"type": "query",
"axis": "x",
"level": "detail",
"offset": 0,
"limit": 100,
"payload": { "data": [...] },
"metrics_meta": { "duration_ms": 123 },
"error": null,
"status": "done",
"result_url": "/api/v1/chart/data/job-uuid"
}
```
To implement this, update your async query Celery tasks (e.g., in `async_queries.py`) to pass these fields to `async_query_manager.update_job`. For example:
```python
async_query_manager.update_job(
job_metadata,
async_query_manager.STATUS_DONE,
request_id="req-uuid",
type="query",
axis="x",
level="detail",
offset=0,
limit=100,
payload=payload,
metrics_meta={"duration_ms": 123},
error=None,
result_url=result_url,
)
```
Any fields you add here will be included in the event data sent to Redis and delivered to the frontend via WebSocket [source](https://github.com/apache/superset/blob/829e4d92d91ceae4b43b1ed3b063ffe45377799c/superset/tasks/async_queries.py#L82-L166).
**Best Practices & Caveats**
- There’s no explicit config option for message format in superset-websocket; you control the payload in the backend code.
- Make sure any consumers (frontend, other services) expect and handle your custom fields.
- The error field is typically a list of error dicts, populated if exceptions occur during async query execution.
- There’s no enforced payload size limit in superset-websocket, but you can set limits in the backend (e.g., `SQLLAB_PAYLOAD_MAX_MB` for SQL Lab) [source](https://github.com/apache/superset/pull/30522).
Let me know if you want code snippets for a specific async query type or more details on frontend handling!
<!-- Dosu Comment Footer -->
*To reply, just mention [@dosu](https://go.dosu.dev/dosubot).*
---
<sup>How did I do? [Good](https://app.dosu.dev/response-feedback/d81218be-f731-4857-8fd4-b69492a5d8d4?feedback_type=great_response) | [Irrelevant](https://app.dosu.dev/response-feedback/d81218be-f731-4857-8fd4-b69492a5d8d4?feedback_type=irrelevant_answer) | [Incorrect](https://app.dosu.dev/response-feedback/d81218be-f731-4857-8fd4-b69492a5d8d4?feedback_type=incorrect_sources) | [Verbose](https://app.dosu.dev/response-feedback/d81218be-f731-4857-8fd4-b69492a5d8d4?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/d81218be-f731-4857-8fd4-b69492a5d8d4?feedback_type=hallucination) | [Report 🐛](https://app.dosu.dev/response-feedback/d81218be-f731-4857-8fd4-b69492a5d8d4?feedback_type=bug_report) | [Other](https://app.dosu.dev/response-feedback/d81218be-f731-4857-8fd4-b69492a5d8d4?feedback_type=other)</sup>
[](https://app.dosu.dev/a28d3c7e-a9d3-459e-9fb6-3a6f9ff4f357/ask?utm_source=github) [](https://cursor.com/link/prompt?text=Here%E2%80%99s%20how%20you%20can%20configure%20superset-websocket%20and%20Superset%20for%20async%20event%20messaging%2C%20and%20implement%20a%20custom%20message%20format%20with%20fields%20like%20request_id%2C%20type%2C%20axis%2C%20level%2C%20offset%2C%20limit%2C%20payload%2C%20metrics_meta%2C%20and%20error%3A%0A%0A%2A%2AConfiguration%20Examples%2A%2A%0A%0AIn%20%60superset_config.py%60%20%28Superset%20backend%29%3A%0A%0A%60%60%60python%0AGLOBAL_ASYNC_QUERIES%20%3D%20True%0AGLOBAL_ASYNC_QUERIES_TRANSPORT%20%3D%20%22ws%22%0AGLOBAL_ASYNC_QUERIES_WEBSOCKET_URL%20%3D%20%22ws%3A//localhost%3A8080/%22%0AGLOBAL_ASYNC_QUERIES_REDIS_CONFIG%20%3D%20%7B%0A%20%20%20%20%22host%22%3A%20%22127.0.0.1%22%2C%0A%20%20%20%20%22port%22%3A%206379%2C%0A%20%20%20%20%22db%22%3A
%202%2C%0A%20%20%20%20%22ssl%22%3A%20False%0A%7D%0AGLOBAL_ASYNC_QUERIES_JWT_SECRET%20%3D%20%22your-secret-key%22%0AGLOBAL_ASYNC_QUERIES_JWT_COOKIE_NAME%20%3D%20%22async-token%22%0AGLOBAL_ASYNC_QUERIES_REDIS_STREAM_PREFIX%20%3D%20%22async-events-%22%0A%60%60%60%0AIn%20%60superset-websocket/config.json%60%3A%0A%0A%60%60%60json%0A%7B%0A%20%20%22port%22%3A%208080%2C%0A%20%20%22redis%22%3A%20%7B%0A%20%20%20%20%22port%22%3A%206379%2C%0A%20%20%20%20%22host%22%3A%20%22127.0.0.1%22%2C%0A%20%20%20%20%22db%22%3A%202%2C%0A%20%20%20%20%22ssl%22%3A%20false%0A%20%20%7D%2C%0A%20%20%22redisStreamPrefix%22%3A%20%22async-events-%22%2C%0A%20%20%22jwtSecret%22%3A%20%22your-secret-key%22%2C%0A%20%20%22jwtCookieName%22%3A%20%22async-token%22%0A%7D%0A%60%60%60%0AMake%20sure%20the%20JWT%20secret%2C%20cookie%20name%2C%20Redis%20details%2C%20and%20stream%20prefix%20match%20in%20both%20configs.%20The%20hostname%20must%20match%20how%20you%20access%20Superset%20in%20your%20browser%20for%20cookies%20to%20work.%20
Both%20the%20Flask%20server%20and%20superset-websocket%20server%20must%20be%20running%20simultaneously%20for%20async%20features%20to%20work%20%5Bsource%5D%28https%3A//github.com/apache/superset/issues/36069%29.%0A%0A%2A%2ACustom%20Message%20Format%20Implementation%2A%2A%0A%0ASuperset%E2%80%99s%20async%20event%20messages%20are%20built%20in%20the%20backend%20%28see%20%60async_query_manager.py%60%29.%20The%20event%20data%20dict%20is%20constructed%20using%20%60build_job_metadata%60%20and%20updated%20via%20%60update_job%60.%20You%20can%20add%20custom%20fields%20%28like%20request_id%2C%20type%2C%20axis%2C%20etc.%29%20by%20passing%20them%20as%20keyword%20arguments%20to%20%60update_job%60.%20These%20fields%20will%20be%20included%20in%20the%20JSON%20payload%20written%20to%20Redis%20and%20forwarded%20by%20superset-websocket%20to%20clients%20%5Bsource%5D%28https%3A//github.com/apache/superset/blob/829e4d92d91ceae4b43b1ed3b063ffe45377799c/superset/async_events/async_query_manager.py%23L53-L308%
29.%0A%0A%2A%2AExample%3A%20Custom%20Event%20Payload%2A%2A%0A%0AHere%E2%80%99s%20what%20a%20custom%20event%20payload%20might%20look%20like%3A%0A%0A%60%60%60json%0A%7B%0A%20%20%22id%22%3A%20%221615426152415-0%22%2C%20%20%20%20%20%20%20%20%20%20%20//%20Redis%20stream%20ID%0A%20%20%22channel_id%22%3A%20%22abc123%22%2C%0A%20%20%22job_id%22%3A%20%22job-uuid%22%2C%0A%20%20%22user_id%22%3A%20%221%22%2C%0A%20%20%22request_id%22%3A%20%22req-uuid%22%2C%0A%20%20%22type%22%3A%20%22query%22%2C%0A%20%20%22axis%22%3A%20%22x%22%2C%0A%20%20%22level%22%3A%20%22detail%22%2C%0A%20%20%22offset%22%3A%200%2C%0A%20%20%22limit%22%3A%20100%2C%0A%20%20%22payload%22%3A%20%7B%20%22data%22%3A%20%5B...%5D%20%7D%2C%0A%20%20%22metrics_meta%22%3A%20%7B%20%22duration_ms%22%3A%20123%20%7D%2C%0A%20%20%22error%22%3A%20null%2C%0A%20%20%22status%22%3A%20%22done%22%2C%0A%20%20%22result_url%22%3A%20%22/api/v1/chart/data/job-uuid%22%0A%7D%0A%60%60%60%0A%0ATo%20implement%20this%2C%20update%20your%20async%20query%20Celery%20ta
sks%20%28e.g.%2C%20in%20%60async_queries.py%60%29%20to%20pass%20these%20fields%20to%20%60async_query_manager.update_job%60.%20For%20example%3A%0A%0A%60%60%60python%0Aasync_query_manager.update_job%28%0A%20%20%20%20job_metadata%2C%0A%20%20%20%20async_query_manager.STATUS_DONE%2C%0A%20%20%20%20request_id%3D%22req-uuid%22%2C%0A%20%20%20%20type%3D%22query%22%2C%0A%20%20%20%20axis%3D%22x%22%2C%0A%20%20%20%20level%3D%22detail%22%2C%0A%20%20%20%20offset%3D0%2C%0A%20%20%20%20limit%3D100%2C%0A%20%20%20%20payload%3Dpayload%2C%0A%20%20%20%20metrics_meta%3D%7B%22duration_ms%22%3A%20123%7D%2C%0A%20%20%20%20error%3DNone%2C%0A%20%20%20%20result_url%3Dresult_url%2C%0A%29%0A%60%60%60%0AAny%20fields%20you%20add%20here%20will%20be%20included%20in%20the%20event%20data%20sent%20to%20Redis%20and%20delivered%20to%20the%20frontend%20via%20WebSocket%20%5Bsource%5D%28https%3A//github.com/apache/superset/blob/829e4d92d91ceae4b43b1ed3b063ffe45377799c/superset/tasks/async_queries.py%23L82-L166%29.%0A%0A%2A%2ABe
st%20Practices%20%26%20Caveats%2A%2A%0A%0A-%20There%E2%80%99s%20no%20explicit%20config%20option%20for%20message%20format%20in%20superset-websocket%3B%20you%20control%20the%20payload%20in%20the%20backend%20code.%0A-%20Make%20sure%20any%20consumers%20%28frontend%2C%20other%20services%29%20expect%20and%20handle%20your%20custom%20fields.%0A-%20The%20error%20field%20is%20typically%20a%20list%20of%20error%20dicts%2C%20populated%20if%20exceptions%20occur%20during%20async%20query%20execution.%0A-%20There%E2%80%99s%20no%20enforced%20payload%20size%20limit%20in%20superset-websocket%2C%20but%20you%20can%20set%20limits%20in%20the%20backend%20%28e.g.%2C%20%60SQLLAB_PAYLOAD_MAX_MB%60%20for%20SQL%20Lab%29%20%5Bsource%5D%28https%3A//github.com/apache/superset/pull/30522%29.%0A%0ALet%20me%20know%20if%20you%20want%20code%20snippets%20for%20a%20specific%20async%20query%20type%20or%20more%20details%20on%20frontend%20handling%21) [](https://go.dosu.dev/discord-bot) [](https://twitter.com/intent/tweet?text=%40dosu_ai%20helped%20me%20solve%20this%20issue!&url=https%3A//github.com/apache/superset/discussions/36383)
GitHub link: https://github.com/apache/superset/discussions/36383#discussioncomment-15146911
----
This is an automatically sent email for notifications@superset.apache.org.
To unsubscribe, please send an email to: notifications-unsubscribe@superset.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [D] How the WebSocket server interacts with Redis ? [superset]
Posted by "dosubot[bot] (via GitHub)" <gi...@apache.org>.
GitHub user dosubot[bot] added a comment to the discussion: How the WebSocket server interacts with Redis ?
<!-- Answer -->
The frontend connects to the superset-websocket server using a WebSocket client, authenticating via a JWT token stored in a cookie (set by the Flask backend). The WebSocket URL is typically configured via `GLOBAL_ASYNC_QUERIES_WEBSOCKET_URL`, and the frontend appends a `last_id` query parameter to support reconnection and missed event delivery. Incoming messages are JSON objects containing async event data, including any custom fields you add in the backend payload (like `request_id`, `type`, `axis`, etc.) [example and details](https://github.com/apache/superset/blob/master/superset-websocket/README.md).
Superset’s frontend manages listeners for async events keyed by `job_id`, so components can react to updates for specific jobs. If a listener isn’t registered when an event arrives, the event is retried a few times before logging a warning. If the WebSocket connection fails, the frontend falls back to polling via a REST API [see asyncEvent.ts](https://github.com/apache/superset/blob/master/superset-frontend/src/middleware/asyncEvent.ts).
**Example: Minimal WebSocket Client (from Superset frontend)**
```typescript
// Simplified from superset-frontend/src/middleware/asyncEvent.ts
let ws: WebSocket | null = null;
let lastReceivedEventId: string | null = null;
function wsConnect() {
let url = config.GLOBAL_ASYNC_QUERIES_WEBSOCKET_URL;
if (lastReceivedEventId) url += `?last_id=${lastReceivedEventId}`;
ws = new WebSocket(url);
ws.addEventListener('open', () => {
// Connected
});
ws.addEventListener('message', event => {
const asyncEvent = JSON.parse(event.data);
// Handle event, e.g. dispatch to listeners keyed by asyncEvent.job_id
});
ws.addEventListener('close', () => {
// Optionally reconnect or fall back to polling
});
}
```
**Example: Utility Client App (from superset-websocket utils)**
```javascript
// Simplified from superset-websocket/utils/client-ws-app/public/javascripts/app.js
const socket = new WebSocket('ws://127.0.0.1:8080?last_id=...');
socket.addEventListener('open', () => {
socket.send('Hello Server!');
});
socket.addEventListener('message', event => {
const data = JSON.parse(event.data);
// Process async event data
});
```
[See full example](https://github.com/apache/superset/blob/master/superset-websocket/utils/client-ws-app/public/javascripts/app.js)
**Listener Management Pattern**
Superset’s frontend keeps a dictionary of listeners by `job_id`:
```typescript
const listenersByJobId: { [jobId: string]: (event: AsyncEvent) => void } = {};
function addListener(jobId: string, fn: (event: AsyncEvent) => void) {
listenersByJobId[jobId] = fn;
}
function removeListener(jobId: string) {
delete listenersByJobId[jobId];
}
```
When a message arrives, it’s dispatched to the appropriate listener based on `job_id` [see asyncEvent.ts](https://github.com/apache/superset/blob/master/superset-frontend/src/middleware/asyncEvent.ts).
**Best Practices**
- Make sure your backend includes all required custom fields in the event payload.
- The frontend expects JSON messages and can access any custom fields you add.
- Use the listener pattern to handle updates for specific jobs.
- Handle reconnection and fallback to polling for reliability.
Let me know if you want a more detailed React hook or integration example!
<!-- Dosu Comment Footer -->
*To reply, just mention [@dosu](https://go.dosu.dev/dosubot).*
---
<sup>How did I do? [Good](https://app.dosu.dev/response-feedback/4b93ade1-562b-4c45-a0d3-56c391040c99?feedback_type=great_response) | [Irrelevant](https://app.dosu.dev/response-feedback/4b93ade1-562b-4c45-a0d3-56c391040c99?feedback_type=irrelevant_answer) | [Incorrect](https://app.dosu.dev/response-feedback/4b93ade1-562b-4c45-a0d3-56c391040c99?feedback_type=incorrect_sources) | [Verbose](https://app.dosu.dev/response-feedback/4b93ade1-562b-4c45-a0d3-56c391040c99?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/4b93ade1-562b-4c45-a0d3-56c391040c99?feedback_type=hallucination) | [Report 🐛](https://app.dosu.dev/response-feedback/4b93ade1-562b-4c45-a0d3-56c391040c99?feedback_type=bug_report) | [Other](https://app.dosu.dev/response-feedback/4b93ade1-562b-4c45-a0d3-56c391040c99?feedback_type=other)</sup>
[](https://app.dosu.dev/a28d3c7e-a9d3-459e-9fb6-3a6f9ff4f357/ask?utm_source=github) [](https://cursor.com/link/prompt?text=The%20frontend%20connects%20to%20the%20superset-websocket%20server%20using%20a%20WebSocket%20client%2C%20authenticating%20via%20a%20JWT%20token%20stored%20in%20a%20cookie%20%28set%20by%20the%20Flask%20backend%29.%20The%20WebSocket%20URL%20is%20typically%20configured%20via%20%60GLOBAL_ASYNC_QUERIES_WEBSOCKET_URL%60%2C%20and%20the%20frontend%20appends%20a%20%60last_id%60%20query%20parameter%20to%20support%20reconnection%20and%20missed%20event%20delivery.%20Incoming%20messages%20are%20JSON%20objects%20containing%20async%20event%20data%2C%20including%20any%20custom%20fields%20you%20add%20in%20the%20backend%20payload%20%28like%20%60request_id%60%2C%20%60type%60%2C%20%60axis%60%2C%20etc.%29%C2%A0%5Bexample%20and%20details%5D%28https%3A//github.com/apache/su
perset/blob/master/superset-websocket/README.md%29.%0A%0ASuperset%E2%80%99s%20frontend%20manages%20listeners%20for%20async%20events%20keyed%20by%20%60job_id%60%2C%20so%20components%20can%20react%20to%20updates%20for%20specific%20jobs.%20If%20a%20listener%20isn%E2%80%99t%20registered%20when%20an%20event%20arrives%2C%20the%20event%20is%20retried%20a%20few%20times%20before%20logging%20a%20warning.%20If%20the%20WebSocket%20connection%20fails%2C%20the%20frontend%20falls%20back%20to%20polling%20via%20a%20REST%20API%C2%A0%5Bsee%20asyncEvent.ts%5D%28https%3A//github.com/apache/superset/blob/master/superset-frontend/src/middleware/asyncEvent.ts%29.%0A%0A%2A%2AExample%3A%20Minimal%20WebSocket%20Client%20%28from%20Superset%20frontend%29%2A%2A%0A%0A%60%60%60typescript%0A//%20Simplified%20from%20superset-frontend/src/middleware/asyncEvent.ts%0Alet%20ws%3A%20WebSocket%20%7C%20null%20%3D%20null%3B%0Alet%20lastReceivedEventId%3A%20string%20%7C%20null%20%3D%20null%3B%0A%0Afunction%20wsConnect%28%29%
20%7B%0A%20%20let%20url%20%3D%20config.GLOBAL_ASYNC_QUERIES_WEBSOCKET_URL%3B%0A%20%20if%20%28lastReceivedEventId%29%20url%20%2B%3D%20%60%3Flast_id%3D%24%7BlastReceivedEventId%7D%60%3B%0A%20%20ws%20%3D%20new%20WebSocket%28url%29%3B%0A%0A%20%20ws.addEventListener%28%27open%27%2C%20%28%29%20%3D%3E%20%7B%0A%20%20%20%20//%20Connected%0A%20%20%7D%29%3B%0A%0A%20%20ws.addEventListener%28%27message%27%2C%20event%20%3D%3E%20%7B%0A%20%20%20%20const%20asyncEvent%20%3D%20JSON.parse%28event.data%29%3B%0A%20%20%20%20//%20Handle%20event%2C%20e.g.%20dispatch%20to%20listeners%20keyed%20by%20asyncEvent.job_id%0A%20%20%7D%29%3B%0A%0A%20%20ws.addEventListener%28%27close%27%2C%20%28%29%20%3D%3E%20%7B%0A%20%20%20%20//%20Optionally%20reconnect%20or%20fall%20back%20to%20polling%0A%20%20%7D%29%3B%0A%7D%0A%60%60%60%0A%0A%2A%2AExample%3A%20Utility%20Client%20App%20%28from%20superset-websocket%20utils%29%2A%2A%0A%0A%60%60%60javascript%0A//%20Simplified%20from%20superset-websocket/utils/client-ws-app/public/java
scripts/app.js%0Aconst%20socket%20%3D%20new%20WebSocket%28%27ws%3A//127.0.0.1%3A8080%3Flast_id%3D...%27%29%3B%0Asocket.addEventListener%28%27open%27%2C%20%28%29%20%3D%3E%20%7B%0A%20%20socket.send%28%27Hello%20Server%21%27%29%3B%0A%7D%29%3B%0Asocket.addEventListener%28%27message%27%2C%20event%20%3D%3E%20%7B%0A%20%20const%20data%20%3D%20JSON.parse%28event.data%29%3B%0A%20%20//%20Process%20async%20event%20data%0A%7D%29%3B%0A%60%60%60%0A%5BSee%20full%20example%5D%28https%3A//github.com/apache/superset/blob/master/superset-websocket/utils/client-ws-app/public/javascripts/app.js%29%0A%0A%2A%2AListener%20Management%20Pattern%2A%2A%0A%0ASuperset%E2%80%99s%20frontend%20keeps%20a%20dictionary%20of%20listeners%20by%20%60job_id%60%3A%0A%0A%60%60%60typescript%0Aconst%20listenersByJobId%3A%20%7B%20%5BjobId%3A%20string%5D%3A%20%28event%3A%20AsyncEvent%29%20%3D%3E%20void%20%7D%20%3D%20%7B%7D%3B%0A%0Afunction%20addListener%28jobId%3A%20string%2C%20fn%3A%20%28event%3A%20AsyncEvent%29%20%3D%3E%20void%
29%20%7B%0A%20%20listenersByJobId%5BjobId%5D%20%3D%20fn%3B%0A%7D%0A%0Afunction%20removeListener%28jobId%3A%20string%29%20%7B%0A%20%20delete%20listenersByJobId%5BjobId%5D%3B%0A%7D%0A%60%60%60%0AWhen%20a%20message%20arrives%2C%20it%E2%80%99s%20dispatched%20to%20the%20appropriate%20listener%20based%20on%20%60job_id%60%C2%A0%5Bsee%20asyncEvent.ts%5D%28https%3A//github.com/apache/superset/blob/master/superset-frontend/src/middleware/asyncEvent.ts%29.%0A%0A%2A%2ABest%20Practices%2A%2A%0A%0A-%20Make%20sure%20your%20backend%20includes%20all%20required%20custom%20fields%20in%20the%20event%20payload.%0A-%20The%20frontend%20expects%20JSON%20messages%20and%20can%20access%20any%20custom%20fields%20you%20add.%0A-%20Use%20the%20listener%20pattern%20to%20handle%20updates%20for%20specific%20jobs.%0A-%20Handle%20reconnection%20and%20fallback%20to%20polling%20for%20reliability.%0A%0ALet%20me%20know%20if%20you%20want%20a%20more%20detailed%20React%20hook%20or%20integration%20example%21) [](https://go.dosu.dev/discord-bot) [](https://twitter.com/intent/tweet?text=%40dosu_ai%20helped%20me%20solve%20this%20issue!&url=https%3A//github.com/apache/superset/discussions/36383)
GitHub link: https://github.com/apache/superset/discussions/36383#discussioncomment-15148250
----
This is an automatically sent email for notifications@superset.apache.org.
To unsubscribe, please send an email to: notifications-unsubscribe@superset.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [D] How the WebSocket server interacts with Redis ? [superset]
Posted by "dosubot[bot] (via GitHub)" <gi...@apache.org>.
GitHub user dosubot[bot] edited a comment on the discussion: How the WebSocket server interacts with Redis ?
<!-- Answer -->
Here’s how you can configure superset-websocket and Superset for async event messaging, and implement a custom message format with fields like request_id, type, axis, level, offset, limit, payload, metrics_meta, and error:
**Configuration Examples**
In `superset_config.py` (Superset backend):
```python
GLOBAL_ASYNC_QUERIES = True
GLOBAL_ASYNC_QUERIES_TRANSPORT = "ws"
GLOBAL_ASYNC_QUERIES_WEBSOCKET_URL = "ws://localhost:8080/"
GLOBAL_ASYNC_QUERIES_REDIS_CONFIG = {
"host": "127.0.0.1",
"port": 6379,
"db": 2,
"ssl": False
}
GLOBAL_ASYNC_QUERIES_JWT_SECRET = "your-secret-key"
GLOBAL_ASYNC_QUERIES_JWT_COOKIE_NAME = "async-token"
GLOBAL_ASYNC_QUERIES_REDIS_STREAM_PREFIX = "async-events-"
```
In `superset-websocket/config.json`:
```json
{
"port": 8080,
"redis": {
"port": 6379,
"host": "127.0.0.1",
"db": 2,
"ssl": false
},
"redisStreamPrefix": "async-events-",
"jwtSecret": "your-secret-key",
"jwtCookieName": "async-token"
}
```
Make sure the JWT secret, cookie name, Redis details, and stream prefix match in both configs. The hostname must match how you access Superset in your browser for cookies to work. Both the Flask server and superset-websocket server must be running simultaneously for async features to work [source](https://github.com/apache/superset/issues/36069).
**Custom Message Format Implementation**
Superset’s async event messages are built in the backend (see `async_query_manager.py`). The event data dict is constructed using `build_job_metadata` and updated via `update_job`. You can add custom fields (like request_id, type, axis, etc.) by passing them as keyword arguments to `update_job`. These fields will be included in the JSON payload written to Redis and forwarded by superset-websocket to clients [source](https://github.com/apache/superset/blob/829e4d92d91ceae4b43b1ed3b063ffe45377799c/superset/async_events/async_query_manager.py#L53-L308).
**Example: Custom Event Payload**
Here’s what a custom event payload might look like:
```json
{
"id": "1615426152415-0", // Redis stream ID
"channel_id": "abc123",
"job_id": "job-uuid",
"user_id": "1",
"request_id": "req-uuid",
"type": "query",
"axis": "x",
"level": "detail",
"offset": 0,
"limit": 100,
"payload": { "data": [...] },
"metrics_meta": { "duration_ms": 123 },
"error": null,
"status": "done",
"result_url": "/api/v1/chart/data/job-uuid"
}
```
To implement this, update your async query Celery tasks (e.g., in `async_queries.py`) to pass these fields to `async_query_manager.update_job`. For example:
```python
async_query_manager.update_job(
job_metadata,
async_query_manager.STATUS_DONE,
request_id="req-uuid",
type="query",
axis="x",
level="detail",
offset=0,
limit=100,
payload=payload,
metrics_meta={"duration_ms": 123},
error=None,
result_url=result_url,
)
```
Any fields you add here will be included in the event data sent to Redis and delivered to the frontend via WebSocket [source](https://github.com/apache/superset/blob/829e4d92d91ceae4b43b1ed3b063ffe45377799c/superset/tasks/async_queries.py#L82-L166).
**Best Practices & Caveats**
- There’s no explicit config option for message format in superset-websocket; you control the payload in the backend code.
- Make sure any consumers (frontend, other services) expect and handle your custom fields.
- The error field is typically a list of error dicts, populated if exceptions occur during async query execution.
- There’s no enforced payload size limit in superset-websocket, but you can set limits in the backend (e.g., `SQLLAB_PAYLOAD_MAX_MB` for SQL Lab) [source](https://github.com/apache/superset/pull/30522).
Let me know if you want code snippets for a specific async query type or more details on frontend handling!
<!-- Dosu Comment Footer -->
*To reply, just mention [@dosu](https://go.dosu.dev/dosubot?utm_source=github&utm_medium=bot-comment&utm_campaign=github-comment-footer-20260415&utm_content=reply-with-mention&utm_term=apache%2Fsuperset).*
---
Share context across your team and agents. Try [Dosu](https://dosu.dev?utm_source=github&utm_medium=bot-comment&utm_campaign=github-comment-footer-20260415&utm_content=knowledge-infrastructure-tagline&utm_term=apache%2Fsuperset).
[](https://app.dosu.dev/response-feedback?utm_source=github&utm_medium=bot-comment&utm_campaign=github-comment-footer-20260415&utm_content=knowledge-infrastructure-feedback&utm_term=apache%2Fsuperset&message_id=d81218be-f731-4857-8fd4-b69492a5d8d4) [](https://github.dosu.com/apache/superset?utm_source=github&utm_medium=bot-comment&utm_campaign=github-comment-footer-20260415&utm_content=knowledge-infrastructure-learn-repo&utm_term=apache%2Fsuperset) [](https://app.dosu.dev/signup?referrer=openSource&source=github-footer&utm_source=github&utm_medium=bot-comment&utm_campaign=github-comment-footer-20260415&utm_content=knowledge-infrastructure-add-team&utm_term=apache%2Fsuperset)
GitHub link: https://github.com/apache/superset/discussions/36383#discussioncomment-15146911
----
This is an automatically sent email for notifications@superset.apache.org.
To unsubscribe, please send an email to: notifications-unsubscribe@superset.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [D] How the WebSocket server interacts with Redis ? [superset]
Posted by "saha23412 (via GitHub)" <gi...@apache.org>.
GitHub user saha23412 added a comment to the discussion: How the WebSocket server interacts with Redis ?
After all that has been suggested, how will the frontend part interact with the backend and the superset-websocket module, can I have some code examples?
GitHub link: https://github.com/apache/superset/discussions/36383#discussioncomment-15148216
----
This is an automatically sent email for notifications@superset.apache.org.
To unsubscribe, please send an email to: notifications-unsubscribe@superset.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org