You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@superset.apache.org by "michael-s-molina (via GitHub)" <gi...@apache.org> on 2026/04/07 19:21:36 UTC
[PR] feat(extensions): add Tier 1 and Tier 2 storage APIs [superset]
michael-s-molina opened a new pull request, #39171:
URL: https://github.com/apache/superset/pull/39171
## Summary
Superset Extensions need a stable, safe, and well-scoped API to persist data. Without a first-class storage API, extensions are forced to roll their own storage β writing directly to localStorage with ad-hoc keys on the client, or inserting rows into Superset's internal key_value table on the server β with no consistent namespacing, no isolation from Superset's own data, and no defined cleanup path on uninstall. This PR introduces the first-class storage API to prevent these issues.
### Changes
- Add managed storage APIs for extensions with automatic namespace isolation
- Tier 1: Browser-based storage (`localState`, `sessionState`) with user isolation
- Tier 2: Server-side cache (`ephemeralState`) with TTL support and Redis/Memcached compatibility
- Storage is automatically bound to extensions before module execution, ensuring data privacy
- Update webpack externals to support subpath imports like `@apache-superset/core/storage`
- Add new documentation page explaining how storage works
### Tier 1: Local State (Browser Storage)
```typescript
import { localState, sessionState } from '@apache-superset/core/storage';
// Persists across browser sessions
await localState.set('sidebar_collapsed', true);
const isCollapsed = await localState.get('sidebar_collapsed');
// Cleared when tab closes
await sessionState.set('wizard_step', 3);
```
### Tier 2: Ephemeral State (Server Cache)
```typescript
import { ephemeralState } from '@apache-superset/core/storage';
// Store with default TTL (1 hour)
await ephemeralState.set('job_progress', { pct: 42, status: 'running' });
// Store with custom TTL (5 minutes)
await ephemeralState.set('quick_cache', data, { ttl: 300 });
// Retrieve
const progress = await ephemeralState.get('job_progress');
```
### Shared State (Cross-User)
```typescript
// Shared across all users
await localState.shared().set('device_id', 'abc-123');
await ephemeralState.shared().set('shared_result', { data: [1, 2, 3] });
```
## Test plan
- [ ] Verify `localState` persists data across page reloads in browser localStorage
- [ ] Verify `sessionState` clears when browser tab is closed
- [ ] Verify `ephemeralState` stores/retrieves data via the backend API
- [ ] Verify storage is isolated per extension (different extensions using same key don't collide)
- [ ] Verify user-scoped storage is isolated per user
- [ ] Verify `shared()` storage is accessible across users
π€ Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(extensions): add a Storage API for extensions [superset]
Posted by "michael-s-molina (via GitHub)" <gi...@apache.org>.
michael-s-molina commented on code in PR #39171:
URL: https://github.com/apache/superset/pull/39171#discussion_r3561220236
##########
superset-frontend/src/core/storage/persistentState.test.ts:
##########
@@ -0,0 +1,249 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import { SupersetClient } from '@superset-ui/core';
+import { createPersistentState } from './persistentState';
+
+jest.mock('@superset-ui/core', () => ({
+ SupersetClient: {
+ get: jest.fn(),
+ put: jest.fn(),
+ delete: jest.fn(),
+ },
+}));
+
+const mockGet = SupersetClient.get as jest.Mock;
+const mockPut = SupersetClient.put as jest.Mock;
+const mockDelete = SupersetClient.delete as jest.Mock;
+
+beforeEach(() => {
+ jest.clearAllMocks();
+ mockGet.mockResolvedValue({ json: { result: null } });
+ mockPut.mockResolvedValue({});
+ mockDelete.mockResolvedValue({});
+});
+
+test('get calls correct URL with publisher/name pattern', async () => {
+ const store = createPersistentState('myorg.myext');
+ await store.get('prefs');
+ expect(mockGet).toHaveBeenCalledWith({
+ endpoint: '/api/v1/extensions/myorg/myext/storage/persistent/prefs',
+ });
+});
+
+test('get returns result from response', async () => {
+ mockGet.mockResolvedValue({ json: { result: { theme: 'dark' } } });
+ const store = createPersistentState('myorg.myext');
+ expect(await store.get('prefs')).toEqual({ theme: 'dark' });
+});
+
+test('get returns null when result is absent', async () => {
+ mockGet.mockResolvedValue({ json: {} });
+ const store = createPersistentState('myorg.myext');
+ expect(await store.get('key')).toBeNull();
+});
+
+test('set calls correct URL with value in body', async () => {
+ const store = createPersistentState('myorg.myext');
+ await store.set('prefs', { theme: 'dark' });
+ expect(mockPut).toHaveBeenCalledWith({
+ endpoint: '/api/v1/extensions/myorg/myext/storage/persistent/prefs',
+ body: JSON.stringify({
+ value: { theme: 'dark' },
+ encrypt: false,
+ codec: 'json',
+ }),
+ headers: { 'Content-Type': 'application/json' },
+ });
+});
+
+test('set passes codec from options', async () => {
+ const store = createPersistentState('myorg.myext');
+ await store.set('prefs', 'sk-...', { codec: 'pickle' });
+ expect(mockPut).toHaveBeenCalledWith({
+ endpoint: '/api/v1/extensions/myorg/myext/storage/persistent/prefs',
+ body: JSON.stringify({
+ value: 'sk-...',
+ encrypt: false,
+ codec: 'pickle',
+ }),
+ headers: { 'Content-Type': 'application/json' },
+ });
Review Comment:
Same as the reply on the `ephemeralState.test.ts:80` thread β this test asserts client-side codec passthrough, not server acceptance, so it's testing real, intended behavior of the SDK layer.
##########
superset-frontend/src/core/storage/persistentState.ts:
##########
@@ -0,0 +1,147 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import type {
+ JsonValue,
+ PersistentListOptions,
+ PersistentListResult,
+ PersistentSetOptions,
+ PersistentStorageAccessor,
+ PersistentStorageTier,
+} from '@apache-superset/core/storage';
+import { SupersetClient } from '@superset-ui/core';
+import { resolveSetPayload } from './binaryCodec';
+
+/**
+ * Create persistent state (database-backed) bound to an extension ID.
+ */
+export function createPersistentState(
+ extensionId: string,
+): PersistentStorageTier {
+ const MAX_KEY_LENGTH = 255;
+ const [publisher, name] = extensionId.split('.');
+
+ const buildUrl = (key: string, shared?: boolean): string => {
+ if (key.length > MAX_KEY_LENGTH) {
+ throw new Error(
+ `Persistent storage key must be ${MAX_KEY_LENGTH} characters or less.`,
+ );
+ }
+ const encodedPublisher = encodeURIComponent(publisher);
+ const encodedName = encodeURIComponent(name);
+ const encodedKey = encodeURIComponent(key);
+ const url = `/api/v1/extensions/${encodedPublisher}/${encodedName}/storage/persistent/${encodedKey}`;
+ return shared ? `${url}?shared=true` : url;
+ };
+
+ const buildListUrl = (
+ options: PersistentListOptions,
+ isShared?: boolean,
+ ): string => {
+ const encodedPublisher = encodeURIComponent(publisher);
+ const encodedName = encodeURIComponent(name);
+ const url = `/api/v1/extensions/${encodedPublisher}/${encodedName}/storage/persistent`;
+ const params = new URLSearchParams();
+ if (isShared) params.set('shared', 'true');
+ if (options.resourceType) params.set('resource_type', options.resourceType);
+ if (options.resourceUuid) params.set('resource_uuid', options.resourceUuid);
+ params.set('page', String(options.page));
+ params.set('page_size', String(options.pageSize));
+ const query = params.toString();
+ return query ? `${url}?${query}` : url;
+ };
+
+ const list = async <T = JsonValue>(
+ options: PersistentListOptions,
+ isShared: boolean,
+ ): Promise<PersistentListResult<T>> => {
+ const response = await SupersetClient.get({
+ endpoint: buildListUrl(options, isShared),
+ });
+ return {
+ entries: response.json?.result ?? [],
+ count: response.json?.count ?? 0,
+ };
+ };
+
+ const shared: PersistentStorageAccessor = {
+ async get<T = JsonValue>(key: string): Promise<T | null> {
+ const response = await SupersetClient.get({
+ endpoint: buildUrl(key, true),
+ });
+ return (response.json?.result ?? null) as T | null;
+ },
+ async set<T = JsonValue>(
+ key: string,
+ value: T,
+ options?: PersistentSetOptions,
+ ): Promise<void> {
+ const payload = resolveSetPayload(value, options?.codec);
+ await SupersetClient.put({
+ endpoint: buildUrl(key, true),
+ body: JSON.stringify({
+ value: payload.value,
+ encrypt: options?.encrypt ?? false,
+ codec: payload.codec,
+ }),
+ headers: { 'Content-Type': 'application/json' },
+ });
+ },
+ async list<T = JsonValue>(
+ options: PersistentListOptions,
+ ): Promise<PersistentListResult<T>> {
+ return list<T>(options, true);
+ },
+ async remove(key: string): Promise<void> {
+ await SupersetClient.delete({ endpoint: buildUrl(key, true) });
+ },
+ };
+
+ return {
+ async get<T = JsonValue>(key: string): Promise<T | null> {
+ const response = await SupersetClient.get({ endpoint: buildUrl(key) });
+ return (response.json?.result ?? null) as T | null;
+ },
+ async set<T = JsonValue>(
+ key: string,
+ value: T,
+ options?: PersistentSetOptions,
+ ): Promise<void> {
+ const payload = resolveSetPayload(value, options?.codec);
+ await SupersetClient.put({
+ endpoint: buildUrl(key),
+ body: JSON.stringify({
+ value: payload.value,
+ encrypt: options?.encrypt ?? false,
+ codec: payload.codec,
+ }),
+ headers: { 'Content-Type': 'application/json' },
+ });
Review Comment:
Not a bug β by design. We deliberately don't duplicate the server's `SAFE_CODECS` allowlist on the client; the REST API is the single source of truth and returns a 400 for a disallowed codec. Maintaining a second copy of that allowlist client-side would risk drifting out of sync with the server's actual list.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(extensions): add a Storage API for extensions [superset]
Posted by "codeant-ai-for-open-source[bot] (via GitHub)" <gi...@apache.org>.
codeant-ai-for-open-source[bot] commented on code in PR #39171:
URL: https://github.com/apache/superset/pull/39171#discussion_r3561221885
##########
superset-frontend/packages/superset-core/src/storage/index.ts:
##########
@@ -0,0 +1,208 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/**
+ * @fileoverview Shared types for extension storage APIs.
+ *
+ * These types are shared across all storage tiers (local, session, ephemeral,
+ * persistent) to ensure a consistent API pattern.
+ */
+
+/**
+ * JSON-compatible value type.
+ * These are the only values that can be safely serialized/deserialized via JSON.
+ */
+export type JsonValue =
+ string | number | boolean | null | JsonValue[] | { [key: string]: JsonValue };
+
+/**
+ * Base interface for a storage accessor.
+ * All storage tiers implement this interface for both user-scoped and shared access.
+ */
+export interface StorageAccessor {
+ /**
+ * Get a value from storage.
+ *
+ * @param key The key to retrieve.
+ * @returns The stored value, or null if not found.
+ */
+ get<T = JsonValue>(key: string): Promise<T | null>;
+
+ /**
+ * Set a value in storage.
+ *
+ * @param key The key to store.
+ * @param value The value to store (must be JSON-serializable).
+ */
+ set<T = JsonValue>(key: string, value: T): Promise<void>;
Review Comment:
β
**Customized review instruction saved!**
**Instruction:**
> Do not flag unconstrained generic storage `set()`/`get()` signatures as API mismatches when the method intentionally supports non-JSON values via a `codec` option; only enforce JSON-compatible constraints where no such codec path exists.
**Applied to:**
- `superset-frontend/packages/superset-core/src/storage/**`
---
π‘ *To manage or update this instruction, visit: [CodeAnt AI Settings](https://app.codeant.ai/org/settings/learnings)*
##########
superset-frontend/src/core/storage/binaryCodec.ts:
##########
@@ -0,0 +1,70 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/** Convert bytes to a base64 string, for transport over the JSON REST API. */
+export function bytesToBase64(bytes: Uint8Array): string {
+ let binary = '';
+ bytes.forEach(byte => {
+ binary += String.fromCharCode(byte);
+ });
+ return btoa(binary);
+}
+
+function isBinaryValue(value: unknown): value is Uint8Array | ArrayBuffer {
+ return value instanceof Uint8Array || value instanceof ArrayBuffer;
+}
+
+/**
+ * Infer which codec to use for `value` when the caller didn't specify one.
+ *
+ * "json" names the serialization format applied to encode the value for
+ * storage/transport -- it is not a claim about the value's shape. Per
+ * RFC 8259, a JSON text may be any JSON value (object, array, string,
+ * number, boolean, or null), so scalars like `1` or `"foo"` are encoded
+ * with the "json" codec too, alongside objects and arrays.
+ *
+ * Binary values (Uint8Array/ArrayBuffer) are the one case JSON cannot
+ * represent, so they're base64-encoded instead.
+ */
+function inferDefaultCodec(value: unknown): 'json' | 'base64' {
+ return isBinaryValue(value) ? 'base64' : 'json';
+}
+
+/**
+ * Resolve the codec and request-body value for a `set()` call.
+ *
+ * If the caller explicitly set `options.codec`, it is always respected
+ * as-is and `value` is passed through untouched -- the caller is
+ * responsible for encoding it correctly (e.g. base64-encoding it
+ * themselves before calling `set()`). Auto base64-encoding only happens
+ * when no codec was specified and `value` is binary.
+ */
+export function resolveSetPayload<T>(
+ value: T,
+ explicitCodec: string | undefined,
+): { value: unknown; codec: string } {
+ if (explicitCodec !== undefined) {
+ return { value, codec: explicitCodec };
Review Comment:
β
**Customized review instruction saved!**
**Instruction:**
> Do not flag the explicit-codec fast path in storage binary codec helpers: when a caller supplies `codec`, pass `value` through unchanged and let the caller be responsible for encoding it correctly.
**Applied to:**
- `superset-frontend/src/core/storage/**`
---
π‘ *To manage or update this instruction, visit: [CodeAnt AI Settings](https://app.codeant.ai/org/settings/learnings)*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(extensions): add Tier 1 and Tier 2 storage APIs [superset]
Posted by "codeant-ai-for-open-source[bot] (via GitHub)" <gi...@apache.org>.
codeant-ai-for-open-source[bot] commented on code in PR #39171:
URL: https://github.com/apache/superset/pull/39171#discussion_r3047372708
##########
superset/utils/cache_manager.py:
##########
@@ -237,6 +247,12 @@ def init_app(self, app: Flask) -> None:
"EXPLORE_FORM_DATA_CACHE_CONFIG",
required=True,
)
+ self._init_cache(
+ app,
+ self._extension_ephemeral_state_cache,
+ app.config["EXTENSIONS_STORAGE"]["EPHEMERAL"],
Review Comment:
**Suggestion:** Directly indexing nested config keys will raise a startup-time exception if a deployment overrides `EXTENSIONS_STORAGE` without defining `EPHEMERAL`. Use a safe lookup with a default empty dict so cache initialization can still fall back through the existing `required=True` logic instead of crashing with `KeyError`. [possible bug]
<details>
<summary><b>Severity Level:</b> Major β οΈ</summary>
```mdx
- β Superset app fails to start when EXTENSIONS_STORAGE misconfigured.
- β οΈ Extensions ephemeral storage API unusable until configuration corrected.
```
</details>
```suggestion
app.config.get("EXTENSIONS_STORAGE", {}).get("EPHEMERAL", {}),
```
<details>
<summary><b>Steps of Reproduction β
</b></summary>
```mdx
1. Configure a deployment with a custom `EXTENSIONS_STORAGE` that omits the `EPHEMERAL`
key (e.g. via `superset_config.py`, which overrides `superset/config.py:1155` where the
default `EXTENSIONS_STORAGE` dict is defined).
2. Start the Superset application so the initialization pipeline runs
`SupersetAppInitializer.configure_cache()` at
`superset/initialization/__init__.py:810-812`, which calls
`cache_manager.init_app(self.superset_app)`.
3. Inside `CacheManager.init_app` at `superset/utils/cache_manager.py:237-255`, the code
executes `_init_cache(app, self._extension_ephemeral_state_cache,
app.config["EXTENSIONS_STORAGE"]["EPHEMERAL"], required=True)`.
4. Because `app.config["EXTENSIONS_STORAGE"]` exists but has no `"EPHEMERAL"` entry,
indexing `["EPHEMERAL"]` at `superset/utils/cache_manager.py:253` raises a `KeyError`
during startup, causing the Superset app to fail to start before any HTTP endpoints
(including the extensions storage API in `superset/extensions/storage/api.py:124-187`) are
available.
```
</details>
<details>
<summary><b>Prompt for AI Agent π€ </b></summary>
```mdx
This is a comment left during a code review.
**Path:** superset/utils/cache_manager.py
**Line:** 253:253
**Comment:**
*Possible Bug: Directly indexing nested config keys will raise a startup-time exception if a deployment overrides `EXTENSIONS_STORAGE` without defining `EPHEMERAL`. Use a safe lookup with a default empty dict so cache initialization can still fall back through the existing `required=True` logic instead of crashing with `KeyError`.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F39171&comment_hash=648a4ff6c6edd48bdd0ed06e1cfb55fc44f4a2ae0e7612be42b461a17ed6e299&reaction=like'>π</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F39171&comment_hash=648a4ff6c6edd48bdd0ed06e1cfb55fc44f4a2ae0e7612be42b461a17ed6e299&reaction=dislike'>π</a>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(extensions): add a Storage API for extensions [superset]
Posted by "codeant-ai-for-open-source[bot] (via GitHub)" <gi...@apache.org>.
codeant-ai-for-open-source[bot] commented on code in PR #39171:
URL: https://github.com/apache/superset/pull/39171#discussion_r3622841518
##########
superset/extensions/storage/persistent_model.py:
##########
@@ -0,0 +1,133 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import uuid as uuid_module
+
+from flask_appbuilder import Model
+from sqlalchemy import (
+ Boolean,
+ Column,
+ ForeignKey,
+ Index,
+ Integer,
+ LargeBinary,
+ String,
+ UniqueConstraint,
+)
+from sqlalchemy_utils import UUIDType
+from superset_core.extensions.storage.models import (
+ ExtensionStorageEntry as CoreExtensionStorageEntry,
+)
+
+from superset.models.helpers import AuditMixinNullable
+
+
+class ExtensionStorage(CoreExtensionStorageEntry, AuditMixinNullable, Model):
+ """Generic persistent key-value storage for extensions (Tier 3).
+
+ Each row is identified by (extension_id, user_fk, resource_type,
+ resource_uuid, key):
+
+ * Global scope β user_fk IS NULL, resource_type IS NULL
+ * User scope β user_fk set, resource_type IS NULL
+ * Resource scope β resource_type + resource_uuid set (user_fk optional)
+
+ The payload is stored as raw bytes (value) alongside the identifier of
+ the codec used to encode it (codec), so the same bytes can be decoded
+ back into a value on read. When is_encrypted is True the value has been
+ encrypted at the DAO layer using Fernet and must be decrypted before
+ decoding.
+ """
+
+ __tablename__ = "extension_storage"
+
+ id = Column(Integer, primary_key=True, autoincrement=True)
+ uuid = Column(
+ UUIDType(binary=True),
+ default=uuid_module.uuid4,
+ unique=True,
+ nullable=False,
+ )
+
+ # Extension identity
+ extension_id = Column(String(255), nullable=False)
+
+ # Scope discriminators β all nullable; NULLs define the scope (see docstring).
+ # No relationship() is declared for user_fk: extension storage rows are
+ # not deleted when their owning user is (ondelete="SET NULL" demotes them
+ # to global scope instead), and nothing reads through such a relationship,
+ # so there's no ORM object here for a future change to attach a cascade to.
+ user_fk = Column(
+ Integer,
+ ForeignKey(
+ "ab_user.id",
+ ondelete="SET NULL",
+ name="fk_extension_storage_user_fk_ab_user",
+ ),
+ nullable=True,
+ )
+ # No FK here: resource_type is a free-form string naming the referenced
+ # table (a built-in Superset model or a resource type defined by another
+ # extension entirely), so the table a given row's resource_uuid points
+ # into varies per row and can't be fixed at the schema level.
+ resource_type = Column(String(64), nullable=True)
+ resource_uuid = Column(String(36), nullable=True)
+
+ # Storage key within the scope
+ key = Column(String(255), nullable=False)
+
+ # Payload. Size is bounded at the application level by
+ # EXTENSIONS_PERSISTENT_STORAGE["MAX_VALUE_SIZE"], enforced in
+ # ExtensionStorageDAO.set(), rather than by a fixed column length here.
+ value = Column(LargeBinary, nullable=False)
+ # Byte length of `value`, kept in sync at write time so the quota SUM in
+ # _check_quota() can be computed from this fixed-width column instead of
+ # reading (and, once TOASTed, detoasting) every extension's stored blobs.
+ value_size = Column(Integer, nullable=False)
+ codec = Column(String(255), nullable=False, default="json")
+ is_encrypted = Column(Boolean, nullable=False, default=False)
+
+ __table_args__ = (
+ # Unique constraint prevents duplicate rows from concurrent writes
+ UniqueConstraint(
+ "extension_id",
+ "user_fk",
+ "resource_type",
+ "resource_uuid",
+ "key",
+ name="uq_extension_storage_scoped_key",
+ ),
Review Comment:
**Suggestion:** The scoped uniqueness guarantee is broken because the unique constraint includes nullable columns (`user_fk`, `resource_type`, `resource_uuid`), and PostgreSQL treats `NULL` values as distinct. This allows duplicate rows for the same logical key in global/user scopes, which can cause ambiguous reads and duplicate writes under concurrency. Use scope-specific partial unique indexes or normalized non-null sentinel columns so logical keys are truly unique. [logic error]
<details>
<summary><b>Severity Level:</b> Critical π¨</summary>
```mdx
- β Duplicate Tier 3 records for same scoped key.
- β οΈ Ambiguous reads, possible DAO MultipleResultsFound.
```
</details>
<details>
<summary><b>Steps of Reproduction β
</b></summary>
```mdx
1. Start Superset with this PR applied and a PostgreSQL backend; the `ExtensionStorage`
ORM model is defined in `superset/extensions/storage/persistent_model.py:38-127`.
2. From an extension, call Tier 3 storage as documented (e.g.,
`ctx.storage.persistent.set('sidebar_collapsed', true)`), which is implemented server-side
via `ExtensionStorageDAO.set()` referenced in the comment at `persistent_model.py:94-95`.
3. Generate two concurrent requests (for example, two browser tabs using the same
extension) that both call `ctx.storage.persistent.set('sidebar_collapsed', true)` for the
same `extension_id` using global scope (so `user_fk`, `resource_type`, and `resource_uuid`
are all `NULL`).
4. Inspect the `extension_storage` table after both requests complete: despite the
intended logical key `(extension_id, user_fk, resource_type, resource_uuid, key)`,
PostgreSQL permits multiple rows with `user_fk=NULL`, `resource_type=NULL`,
`resource_uuid=NULL` and identical `extension_id` and `key`, because the unique constraint
`uq_extension_storage_scoped_key` at `persistent_model.py:106-113` does not treat `NULL`
values as equal, resulting in duplicate logical keys under concurrency.
```
</details>
[](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=82fddf7ea8d44d16ade36a638281e506&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=82fddf7ea8d44d16ade36a638281e506&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset)
*(Use Cmd/Ctrl + Click for best experience)*
<details>
<summary><b>Prompt for AI Agent π€ </b></summary>
```mdx
This is a comment left during a code review.
**Path:** superset/extensions/storage/persistent_model.py
**Line:** 106:113
**Comment:**
*Logic Error: The scoped uniqueness guarantee is broken because the unique constraint includes nullable columns (`user_fk`, `resource_type`, `resource_uuid`), and PostgreSQL treats `NULL` values as distinct. This allows duplicate rows for the same logical key in global/user scopes, which can cause ambiguous reads and duplicate writes under concurrency. Use scope-specific partial unique indexes or normalized non-null sentinel columns so logical keys are truly unique.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F39171&comment_hash=b0dc7572751ac725b0dd5ccfee0d8b0f4d3d9c57ca290f0e3abc7446d80697aa&reaction=like'>π</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F39171&comment_hash=b0dc7572751ac725b0dd5ccfee0d8b0f4d3d9c57ca290f0e3abc7446d80697aa&reaction=dislike'>π</a>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(extensions): add a Storage API for extensions [superset]
Posted by "villebro (via GitHub)" <gi...@apache.org>.
villebro commented on code in PR #39171:
URL: https://github.com/apache/superset/pull/39171#discussion_r3539606790
##########
superset-frontend/packages/superset-core/src/common/index.ts:
##########
@@ -227,8 +227,6 @@ export interface Extension {
id: string;
/** Human-readable name of the extension */
name: string;
- /** URL or path to the extension's remote entry point */
- remoteEntry: string;
Review Comment:
It took me a while to re-remember that `LoadedExtension` extends `Extension`, and has `remoteEntry`. As a bycatch, maybe we could also update the description to explain that this is the base metadata, rather than only explaining what an extension is (the description for `LoadedExtension` was very descriptive for a developer working on the frontend codebase).
##########
superset/extensions/storage/ephemeral.py:
##########
@@ -0,0 +1,181 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""
+Host implementation for Ephemeral State (Tier 2 Storage).
+
+Provides the concrete cache-backed implementation that is injected into
+superset_core.extensions.storage.ephemeral at startup.
+"""
+
+from __future__ import annotations
+
+from typing import Any, ClassVar
+
+from flask import g
+from superset_core.extensions.storage.ephemeral import (
+ EphemeralSetOptions,
+ EphemeralState as CoreEphemeralState,
+)
+
+from superset.extensions import cache_manager
+from superset.extensions.context import get_current_extension_context
+
+# Key separator
+SEPARATOR = ":"
+
+# Key prefix for extension ephemeral state
+KEY_PREFIX = "superset-ext"
+
+
+def _get_extension_id() -> str:
+ """Get the current extension ID from context."""
+ context = get_current_extension_context()
+ if context is None:
+ raise RuntimeError(
+ "ephemeral_state can only be used within an extension context. "
+ "Ensure this code is being executed during extension loading or "
+ "within an extension API request handler."
+ )
+ return context.manifest.id
+
+
+def _get_current_user_id() -> int:
+ """Get the current authenticated user's ID."""
+ user = getattr(g, "user", None)
+ if user is None or not hasattr(user, "id"):
+ raise RuntimeError(
+ "ephemeral_state requires an authenticated user. "
+ "Ensure the request has been authenticated."
+ )
+ return user.id
+
+
+def _build_cache_key(*parts: Any) -> str:
+ """Build a namespaced cache key from parts."""
+ return SEPARATOR.join(str(part) for part in parts)
Review Comment:
The fact that this method is duplicated in `api.py` shows that this should at a minimum be relocated to `utils.py` for DRYing
##########
docs/developer_docs/extensions/storage.md:
##########
@@ -0,0 +1,348 @@
+---
+title: Storage
+sidebar_position: 8
+---
+
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Storage
+
+Superset Extensions have access to a managed storage API for persisting data. The storage system provides multiple tiers with different persistence characteristics, allowing extensions to choose the right storage for their needs.
+
+Each extension receives its own isolated storage namespace. When Superset loads your extension, it binds storage to your extension's unique identifier, ensuring data privacyβtwo extensions using the same key will never collide, and extensions cannot access each other's data.
+
+Storage is always accessed through the extension context via `getContext()`. This binding to the context is what ties every operation to the current extension's namespace.
+
+## Storage Tiers
+
+| Tier | Storage Type | Context Property | Use Case |
+| ---- | ----------------- | ------------------------------------------ | -------------------------------------- |
+| 1 | Browser storage | `ctx.storage.local`, `ctx.storage.session` | UI state, wizard progress, draft forms |
+| 2 | Server-side cache | `ctx.storage.ephemeral` | Job progress, temporary results |
+| 3 | Database | `ctx.storage.persistent` | Saved artifacts, durable config |
+
+## Tier 1: Local State
+
+Browser-based storage that persists on the user's device. Use this for UI state and settings that don't need to sync across devices.
+
+### Why Use the API Instead of localStorage Directly?
+
+You might wonder why extensions should use `ctx.storage.local` instead of directly accessing `window.localStorage`. The managed API provides several benefits:
+
+- **Automatic namespacing**: Each extension's data is isolated. Two extensions using the same key name won't collide.
+- **User isolation**: By default, data is scoped to the current user, preventing data leakage between users on shared devices.
+- **Clean uninstall**: When an extension is uninstalled, all its data can be cleanly removed using prefix-based deletion.
+- **Future sandboxing**: The async API is designed for a future sandboxed execution model where extensions run in isolated contexts without direct DOM access.
+- **Consistent patterns**: The same API shape works across all storage tiers, making it easy to switch between them.
+
+### localState
+
+Data persists across browser sessions until explicitly deleted or the user clears browser storage.
+
+```typescript
+import { extensions } from '@apache-superset/core';
+
+const ctx = extensions.getContext();
+
+// Save sidebar state
+await ctx.storage.local.set('sidebar_collapsed', true);
+
+// Retrieve it later
+const isCollapsed = await ctx.storage.local.get('sidebar_collapsed');
+
+// Remove it
+await ctx.storage.local.remove('sidebar_collapsed');
+```
+
+### sessionState
+
+Data is cleared when the browser tab is closed. Use for transient state within a single session.
+
+```typescript
+import { extensions } from '@apache-superset/core';
+
+const ctx = extensions.getContext();
+
+// Save wizard progress (lost when tab closes)
+await ctx.storage.session.set('wizard_step', 3);
+await ctx.storage.session.set('unsaved_form', { name: 'Draft' });
+
+// Retrieve on page reload within same tab
+const step = await ctx.storage.session.get('wizard_step');
+```
+
+Tier 1 has no `shared` accessor. Browser storage is per-device, so a "shared" value would only be visible to other users of that same browser, not to other users of the extension generally β see `.shared` on [Tier 2](#tier-2-ephemeral-state) or [Tier 3](#tier-3-persistent-state) for storage that's actually shared across users.
+
+### When to Use Tier 1
+
+- UI state (sidebar collapsed, panel sizes)
+- Recently used items
+- Draft form values
+- Any data acceptable to lose if user clears browser
+
+### Limitations
+
+- Per-browser, per-device (not shared across devices)
+- Subject to browser storage quotas (~5-10 MB)
+- Not accessible from backend code
+
+## Tier 2: Ephemeral State
+
+Server-side cache storage with automatic TTL expiration. Use for temporary data that needs to be shared between frontend and backend, or persist across page reloads.
+
+TTL is required for every `set` call. The maximum allowed TTL is controlled by `MAX_TTL` in the server configuration.
+
+### Frontend Usage
+
+```typescript
+import { extensions } from '@apache-superset/core';
+
+const ctx = extensions.getContext();
+
+// Store with a 5-minute TTL
+await ctx.storage.ephemeral.set('job_progress', { pct: 42, status: 'running' }, { ttl: 300 });
+
+// Retrieve
+const progress = await ctx.storage.ephemeral.get('job_progress');
+
+// Remove
+await ctx.storage.ephemeral.remove('job_progress');
+```
+
+### Backend Usage
+
+```python
+from superset_core.extensions.context import get_context
+from superset_core.extensions.storage.ephemeral import EphemeralSetOptions
+
+ctx = get_context()
+
+# Store job progress with a 5-minute TTL
+ctx.storage.ephemeral.set(
+ 'job_progress', {'pct': 42, 'status': 'running'}, EphemeralSetOptions(ttl=300)
+)
+
+# Retrieve
+progress = ctx.storage.ephemeral.get('job_progress')
+
+# Remove
+ctx.storage.ephemeral.remove('job_progress')
+```
+
+### Shared State
+
+For data that needs to be visible to all users:
+
+```typescript
+import { extensions } from '@apache-superset/core';
+
+const ctx = extensions.getContext();
+
+await ctx.storage.ephemeral.shared.set('shared_result', { data: [1, 2, 3] }, { ttl: 3600 });
+const result = await ctx.storage.ephemeral.shared.get('shared_result');
+```
+
+```python
+from superset_core.extensions.context import get_context
+from superset_core.extensions.storage.ephemeral import EphemeralSetOptions
+
+ctx = get_context()
+
+ctx.storage.ephemeral.shared.set(
+ 'shared_result', {'data': [1, 2, 3]}, EphemeralSetOptions(ttl=3600)
+)
+result = ctx.storage.ephemeral.shared.get('shared_result')
+```
+
+### When to Use Tier 2
+
+- Background job progress indicators
+- Cross-request intermediate state
+- Query result previews
+- Temporary computation results
+- Any data that can be recomputed if lost
+
+### Limitations
+
+- Not guaranteed to survive server restarts
+- Subject to cache eviction under memory pressure
+- TTL-based expiration (data disappears after timeout)
+- TTL is required on every `set` call
+
+## Tier 3: Persistent State
+
+Database-backed storage that survives server restarts, cache evictions, and browser clears. Use for any data that must not be lost.
+
+### Frontend Usage
+
+```typescript
+import { extensions } from '@apache-superset/core';
+
+const ctx = extensions.getContext();
+
+// Store a saved SQL snippet
+await ctx.storage.persistent.set('snippet:top_customers', { sql: 'SELECT ...' });
+
+// Retrieve
+const snippet = await ctx.storage.persistent.get('snippet:top_customers');
+
+// Remove
+await ctx.storage.persistent.remove('snippet:top_customers');
+```
+
+### Backend Usage
+
+```python
+from superset_core.extensions.context import get_context
+
+ctx = get_context()
+
+# Store a saved SQL snippet
+ctx.storage.persistent.set('snippet:top_customers', {'sql': 'SELECT ...'})
+
+# Retrieve
+snippet = ctx.storage.persistent.get('snippet:top_customers')
+
+# Remove
+ctx.storage.persistent.remove('snippet:top_customers')
+```
+
+### Shared State
+
+For data that should be visible to all users of the extension:
+
+```typescript
+import { extensions } from '@apache-superset/core';
+
+const ctx = extensions.getContext();
+
+await ctx.storage.persistent.shared.set('global_config', { version: 2 });
+const config = await ctx.storage.persistent.shared.get('global_config');
+```
+
+```python
+from superset_core.extensions.context import get_context
+
+ctx = get_context()
+
+ctx.storage.persistent.shared.set('global_config', {'version': 2})
+config = ctx.storage.persistent.shared.get('global_config')
+```
+
+### Enumerating and Managing Storage from the Backend
+
+The `ctx.storage.persistent` accessor is for single-key reads and writes. Backend extension code that needs to enumerate its own rows in bulk β for example, a cleanup routine that removes storage linked to resources that no longer exist β can use `superset_core.extensions.storage.dao.ExtensionStorageDAO` instead:
+
+```python
+from superset_core.extensions.storage.dao import ExtensionStorageDAO
+
+# All entries this extension has linked to a given resource type
+entries = ExtensionStorageDAO.filter_by(resource_type="my-resource-type")
+
+# Bulk-delete rows for resources that no longer exist
+orphaned = [e for e in entries if e.resource_uuid not in my_extension_live_resource_ids()]
+ExtensionStorageDAO.delete(orphaned)
+```
+
+`ExtensionStorageDAO` is automatically scoped to the calling extension's own rows β `extension_id` is never a parameter you pass, so there is no way to reach another extension's storage through this API. It supports the standard DAO operations (`find_all`, `find_one_or_none`, `filter_by`, `query`, `create`, `update`, `delete`), and, unlike `ctx.storage.persistent`, gives direct access to each row's scope columns (`resource_type`, `resource_uuid`, `category`, `description`) for entries created with those set.
+
+### When to Use Tier 3
+
+- Extension configuration that must survive restarts
+- User-specific saved artifacts that need to roam across devices and browsers (e.g. saved SQL snippets, annotations)
+- Any data where loss is unacceptable
+
+### Limitations
+
+- Higher latency than Tiers 1β2 (database round-trip per operation)
+- Subject to the 16 MB value size limit
+- Subject to the per-extension quota configured via `EXTENSIONS_PERSISTENT_STORAGE.QUOTA_PER_EXTENSION` (see [Quotas](#quotas))
+- Requires a database migration when first deployed
+
+## Key Patterns
+
+All storage keys are automatically namespaced:
+
+| Scope | Key Pattern |
+| ---------------------- | --------------------------------------------------- |
+| User-scoped | `superset-ext:{extension_id}:user:{user_id}:{key}` |
+| Shared (Tiers 2 and 3) | `superset-ext:{extension_id}:shared:{key}` |
Review Comment:
nit, typical AI indentation error:
```suggestion
| Scope | Key Pattern |
| ---------------------- | -------------------------------------------------- |
| User-scoped | `superset-ext:{extension_id}:user:{user_id}:{key}` |
| Shared (Tiers 2 and 3) | `superset-ext:{extension_id}:shared:{key}` |
```
##########
superset/extensions/storage/persistent_model.py:
##########
@@ -0,0 +1,129 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import uuid as uuid_module
+
+from flask_appbuilder import Model
+from sqlalchemy import (
+ Boolean,
+ Column,
+ ForeignKey,
+ Index,
+ Integer,
+ LargeBinary,
+ String,
+ Text,
+ UniqueConstraint,
+)
+from sqlalchemy.orm import backref, relationship
+from sqlalchemy_utils import UUIDType
+from superset_core.extensions.storage.models import (
+ ExtensionStorageEntry as CoreExtensionStorageEntry,
+)
+
+from superset.models.helpers import AuditMixinNullable
+
+# 16 MB β matches the KeyValue store limit.
+EXTENSION_STORAGE_MAX_SIZE = 2**24 - 1
+
+
+class ExtensionStorage(CoreExtensionStorageEntry, AuditMixinNullable, Model):
+ """Generic persistent key-value storage for extensions (Tier 3).
+
+ Each row is identified by (extension_id, user_fk, resource_type,
+ resource_uuid, key):
+
+ * Global scope β user_fk IS NULL, resource_type IS NULL
+ * User scope β user_fk set, resource_type IS NULL
+ * Resource scope β resource_type + resource_uuid set (user_fk optional)
+
+ The payload is stored as raw bytes (value) with a MIME-type hint
+ (value_type). When is_encrypted is True the value has been encrypted
+ at the DAO layer using Fernet and must be decrypted before use.
+ """
+
+ __tablename__ = "extension_storage"
+
+ id = Column(Integer, primary_key=True, autoincrement=True)
+ uuid = Column(
+ UUIDType(binary=True),
+ default=uuid_module.uuid4,
+ unique=True,
+ nullable=False,
+ )
+
+ # Extension identity
+ extension_id = Column(String(255), nullable=False)
+
+ # Scope discriminators β all nullable; NULLs define the scope (see docstring)
+ user_fk = Column(
+ Integer,
+ ForeignKey(
+ "ab_user.id",
+ ondelete="SET NULL",
+ name="fk_extension_storage_user_fk_ab_user",
+ ),
+ nullable=True,
+ )
+ resource_type = Column(String(64), nullable=True)
+ resource_uuid = Column(String(36), nullable=True)
Review Comment:
Maybe add a comment here why FK enforcement isn't possible (the reference table is variable)
##########
superset/extensions/storage/persistent_dao.py:
##########
@@ -0,0 +1,313 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from __future__ import annotations
+
+import hashlib
+import hmac
+import logging
+
+from flask import current_app
+from flask_babel import gettext as _
+from sqlalchemy import and_, func, LargeBinary
+
+from superset import db
+from superset.daos.base import BaseDAO
+from superset.exceptions import SupersetGenericErrorException
+from superset.extensions.storage.filters import ExtensionStorageFilter
+from superset.extensions.storage.persistent_model import ExtensionStorage
+from superset.utils.encrypt import (
+ DEFAULT_ENCRYPTION_ENGINE_NAME,
+ EncryptedType,
+ resolve_encryption_engine,
+)
+
+logger = logging.getLogger(__name__)
+
+
+class ExtensionStorageQuotaExceeded(SupersetGenericErrorException):
+ """Raised when a write would exceed an extension's persistent storage quota."""
+
+ status = 413
+
+ def __init__(self, extension_id: str, quota: int) -> None:
+ super().__init__(
+ message=_(
+ "Extension '%(extension_id)s' has exceeded its persistent "
+ "storage quota of %(quota)d bytes.",
+ extension_id=extension_id,
+ quota=quota,
+ ),
+ )
+
+
+def _derive_key(secret: str, user_fk: int) -> str:
+ """Derive a per-user encryption key from the master secret and user ID.
+
+ Uses HMAC-SHA256 so the derived key is cryptographically bound to both.
+ Ciphertext encrypted for one user cannot be decrypted as another.
+ """
+ return hmac.new(
+ secret.encode(),
+ str(user_fk).encode(),
+ hashlib.sha256,
+ ).hexdigest()
+
+
+def _enc_type(user_fk: int | None) -> EncryptedType:
+ """Return an EncryptedType for the given scope.
+
+ User-scoped rows use a key derived from SECRET_KEY and the user ID, so
+ ciphertext written for one user cannot be decrypted as another.
+ Shared rows (user_fk=None) use SECRET_KEY directly.
+ """
+ secret = current_app.config["SECRET_KEY"]
+ key = _derive_key(secret, user_fk) if user_fk is not None else secret
+ engine_name = current_app.config.get(
+ "SQLALCHEMY_ENCRYPTED_FIELD_ENGINE", DEFAULT_ENCRYPTION_ENGINE_NAME
+ )
+ return EncryptedType(
+ LargeBinary,
+ key=key,
+ engine=resolve_encryption_engine(engine_name),
+ )
+
+
+def _get_quota() -> int | None:
+ """Return the configured per-extension persistent storage quota in bytes.
+
+ Returns None when no quota is configured (unlimited).
+ """
+ config = current_app.config.get("EXTENSIONS_PERSISTENT_STORAGE", {})
+ return config.get("QUOTA_PER_EXTENSION")
+
+
+def _check_quota(extension_id: str, new_size: int, existing_size: int = 0) -> None:
+ """Raise if writing `new_size` bytes would push the extension over quota.
+
+ `existing_size` is the size of the row being replaced (0 for inserts),
+ subtracted from current usage so overwriting a key doesn't double-count
+ it against the quota it already occupies.
+ """
+ quota = _get_quota()
+ if quota is None:
+ return
+ current_usage = (
+ db.session.query(
+ func.coalesce(func.sum(func.length(ExtensionStorage.value)), 0)
+ )
+ .filter(ExtensionStorage.extension_id == extension_id)
+ .scalar()
+ )
+ if current_usage - existing_size + new_size > quota:
+ raise ExtensionStorageQuotaExceeded(extension_id, quota)
Review Comment:
Assuming we're indexing by extension this `SUM` operation should be very efficient, but I'd still like to call out that this can get expensive if an extension places lots of data in persistent storage.
##########
superset/extensions/storage/api.py:
##########
@@ -0,0 +1,499 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""
+REST API for extension storage.
+
+Provides HTTP endpoints for frontend extensions to access server-side
+ephemeral storage without direct backend code.
+
+All operations are user-scoped by default. Use `?shared=true` query param
+to access shared state visible to all users.
+"""
+
+from __future__ import annotations
+
+from typing import Any
+
+from flask import current_app, g, request
+from flask.wrappers import Response
+from flask_appbuilder.api import BaseApi, expose, protect, safe
+
+from superset.extensions import cache_manager
+from superset.extensions.storage.persistent_dao import (
+ ExtensionStorageDAO,
+ ExtensionStorageQuotaExceeded,
+)
+from superset.extensions.types import LoadedExtension
+from superset.extensions.utils import get_extensions
+from superset.utils import json
+from superset.utils.decorators import transaction
+
+# Key separator
+SEPARATOR = ":"
+
+# Key prefix for extension ephemeral state
+KEY_PREFIX = "superset-ext"
+
+
+def _build_cache_key(*parts: Any) -> str:
+ """Build a namespaced cache key from parts."""
+ return SEPARATOR.join(str(part) for part in parts)
+
+
+def _get_extension_or_404(extension_id: str) -> LoadedExtension | None:
+ """Get extension by ID or return None if not found."""
+ extensions = get_extensions()
+ return extensions.get(extension_id)
+
+
+def _parse_ttl(body: dict[str, Any]) -> tuple[int | None, str | None]:
+ """Parse and validate TTL from request body.
+
+ Returns:
+ (ttl, error_message) - error_message is set if the value is missing or invalid.
+ """
+ if "ttl" not in body:
+ return None, "Field 'ttl' is required"
+ try:
+ ttl = int(body["ttl"])
+ except (TypeError, ValueError):
+ return None, "Field 'ttl' must be a positive integer"
+ if ttl <= 0:
+ return None, "Field 'ttl' must be a positive integer"
+ max_ttl = current_app.config.get("EXTENSIONS_EPHEMERAL_STORAGE", {}).get("MAX_TTL")
+ if max_ttl is not None and ttl > max_ttl:
+ return None, f"Field 'ttl' must not exceed {max_ttl} seconds"
+ return ttl, None
+
+
+def _build_storage_key(extension_id: str, key: str, shared: bool) -> str:
+ """Build the cache key based on scope (user or shared)."""
+ if shared:
+ return _build_cache_key(KEY_PREFIX, extension_id, "shared", key)
+ user_id = g.user.id
+ return _build_cache_key(KEY_PREFIX, extension_id, "user", user_id, key)
Review Comment:
As these are super critical to the storage API, should we not move these to a separate `superset/extensions/storage/utils.py` and add a few simple parametrized tests to both document their typical usage patterns and protect against regressions?
##########
superset/extensions/storage/persistent_dao.py:
##########
@@ -0,0 +1,313 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from __future__ import annotations
+
+import hashlib
+import hmac
+import logging
+
+from flask import current_app
+from flask_babel import gettext as _
+from sqlalchemy import and_, func, LargeBinary
+
+from superset import db
+from superset.daos.base import BaseDAO
+from superset.exceptions import SupersetGenericErrorException
+from superset.extensions.storage.filters import ExtensionStorageFilter
+from superset.extensions.storage.persistent_model import ExtensionStorage
+from superset.utils.encrypt import (
+ DEFAULT_ENCRYPTION_ENGINE_NAME,
+ EncryptedType,
+ resolve_encryption_engine,
+)
+
+logger = logging.getLogger(__name__)
+
+
+class ExtensionStorageQuotaExceeded(SupersetGenericErrorException):
+ """Raised when a write would exceed an extension's persistent storage quota."""
+
+ status = 413
+
+ def __init__(self, extension_id: str, quota: int) -> None:
+ super().__init__(
+ message=_(
+ "Extension '%(extension_id)s' has exceeded its persistent "
+ "storage quota of %(quota)d bytes.",
+ extension_id=extension_id,
+ quota=quota,
+ ),
+ )
+
+
+def _derive_key(secret: str, user_fk: int) -> str:
+ """Derive a per-user encryption key from the master secret and user ID.
+
+ Uses HMAC-SHA256 so the derived key is cryptographically bound to both.
+ Ciphertext encrypted for one user cannot be decrypted as another.
+ """
+ return hmac.new(
+ secret.encode(),
+ str(user_fk).encode(),
+ hashlib.sha256,
+ ).hexdigest()
+
+
+def _enc_type(user_fk: int | None) -> EncryptedType:
+ """Return an EncryptedType for the given scope.
+
+ User-scoped rows use a key derived from SECRET_KEY and the user ID, so
+ ciphertext written for one user cannot be decrypted as another.
+ Shared rows (user_fk=None) use SECRET_KEY directly.
+ """
+ secret = current_app.config["SECRET_KEY"]
+ key = _derive_key(secret, user_fk) if user_fk is not None else secret
+ engine_name = current_app.config.get(
+ "SQLALCHEMY_ENCRYPTED_FIELD_ENGINE", DEFAULT_ENCRYPTION_ENGINE_NAME
+ )
+ return EncryptedType(
+ LargeBinary,
+ key=key,
+ engine=resolve_encryption_engine(engine_name),
+ )
+
+
+def _get_quota() -> int | None:
+ """Return the configured per-extension persistent storage quota in bytes.
+
+ Returns None when no quota is configured (unlimited).
+ """
+ config = current_app.config.get("EXTENSIONS_PERSISTENT_STORAGE", {})
Review Comment:
Nit: For keys in `config.py` we generally don't do `config.get()` but rather `config[]`.
##########
superset/extensions/context.py:
##########
@@ -16,75 +16,124 @@
# under the License.
"""
-Extension Context Management - provides ambient context during extension loading.
+Extension Context Management - provides ambient context for extensions.
-This module provides a thread-local context system that allows decorators to
-automatically detect whether they are being called in host or extension code
-during extension loading.
+This module provides a context system using Python's contextvars that allows
+extensions to access their context (metadata and scoped resources) via get_context().
+
+The context is set during extension loading and when extension callbacks are invoked.
+Uses ContextVar for thread-safe and async-safe context management with automatic
+save/restore for nested contexts.
"""
from __future__ import annotations
-import contextlib
-from threading import local
-from typing import Any, Generator
+from contextlib import contextmanager
+from contextvars import ContextVar
+from typing import Any, Iterator
from superset_core.extensions.types import Manifest
-# Thread-local storage for extension context
-_extension_context: local = local()
-
-class ExtensionContext:
- """Manages ambient extension context during loading."""
+class ExtensionStorage:
+ """Extension storage with all available tiers."""
- def __init__(self, manifest: Manifest):
- self.manifest = manifest
+ @property
+ def ephemeral(self) -> Any:
+ from superset.extensions.storage.ephemeral import EphemeralState
- def __enter__(self) -> "ExtensionContext":
- if getattr(_extension_context, "current", None) is not None:
- current_extension = _extension_context.current.manifest.id
- raise RuntimeError(
- f"Cannot initialize extension {self.manifest.id} while extension "
- f"{current_extension} is already being initialized. "
- f"Nested extension initialization is not supported."
- )
+ return EphemeralState
- _extension_context.current = self
- return self
+ @property
+ def persistent(self) -> Any:
+ from superset.extensions.storage.persistent import PersistentState
- def __exit__(self, exc_type: Any, exc_val: Any, exc_tb: Any) -> None:
- # Clear the current context
- _extension_context.current = None
+ return PersistentState
-class ExtensionContextWrapper:
- """Wrapper for extension context with extensible properties."""
+class ConcreteExtensionContext:
+ """Concrete implementation of ExtensionContext for the host."""
def __init__(self, manifest: Manifest):
self._manifest = manifest
+ self._storage = ExtensionStorage()
+
+ @property
+ def extension(self) -> Manifest:
+ """Extension metadata (new API)."""
+ return self._manifest
@property
def manifest(self) -> Manifest:
- """Get the extension manifest."""
+ """Extension manifest (for backward compatibility)."""
Review Comment:
Should we set a deprecation version for this field?
##########
superset/extensions/storage/persistent.py:
##########
@@ -0,0 +1,193 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""
+Host implementation for Persistent State (Tier 3 Storage).
+
+Provides the concrete database-backed implementation that is injected into
+superset_core.extensions.storage.persistent at startup.
+"""
+
+from __future__ import annotations
+
+from typing import Any, ClassVar
+
+from flask import g
+from superset_core.extensions.storage.persistent import (
+ PersistentSetOptions,
+ PersistentState as CorePersistentState,
+)
+
+from superset.extensions.context import get_current_extension_context
+from superset.extensions.storage.persistent_dao import ExtensionStorageDAO
+from superset.utils import json
+from superset.utils.decorators import transaction
+
+
+def _get_extension_id() -> str:
+ """Get the current extension ID from context."""
+ context = get_current_extension_context()
+ if context is None:
+ raise RuntimeError(
+ "persistent_state can only be used within an extension context. "
+ "Ensure this code is being executed during extension loading or "
+ "within an extension API request handler."
+ )
+ return context.manifest.id
+
+
+def _get_current_user_id() -> int:
+ """Get the current authenticated user's ID."""
+ user = getattr(g, "user", None)
Review Comment:
we already have `get_user()` in `superset/utils/core.py`, so we should reuse that at a minimum.
##########
superset/extensions/storage/persistent_model.py:
##########
@@ -0,0 +1,129 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import uuid as uuid_module
+
+from flask_appbuilder import Model
+from sqlalchemy import (
+ Boolean,
+ Column,
+ ForeignKey,
+ Index,
+ Integer,
+ LargeBinary,
+ String,
+ Text,
+ UniqueConstraint,
+)
+from sqlalchemy.orm import backref, relationship
+from sqlalchemy_utils import UUIDType
+from superset_core.extensions.storage.models import (
+ ExtensionStorageEntry as CoreExtensionStorageEntry,
+)
+
+from superset.models.helpers import AuditMixinNullable
+
+# 16 MB β matches the KeyValue store limit.
+EXTENSION_STORAGE_MAX_SIZE = 2**24 - 1
+
+
+class ExtensionStorage(CoreExtensionStorageEntry, AuditMixinNullable, Model):
+ """Generic persistent key-value storage for extensions (Tier 3).
+
+ Each row is identified by (extension_id, user_fk, resource_type,
+ resource_uuid, key):
+
+ * Global scope β user_fk IS NULL, resource_type IS NULL
+ * User scope β user_fk set, resource_type IS NULL
+ * Resource scope β resource_type + resource_uuid set (user_fk optional)
+
+ The payload is stored as raw bytes (value) with a MIME-type hint
+ (value_type). When is_encrypted is True the value has been encrypted
+ at the DAO layer using Fernet and must be decrypted before use.
+ """
+
+ __tablename__ = "extension_storage"
+
+ id = Column(Integer, primary_key=True, autoincrement=True)
+ uuid = Column(
+ UUIDType(binary=True),
+ default=uuid_module.uuid4,
+ unique=True,
+ nullable=False,
+ )
+
+ # Extension identity
+ extension_id = Column(String(255), nullable=False)
+
+ # Scope discriminators β all nullable; NULLs define the scope (see docstring)
+ user_fk = Column(
+ Integer,
+ ForeignKey(
+ "ab_user.id",
+ ondelete="SET NULL",
+ name="fk_extension_storage_user_fk_ab_user",
+ ),
+ nullable=True,
+ )
+ resource_type = Column(String(64), nullable=True)
+ resource_uuid = Column(String(36), nullable=True)
+
+ # Storage key within the scope
+ key = Column(String(255), nullable=False)
+
+ # Optional metadata
+ category = Column(String(64), nullable=True)
+ description = Column(Text, nullable=True)
+
+ # Payload
+ value = Column(LargeBinary(EXTENSION_STORAGE_MAX_SIZE), nullable=False)
+ value_type = Column(String(255), nullable=False, default="application/json")
+ is_encrypted = Column(Boolean, nullable=False, default=False)
+
+ user = relationship(
+ "User",
+ backref=backref("extension_storage_entries", cascade="all, delete-orphan"),
Review Comment:
As we have `ondelete="SET NULL"` on the foreign key definition, isn't `cascade="all, delete-orphan"` incorrect here?
##########
superset/extensions/storage/persistent.py:
##########
@@ -0,0 +1,193 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""
+Host implementation for Persistent State (Tier 3 Storage).
+
+Provides the concrete database-backed implementation that is injected into
+superset_core.extensions.storage.persistent at startup.
+"""
+
+from __future__ import annotations
+
+from typing import Any, ClassVar
+
+from flask import g
+from superset_core.extensions.storage.persistent import (
+ PersistentSetOptions,
+ PersistentState as CorePersistentState,
+)
+
+from superset.extensions.context import get_current_extension_context
+from superset.extensions.storage.persistent_dao import ExtensionStorageDAO
+from superset.utils import json
+from superset.utils.decorators import transaction
+
+
+def _get_extension_id() -> str:
+ """Get the current extension ID from context."""
+ context = get_current_extension_context()
+ if context is None:
+ raise RuntimeError(
+ "persistent_state can only be used within an extension context. "
+ "Ensure this code is being executed during extension loading or "
+ "within an extension API request handler."
+ )
+ return context.manifest.id
+
+
+def _get_current_user_id() -> int:
+ """Get the current authenticated user's ID."""
+ user = getattr(g, "user", None)
+ if user is None or not hasattr(user, "id"):
+ raise RuntimeError(
+ "persistent_state requires an authenticated user. "
+ "Ensure the request has been authenticated."
+ )
+ return user.id
+
+
+def _decode(raw: bytes | None) -> Any:
+ """Decode stored bytes back to a Python value."""
+ if raw is None:
+ return None
+ return json.loads(raw)
+
+
+def _encode(value: Any) -> bytes:
+ """Encode a Python value for database storage."""
+ return json.dumps(value).encode()
Review Comment:
I think it's probably fine to default to JSON, but I think it would be very valuable to make the Codec overridable as we do in KeyValue where we support multiple different formats (JSON, Pickle, Marshmallow).
##########
superset/extensions/context.py:
##########
@@ -16,75 +16,124 @@
# under the License.
"""
-Extension Context Management - provides ambient context during extension loading.
+Extension Context Management - provides ambient context for extensions.
-This module provides a thread-local context system that allows decorators to
-automatically detect whether they are being called in host or extension code
-during extension loading.
+This module provides a context system using Python's contextvars that allows
+extensions to access their context (metadata and scoped resources) via get_context().
+
+The context is set during extension loading and when extension callbacks are invoked.
+Uses ContextVar for thread-safe and async-safe context management with automatic
+save/restore for nested contexts.
"""
from __future__ import annotations
-import contextlib
-from threading import local
-from typing import Any, Generator
+from contextlib import contextmanager
+from contextvars import ContextVar
+from typing import Any, Iterator
from superset_core.extensions.types import Manifest
-# Thread-local storage for extension context
-_extension_context: local = local()
-
-class ExtensionContext:
- """Manages ambient extension context during loading."""
+class ExtensionStorage:
+ """Extension storage with all available tiers."""
- def __init__(self, manifest: Manifest):
- self.manifest = manifest
+ @property
+ def ephemeral(self) -> Any:
+ from superset.extensions.storage.ephemeral import EphemeralState
- def __enter__(self) -> "ExtensionContext":
- if getattr(_extension_context, "current", None) is not None:
- current_extension = _extension_context.current.manifest.id
- raise RuntimeError(
- f"Cannot initialize extension {self.manifest.id} while extension "
- f"{current_extension} is already being initialized. "
- f"Nested extension initialization is not supported."
- )
+ return EphemeralState
- _extension_context.current = self
- return self
+ @property
+ def persistent(self) -> Any:
+ from superset.extensions.storage.persistent import PersistentState
- def __exit__(self, exc_type: Any, exc_val: Any, exc_tb: Any) -> None:
- # Clear the current context
- _extension_context.current = None
+ return PersistentState
-class ExtensionContextWrapper:
- """Wrapper for extension context with extensible properties."""
+class ConcreteExtensionContext:
+ """Concrete implementation of ExtensionContext for the host."""
def __init__(self, manifest: Manifest):
self._manifest = manifest
+ self._storage = ExtensionStorage()
+
+ @property
+ def extension(self) -> Manifest:
+ """Extension metadata (new API)."""
Review Comment:
Let's not call out new APIs, but rather only call out deprecated ones:
```suggestion
"""Extension metadata."""
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(extensions): add Tier 1 and Tier 2 storage APIs [superset]
Posted by "villebro (via GitHub)" <gi...@apache.org>.
villebro commented on code in PR #39171:
URL: https://github.com/apache/superset/pull/39171#discussion_r3054643549
##########
superset/extensions/storage/api.py:
##########
@@ -0,0 +1,265 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""
+REST API for extension storage.
+
+Provides HTTP endpoints for frontend extensions to access server-side
+ephemeral storage without direct backend code.
+
+All operations are user-scoped by default. Use `?shared=true` query param
+to access shared state visible to all users.
+"""
+
+from __future__ import annotations
+
+from typing import Any
+
+from flask import g, request
+from flask.wrappers import Response
+from flask_appbuilder.api import BaseApi, expose, protect, safe
+
+from superset.extensions import cache_manager
+from superset.extensions.types import LoadedExtension
+from superset.extensions.utils import get_extensions
+
+# Key separator
+SEPARATOR = ":"
Review Comment:
I personally prefer to use `\n` for these separators whenever possible, as newlines tend to be very uncommon in any typical concatenated identifiers.
##########
docs/developer_docs/extensions/storage.md:
##########
@@ -0,0 +1,231 @@
+---
+title: Storage
+sidebar_position: 8
+---
+
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Storage
+
+Superset Extensions have access to a managed storage API for persisting data. The storage system provides multiple tiers with different persistence characteristics, allowing extensions to choose the right storage for their needs.
+
+Each extension receives its own isolated storage namespace. When Superset loads your extension, it binds storage to your extension's unique identifier, ensuring data privacyβtwo extensions using the same key will never collide, and extensions cannot access each other's data.
+
+## Storage Tiers
+
+| Tier | Storage Type | Context Property | Use Case |
+| ---- | ----------------- | ------------------------------------------ | ------------------------------------------------ |
+| 1 | Browser storage | `ctx.storage.local`, `ctx.storage.session` | UI state, wizard progress, draft forms |
+| 2 | Server-side cache | `ctx.storage.ephemeral` | Job progress, temporary results |
+| 3 | Database | `ctx.storage.persistent` | User preferences, extension config (coming soon) |
Review Comment:
Should we consider only adding this section in the follow-up PR to make sure things are consistent?
##########
superset/extensions/storage/api.py:
##########
@@ -0,0 +1,265 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""
+REST API for extension storage.
+
+Provides HTTP endpoints for frontend extensions to access server-side
+ephemeral storage without direct backend code.
+
+All operations are user-scoped by default. Use `?shared=true` query param
+to access shared state visible to all users.
+"""
+
+from __future__ import annotations
+
+from typing import Any
+
+from flask import g, request
+from flask.wrappers import Response
+from flask_appbuilder.api import BaseApi, expose, protect, safe
+
+from superset.extensions import cache_manager
+from superset.extensions.types import LoadedExtension
+from superset.extensions.utils import get_extensions
+
+# Key separator
+SEPARATOR = ":"
+
+# Key prefix for extension ephemeral state
+KEY_PREFIX = "superset-ext"
+
+# Default TTL: 1 hour
+DEFAULT_TTL = 3600
+
+
+def _build_cache_key(*parts: Any) -> str:
+ """Build a namespaced cache key from parts."""
+ return SEPARATOR.join(str(part) for part in parts)
+
+
+def _get_extension_or_404(extension_id: str) -> LoadedExtension | None:
+ """Get extension by ID or return None if not found."""
+ extensions = get_extensions()
+ return extensions.get(extension_id)
+
+
+def _parse_ttl(body: dict[str, Any]) -> tuple[int | None, str | None]:
+ """Parse and validate TTL from request body.
+
+ Returns:
+ (ttl, error_message) - ttl is the parsed value, error_message is set if invalid
+ """
+ try:
+ ttl = int(body.get("ttl", DEFAULT_TTL))
+ except (TypeError, ValueError):
+ return None, "Field 'ttl' must be a positive integer"
+ if ttl <= 0:
+ return None, "Field 'ttl' must be a positive integer"
+ return ttl, None
+
+
+def _build_storage_key(extension_id: str, key: str, shared: bool) -> str:
+ """Build the cache key based on scope (user or shared)."""
+ if shared:
+ return _build_cache_key(KEY_PREFIX, extension_id, "shared", key)
+ user_id = g.user.id
+ return _build_cache_key(KEY_PREFIX, extension_id, "user", user_id, key)
+
+
+class ExtensionStorageRestApi(BaseApi):
+ """REST API for extension ephemeral state storage."""
+
+ allow_browser_login = True
+ route_base = "/api/v1/extensions/storage"
+
+ def response(self, status_code: int, **kwargs: Any) -> Response:
+ """Helper method to create JSON responses."""
+ from flask import jsonify
+
+ return jsonify(kwargs), status_code
+
+ def response_404(self, message: str = "Not found") -> Response:
+ """Helper method to create 404 responses."""
+ from flask import jsonify
+
+ return jsonify({"message": message}), 404
+
+ def response_400(self, message: str) -> Response:
+ """Helper method to create 400 responses."""
+ from flask import jsonify
+
+ return jsonify({"message": message}), 400
+
+ @protect()
+ @safe
+ @expose("/ephemeral/<extension_id>/<key>", methods=("GET",))
Review Comment:
I think we need to reconsider our extension namespace. We currently have these:
```
GET /api/v1/extensions/ (list view)
GET /api/v1/extensions/_info
GET /api/v1/extensions/{publisher}/{name} (metadata)
GET /api/v1/extensions/{publisher}/{name}/{file} (for FE chunks)
```
I think we should probably rename the chunk endpoint, and move the storage API under the main resource, like so (using "chunk" as a placeholder here, there's probably a more elegant name for it):
```
GET /api/v1/extensions/ (list view)
GET /api/v1/extensions/_info
GET /api/v1/extensions/{publisher}/{name} (metadata)
GET /api/v1/extensions/{publisher}/{name}/chunks/{file}
GET /api/v1/extensions/{publisher}/{name}/storage/persistent/{key}
PUT /api/v1/extensions/{publisher}/{name}/storage/persistent/{key}
GET /api/v1/extensions/{publisher}/{name}/storage/ephemeral/{key}
PUT /api/v1/extensions/{publisher}/{name}/storage/ephemeral/{key}
```
Alternatively we just start using `/{extension_id}/` everywhere we currently have `/{publisher}/{name}/` - both are ok by me.
##########
docs/developer_docs/extensions/storage.md:
##########
@@ -0,0 +1,231 @@
+---
+title: Storage
+sidebar_position: 8
+---
+
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Storage
+
+Superset Extensions have access to a managed storage API for persisting data. The storage system provides multiple tiers with different persistence characteristics, allowing extensions to choose the right storage for their needs.
+
+Each extension receives its own isolated storage namespace. When Superset loads your extension, it binds storage to your extension's unique identifier, ensuring data privacyβtwo extensions using the same key will never collide, and extensions cannot access each other's data.
+
+## Storage Tiers
+
+| Tier | Storage Type | Context Property | Use Case |
+| ---- | ----------------- | ------------------------------------------ | ------------------------------------------------ |
+| 1 | Browser storage | `ctx.storage.local`, `ctx.storage.session` | UI state, wizard progress, draft forms |
+| 2 | Server-side cache | `ctx.storage.ephemeral` | Job progress, temporary results |
+| 3 | Database | `ctx.storage.persistent` | User preferences, extension config (coming soon) |
+
+## Tier 1: Local State
+
+Browser-based storage that persists on the user's device. Use this for UI state and settings that don't need to sync across devices.
+
+### Why Use the API Instead of localStorage Directly?
+
+You might wonder why extensions should use `ctx.storage.local` instead of directly accessing `window.localStorage`. The managed API provides several benefits:
+
+- **Automatic namespacing**: Each extension's data is isolated. Two extensions using the same key name won't collide.
+- **User isolation**: By default, data is scoped to the current user, preventing data leakage between users on shared devices.
+- **Clean uninstall**: When an extension is uninstalled, all its data can be cleanly removed using prefix-based deletion.
+- **Future sandboxing**: The async API is designed for a future sandboxed execution model where extensions run in isolated contexts without direct DOM access.
+- **Consistent patterns**: The same API shape works across all storage tiers, making it easy to switch between them.
+
+### localState
+
+Data persists across browser sessions until explicitly deleted or the user clears browser storage.
+
+```typescript
+import { getContext } from '@apache-superset/core/extensions';
+
+const ctx = getContext();
+
+// Save sidebar state
+await ctx.storage.local.set('sidebar_collapsed', true);
+
+// Retrieve it later
+const isCollapsed = await ctx.storage.local.get('sidebar_collapsed');
+
+// Remove it
+await ctx.storage.local.remove('sidebar_collapsed');
+```
+
+### sessionState
+
+Data is cleared when the browser tab is closed. Use for transient state within a single session.
+
+```typescript
+import { getContext } from '@apache-superset/core/extensions';
+
+const ctx = getContext();
+
+// Save wizard progress (lost when tab closes)
+await ctx.storage.session.set('wizard_step', 3);
+await ctx.storage.session.set('unsaved_form', { name: 'Draft' });
+
+// Retrieve on page reload within same tab
+const step = await ctx.storage.session.get('wizard_step');
+```
+
+### Shared State
+
+By default, data is scoped to the current user. Use `shared` for data that should be accessible to all users on the same device.
+
+```typescript
+import { getContext } from '@apache-superset/core/extensions';
+
+const ctx = getContext();
+
+// Shared across all users on this device
+await ctx.storage.local.shared.set('device_id', 'abc-123');
+const deviceId = await ctx.storage.local.shared.get('device_id');
+```
+
+### When to Use Tier 1
+
+- UI state (sidebar collapsed, panel sizes)
+- Recently used items
+- Draft form values
+- Any data acceptable to lose if user clears browser
+
+### Limitations
+
+- Per-browser, per-device (not shared across devices)
+- Subject to browser storage quotas (~5-10 MB)
+- Not accessible from backend code
+
+## Tier 2: Ephemeral State
+
+Server-side cache storage with automatic TTL expiration. Use for temporary data that needs to be shared between frontend and backend, or persist across page reloads.
+
+### Frontend Usage
+
+```typescript
+import { getContext } from '@apache-superset/core/extensions';
+
+const ctx = getContext();
+
+// Store with default TTL (1 hour)
+await ctx.storage.ephemeral.set('job_progress', { pct: 42, status: 'running' });
+
+// Store with custom TTL (5 minutes)
+await ctx.storage.ephemeral.set(
+ 'quick_cache',
+ { results: [1, 2, 3] },
+ { ttl: 300 },
+);
+
+// Retrieve
+const progress = await ctx.storage.ephemeral.get('job_progress');
+
+// Remove
+await ctx.storage.ephemeral.remove('job_progress');
+```
+
+### Backend Usage
+
+```python
+from superset_core.extensions.context import get_context
+
+ctx = get_context()
+
+# Store job progress
+ctx.storage.ephemeral.set('job_progress', {'pct': 42, 'status': 'running'}, ttl=3600)
+
+# Retrieve
+progress = ctx.storage.ephemeral.get('job_progress')
+
+# Remove
+ctx.storage.ephemeral.remove('job_progress')
+```
+
+### Shared State
+
+For data that needs to be visible to all users:
+
+```typescript
+import { getContext } from '@apache-superset/core/extensions';
+
+const ctx = getContext();
+
+await ctx.storage.ephemeral.shared.set('shared_result', { data: [1, 2, 3] });
+const result = await ctx.storage.ephemeral.shared.get('shared_result');
+```
+
+```python
+from superset_core.extensions.context import get_context
+
+ctx = get_context()
+
+ctx.storage.ephemeral.shared.set('shared_result', {'data': [1, 2, 3]})
+result = ctx.storage.ephemeral.shared.get('shared_result')
+```
+
+### When to Use Tier 2
+
+- Background job progress indicators
+- Cross-request intermediate state
+- Query result previews
+- Temporary computation results
+- Any data that can be recomputed if lost
+
+### Limitations
+
+- Not guaranteed to survive server restarts
+- Subject to cache eviction under memory pressure
+- TTL-based expiration (data disappears after timeout)
+
+## Tier 3: Persistent State
+
+Coming soon.
+
+## Key Patterns
+
+All storage keys are automatically namespaced:
+
+| Scope | Key Pattern |
+| ----------- | -------------------------------------------------- |
+| User-scoped | `superset-ext:{extension_id}:user:{user_id}:{key}` |
+| Shared | `superset-ext:{extension_id}:shared:{key}` |
+
+This ensures:
+
+- Extensions cannot accidentally access each other's data
+- Users cannot see other users' data (by default)
+- Clean prefix-based deletion on uninstall
+
+## Configuration
+
+Administrators can configure Tier 2 storage in `superset_config.py`:
+
+```python
+EXTENSIONS_STORAGE = {
+ "EPHEMERAL": {
+ # Use Redis for better performance in production
+ "CACHE_TYPE": "RedisCache",
+ "CACHE_REDIS_URL": "redis://localhost:6379/2",
+ "CACHE_DEFAULT_TIMEOUT": 3600, # 1 hour default TTL
+ },
+}
+```
+
+For development, the default `SupersetMetastoreCache` stores data in the metadata database.
Review Comment:
This is a good solution π
##########
superset-frontend/packages/superset-core/src/storage/ephemeralState.ts:
##########
@@ -0,0 +1,160 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import type { JsonValue, StorageAccessor } from './types';
+
+/**
+ * @fileoverview Ephemeral State API for Superset extensions (Tier 2 Storage).
+ *
+ * Provides short-lived KV storage backed by the configured server-side cache
+ * backend (Redis, Memcached, or filesystem). Automatically expires based on TTL.
+ * Not guaranteed to survive server restarts.
+ *
+ * By default, all operations are user-scoped (private to the current user).
+ * Use `shared` to access state that is visible to all users.
+ *
+ * Cache keys are namespaced automatically:
+ * - User-scoped (default): superset-ext:{extension_id}:user:{user_id}:{key}
+ * - Shared (global): superset-ext:{extension_id}:{key}
+ *
+ * @example
+ * ```typescript
+ * import { ephemeralState } from '@apache-superset/core/storage';
+ *
+ * // User-scoped state (default - private to current user)
+ * const progress = await ephemeralState.get('job_progress');
+ * await ephemeralState.set('job_progress', { pct: 42 }, { ttl: 3600 });
+ * await ephemeralState.remove('job_progress');
+ *
+ * // Shared state (explicit opt-in - visible to all users)
+ * const result = await ephemeralState.shared.get('shared_result');
+ * await ephemeralState.shared.set('shared_result', { data: [1, 2, 3] });
+ * await ephemeralState.shared.remove('shared_result');
+ * ```
+ */
+
+/**
+ * Default TTL in seconds (1 hour).
+ */
+export const DEFAULT_TTL = 3600;
Review Comment:
again, for better or for worse, but these have typically been defined in `config.py` to make them configurable. I would almost prefer to not have a default value at all, and make TTL a required parameter, than having this as a frontend defined fixed constant.
##########
docs/developer_docs/extensions/storage.md:
##########
@@ -0,0 +1,231 @@
+---
+title: Storage
+sidebar_position: 8
+---
+
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Storage
+
+Superset Extensions have access to a managed storage API for persisting data. The storage system provides multiple tiers with different persistence characteristics, allowing extensions to choose the right storage for their needs.
+
+Each extension receives its own isolated storage namespace. When Superset loads your extension, it binds storage to your extension's unique identifier, ensuring data privacyβtwo extensions using the same key will never collide, and extensions cannot access each other's data.
+
+## Storage Tiers
+
+| Tier | Storage Type | Context Property | Use Case |
+| ---- | ----------------- | ------------------------------------------ | ------------------------------------------------ |
+| 1 | Browser storage | `ctx.storage.local`, `ctx.storage.session` | UI state, wizard progress, draft forms |
+| 2 | Server-side cache | `ctx.storage.ephemeral` | Job progress, temporary results |
+| 3 | Database | `ctx.storage.persistent` | User preferences, extension config (coming soon) |
+
+## Tier 1: Local State
+
+Browser-based storage that persists on the user's device. Use this for UI state and settings that don't need to sync across devices.
+
+### Why Use the API Instead of localStorage Directly?
+
+You might wonder why extensions should use `ctx.storage.local` instead of directly accessing `window.localStorage`. The managed API provides several benefits:
+
+- **Automatic namespacing**: Each extension's data is isolated. Two extensions using the same key name won't collide.
+- **User isolation**: By default, data is scoped to the current user, preventing data leakage between users on shared devices.
+- **Clean uninstall**: When an extension is uninstalled, all its data can be cleanly removed using prefix-based deletion.
+- **Future sandboxing**: The async API is designed for a future sandboxed execution model where extensions run in isolated contexts without direct DOM access.
+- **Consistent patterns**: The same API shape works across all storage tiers, making it easy to switch between them.
+
+### localState
+
+Data persists across browser sessions until explicitly deleted or the user clears browser storage.
+
+```typescript
+import { getContext } from '@apache-superset/core/extensions';
+
+const ctx = getContext();
+
+// Save sidebar state
+await ctx.storage.local.set('sidebar_collapsed', true);
+
+// Retrieve it later
+const isCollapsed = await ctx.storage.local.get('sidebar_collapsed');
+
+// Remove it
+await ctx.storage.local.remove('sidebar_collapsed');
+```
+
+### sessionState
+
+Data is cleared when the browser tab is closed. Use for transient state within a single session.
+
+```typescript
+import { getContext } from '@apache-superset/core/extensions';
+
+const ctx = getContext();
+
+// Save wizard progress (lost when tab closes)
+await ctx.storage.session.set('wizard_step', 3);
+await ctx.storage.session.set('unsaved_form', { name: 'Draft' });
+
+// Retrieve on page reload within same tab
+const step = await ctx.storage.session.get('wizard_step');
+```
+
+### Shared State
+
+By default, data is scoped to the current user. Use `shared` for data that should be accessible to all users on the same device.
+
+```typescript
+import { getContext } from '@apache-superset/core/extensions';
+
+const ctx = getContext();
+
+// Shared across all users on this device
+await ctx.storage.local.shared.set('device_id', 'abc-123');
+const deviceId = await ctx.storage.local.shared.get('device_id');
+```
+
+### When to Use Tier 1
+
+- UI state (sidebar collapsed, panel sizes)
+- Recently used items
+- Draft form values
+- Any data acceptable to lose if user clears browser
+
+### Limitations
+
+- Per-browser, per-device (not shared across devices)
+- Subject to browser storage quotas (~5-10 MB)
+- Not accessible from backend code
+
+## Tier 2: Ephemeral State
+
+Server-side cache storage with automatic TTL expiration. Use for temporary data that needs to be shared between frontend and backend, or persist across page reloads.
+
+### Frontend Usage
+
+```typescript
+import { getContext } from '@apache-superset/core/extensions';
+
+const ctx = getContext();
+
+// Store with default TTL (1 hour)
+await ctx.storage.ephemeral.set('job_progress', { pct: 42, status: 'running' });
+
+// Store with custom TTL (5 minutes)
+await ctx.storage.ephemeral.set(
+ 'quick_cache',
+ { results: [1, 2, 3] },
+ { ttl: 300 },
+);
+
+// Retrieve
+const progress = await ctx.storage.ephemeral.get('job_progress');
+
+// Remove
+await ctx.storage.ephemeral.remove('job_progress');
+```
+
+### Backend Usage
+
+```python
+from superset_core.extensions.context import get_context
+
+ctx = get_context()
+
+# Store job progress
+ctx.storage.ephemeral.set('job_progress', {'pct': 42, 'status': 'running'}, ttl=3600)
+
+# Retrieve
+progress = ctx.storage.ephemeral.get('job_progress')
+
+# Remove
+ctx.storage.ephemeral.remove('job_progress')
+```
+
+### Shared State
+
+For data that needs to be visible to all users:
+
+```typescript
+import { getContext } from '@apache-superset/core/extensions';
+
+const ctx = getContext();
+
+await ctx.storage.ephemeral.shared.set('shared_result', { data: [1, 2, 3] });
+const result = await ctx.storage.ephemeral.shared.get('shared_result');
+```
+
+```python
+from superset_core.extensions.context import get_context
+
+ctx = get_context()
+
+ctx.storage.ephemeral.shared.set('shared_result', {'data': [1, 2, 3]})
+result = ctx.storage.ephemeral.shared.get('shared_result')
+```
+
+### When to Use Tier 2
+
+- Background job progress indicators
+- Cross-request intermediate state
+- Query result previews
+- Temporary computation results
+- Any data that can be recomputed if lost
+
+### Limitations
+
+- Not guaranteed to survive server restarts
+- Subject to cache eviction under memory pressure
+- TTL-based expiration (data disappears after timeout)
+
+## Tier 3: Persistent State
+
+Coming soon.
+
+## Key Patterns
+
+All storage keys are automatically namespaced:
+
+| Scope | Key Pattern |
+| ----------- | -------------------------------------------------- |
+| User-scoped | `superset-ext:{extension_id}:user:{user_id}:{key}` |
+| Shared | `superset-ext:{extension_id}:shared:{key}` |
+
+This ensures:
+
+- Extensions cannot accidentally access each other's data
+- Users cannot see other users' data (by default)
+- Clean prefix-based deletion on uninstall
+
+## Configuration
+
+Administrators can configure Tier 2 storage in `superset_config.py`:
+
+```python
+EXTENSIONS_STORAGE = {
+ "EPHEMERAL": {
+ # Use Redis for better performance in production
+ "CACHE_TYPE": "RedisCache",
+ "CACHE_REDIS_URL": "redis://localhost:6379/2",
+ "CACHE_DEFAULT_TIMEOUT": 3600, # 1 hour default TTL
+ },
+}
Review Comment:
Is there some reason we're using a literal string key? For better or worse, we define these backends as top level configs, in which case the typical solution would be
```python
EXTENSIONS_EPHEMERAL_STORAGE = {}
"CACHE_TYPE": ...
}
```
with a separate `EXTENSIONS_PERSISTENT_STORAGE` option later on...
##########
superset/extensions/storage/__init__.py:
##########
@@ -15,10 +15,19 @@
# specific language governing permissions and limitations
# under the License.
-from .get_database_info import get_database_info
-from .list_databases import list_databases
+"""
+Host implementations for extension storage APIs.
+
+This module provides concrete implementations that are injected into
+superset_core.extensions.storage at startup.
+"""
+
+from superset.extensions.storage.ephemeral_state import (
+ EphemeralStateImpl,
+ SharedEphemeralStateAccessor,
+)
__all__ = [
- "list_databases",
- "get_database_info",
+ "EphemeralStateImpl",
+ "SharedEphemeralStateAccessor",
]
Review Comment:
Did we decide not to use these? They're kind of a good security feature, but can be a pain to maintain..
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(extensions): add a Storage API for extensions [superset]
Posted by "michael-s-molina (via GitHub)" <gi...@apache.org>.
michael-s-molina commented on PR #39171:
URL: https://github.com/apache/superset/pull/39171#issuecomment-4770746595
@matebenyovszky Thank you for your fixes. I merged them into the branch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(extensions): add a Storage API for extensions [superset]
Posted by "bito-code-review[bot] (via GitHub)" <gi...@apache.org>.
bito-code-review[bot] commented on PR #39171:
URL: https://github.com/apache/superset/pull/39171#issuecomment-4771322319
**Bito Automatic Review Skipped - Draft PR**<br/><br/>Bito didn't auto-review because this pull request is in draft status.<br/>No action is needed if you didn't intend for the agent to review it. Otherwise, to manually trigger a review, type `/review` in a comment and save.<br/>You can change draft PR review settings [here](https://alpha.bito.ai/home/ai-agents/code-review-agent), or contact your Bito workspace admin at [evan@preset.io](mailto:evan@preset.io).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(extensions): add a Storage API for extensions [superset]
Posted by "michael-s-molina (via GitHub)" <gi...@apache.org>.
michael-s-molina commented on code in PR #39171:
URL: https://github.com/apache/superset/pull/39171#discussion_r3066489111
##########
superset/extensions/storage/api.py:
##########
@@ -0,0 +1,265 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""
+REST API for extension storage.
+
+Provides HTTP endpoints for frontend extensions to access server-side
+ephemeral storage without direct backend code.
+
+All operations are user-scoped by default. Use `?shared=true` query param
+to access shared state visible to all users.
+"""
+
+from __future__ import annotations
+
+from typing import Any
+
+from flask import g, request
+from flask.wrappers import Response
+from flask_appbuilder.api import BaseApi, expose, protect, safe
+
+from superset.extensions import cache_manager
+from superset.extensions.types import LoadedExtension
+from superset.extensions.utils import get_extensions
+
+# Key separator
+SEPARATOR = ":"
+
+# Key prefix for extension ephemeral state
+KEY_PREFIX = "superset-ext"
+
+# Default TTL: 1 hour
+DEFAULT_TTL = 3600
+
+
+def _build_cache_key(*parts: Any) -> str:
+ """Build a namespaced cache key from parts."""
+ return SEPARATOR.join(str(part) for part in parts)
+
+
+def _get_extension_or_404(extension_id: str) -> LoadedExtension | None:
+ """Get extension by ID or return None if not found."""
+ extensions = get_extensions()
+ return extensions.get(extension_id)
+
+
+def _parse_ttl(body: dict[str, Any]) -> tuple[int | None, str | None]:
+ """Parse and validate TTL from request body.
+
+ Returns:
+ (ttl, error_message) - ttl is the parsed value, error_message is set if invalid
+ """
+ try:
+ ttl = int(body.get("ttl", DEFAULT_TTL))
+ except (TypeError, ValueError):
+ return None, "Field 'ttl' must be a positive integer"
+ if ttl <= 0:
+ return None, "Field 'ttl' must be a positive integer"
+ return ttl, None
+
+
+def _build_storage_key(extension_id: str, key: str, shared: bool) -> str:
+ """Build the cache key based on scope (user or shared)."""
+ if shared:
+ return _build_cache_key(KEY_PREFIX, extension_id, "shared", key)
+ user_id = g.user.id
+ return _build_cache_key(KEY_PREFIX, extension_id, "user", user_id, key)
+
+
+class ExtensionStorageRestApi(BaseApi):
+ """REST API for extension ephemeral state storage."""
+
+ allow_browser_login = True
+ route_base = "/api/v1/extensions/storage"
+
+ def response(self, status_code: int, **kwargs: Any) -> Response:
+ """Helper method to create JSON responses."""
+ from flask import jsonify
+
+ return jsonify(kwargs), status_code
+
+ def response_404(self, message: str = "Not found") -> Response:
+ """Helper method to create 404 responses."""
+ from flask import jsonify
+
+ return jsonify({"message": message}), 404
+
+ def response_400(self, message: str) -> Response:
+ """Helper method to create 400 responses."""
+ from flask import jsonify
+
+ return jsonify({"message": message}), 400
+
+ @protect()
+ @safe
+ @expose("/ephemeral/<extension_id>/<key>", methods=("GET",))
Review Comment:
Addressed in [f08bcd4](https://github.com/apache/superset/pull/39171/commits/f08bcd4d09aa26a00a57c91e3a6d78db63494f4c)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(extensions): add a Storage API for extensions [superset]
Posted by "michael-s-molina (via GitHub)" <gi...@apache.org>.
michael-s-molina commented on code in PR #39171:
URL: https://github.com/apache/superset/pull/39171#discussion_r3560998563
##########
superset/extensions/context.py:
##########
@@ -16,75 +16,124 @@
# under the License.
"""
-Extension Context Management - provides ambient context during extension loading.
+Extension Context Management - provides ambient context for extensions.
-This module provides a thread-local context system that allows decorators to
-automatically detect whether they are being called in host or extension code
-during extension loading.
+This module provides a context system using Python's contextvars that allows
+extensions to access their context (metadata and scoped resources) via get_context().
+
+The context is set during extension loading and when extension callbacks are invoked.
+Uses ContextVar for thread-safe and async-safe context management with automatic
+save/restore for nested contexts.
"""
from __future__ import annotations
-import contextlib
-from threading import local
-from typing import Any, Generator
+from contextlib import contextmanager
+from contextvars import ContextVar
+from typing import Any, Iterator
from superset_core.extensions.types import Manifest
-# Thread-local storage for extension context
-_extension_context: local = local()
-
-class ExtensionContext:
- """Manages ambient extension context during loading."""
+class ExtensionStorage:
+ """Extension storage with all available tiers."""
- def __init__(self, manifest: Manifest):
- self.manifest = manifest
+ @property
+ def ephemeral(self) -> Any:
+ from superset.extensions.storage.ephemeral import EphemeralState
- def __enter__(self) -> "ExtensionContext":
- if getattr(_extension_context, "current", None) is not None:
- current_extension = _extension_context.current.manifest.id
- raise RuntimeError(
- f"Cannot initialize extension {self.manifest.id} while extension "
- f"{current_extension} is already being initialized. "
- f"Nested extension initialization is not supported."
- )
+ return EphemeralState
- _extension_context.current = self
- return self
+ @property
+ def persistent(self) -> Any:
+ from superset.extensions.storage.persistent import PersistentState
- def __exit__(self, exc_type: Any, exc_val: Any, exc_tb: Any) -> None:
- # Clear the current context
- _extension_context.current = None
+ return PersistentState
-class ExtensionContextWrapper:
- """Wrapper for extension context with extensible properties."""
+class ConcreteExtensionContext:
+ """Concrete implementation of ExtensionContext for the host."""
def __init__(self, manifest: Manifest):
self._manifest = manifest
+ self._storage = ExtensionStorage()
+
+ @property
+ def extension(self) -> Manifest:
+ """Extension metadata (new API)."""
+ return self._manifest
@property
def manifest(self) -> Manifest:
- """Get the extension manifest."""
+ """Extension manifest (for backward compatibility)."""
Review Comment:
I changed the API and kept only one. No backward compatibility needed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(extensions): add a Storage API for extensions [superset]
Posted by "codeant-ai-for-open-source[bot] (via GitHub)" <gi...@apache.org>.
codeant-ai-for-open-source[bot] commented on code in PR #39171:
URL: https://github.com/apache/superset/pull/39171#discussion_r3622522621
##########
superset/extensions/storage/api.py:
##########
@@ -0,0 +1,732 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""
+REST API for extension storage.
+
+Provides HTTP endpoints for frontend extensions to access server-side
+ephemeral storage without direct backend code.
+
+All operations are user-scoped by default. Use `?shared=true` query param
+to access shared state visible to all users.
+"""
+
+from __future__ import annotations
+
+import base64
+from typing import Any
+
+from flask import g, request
+from flask.wrappers import Response
+from flask_appbuilder.api import BaseApi, expose, protect, safe
+
+from superset.extensions.storage.codecs import DEFAULT_CODEC, get_codec, SAFE_CODECS
+from superset.extensions.storage.ephemeral_dao import (
+ ExtensionEphemeralDAO,
+ ExtensionEphemeralTTLInvalid,
+ ExtensionEphemeralValueTooLarge,
+)
+from superset.extensions.storage.persistent_dao import (
+ ExtensionStorageDAO,
+ ExtensionStorageKeyTooLong,
+ ExtensionStorageListPayloadTooLarge,
+ ExtensionStorageQuotaExceeded,
+ ExtensionStorageValueTooLarge,
+)
+from superset.extensions.storage.utils import get_extension_or_404, parse_ttl
+from superset.utils.decorators import transaction
+
+
+def _decoded_result_for_wire(decoded: Any) -> tuple[Any, bool]:
+ """Convert a codec-decoded value into its JSON wire representation.
+
+ JSON has no byte type, so a raw `bytes` value is base64-encoded to a
+ string for the response and flagged as such; every other value is
+ already JSON-representable as-is. Checked on the decoded value's
+ actual type, not the codec's name, so this keeps working for a codec
+ this module didn't define.
+
+ :returns: (wire_value, is_binary)
+ """
+ if isinstance(decoded, bytes):
+ return base64.b64encode(decoded).decode("ascii"), True
+ return decoded, False
+
+
+def _wire_value_for_request(value: Any, is_binary: bool) -> Any:
+ """Convert a request body's JSON `value` into a codec's input type.
+
+ `is_binary` is an explicit flag from the caller: JSON has no byte
+ type, so there is no way to tell, from the JSON `value` alone,
+ whether it is a base64 string that must be decoded to bytes before
+ being handed to the codec's `encode`, or a literal value to pass
+ through as-is. Only the caller knows which it sent.
+
+ :raises ValueError: if `is_binary` is set and `value` is not a valid
+ base64 string.
+ """
+ if is_binary:
+ return base64.b64decode(value, validate=True)
+ return value
+
+
+class ExtensionStorageRestApi(BaseApi):
+ """REST API for extension ephemeral state storage."""
+
+ allow_browser_login = True
+ route_base = "/api/v1/extensions"
+
+ def response(self, status_code: int, **kwargs: Any) -> Response:
+ """Helper method to create JSON responses."""
+ from flask import jsonify
+
+ return jsonify(kwargs), status_code
+
+ def response_404(self, message: str = "Not found") -> Response:
+ """Helper method to create 404 responses."""
+ from flask import jsonify
+
+ return jsonify({"message": message}), 404
+
+ def response_400(self, message: str) -> Response:
+ """Helper method to create 400 responses."""
+ from flask import jsonify
+
+ return jsonify({"message": message}), 400
+
+ @protect()
+ @safe
+ @expose("/<publisher>/<name>/storage/ephemeral/<key>", methods=("GET",))
+ def get_ephemeral(
+ self, publisher: str, name: str, key: str, **kwargs: Any
+ ) -> Response:
+ """Get a value from ephemeral state.
+ ---
+ get:
+ summary: Get a value from ephemeral state
+ parameters:
+ - in: path
+ name: publisher
+ schema:
+ type: string
+ required: true
+ description: Extension publisher
+ - in: path
+ name: name
+ schema:
+ type: string
+ required: true
+ description: Extension name
+ - in: path
+ name: key
+ schema:
+ type: string
+ required: true
+ description: Storage key
+ - in: query
+ name: shared
+ schema:
+ type: boolean
+ required: false
+ description: If true, read from shared state visible to all users
+ responses:
+ 200:
+ description: Value retrieved successfully
+ content:
+ application/json:
+ schema:
+ type: object
+ properties:
+ result:
+ description: The stored value. When 'isBinary' is
+ true, this is a base64 string that must be
+ decoded to get the actual value.
+ codec:
+ type: string
+ description: Name of the codec 'result' was encoded
+ with, e.g. "json" (default) or "binary"
+ isBinary:
+ type: boolean
+ description: Whether the stored value is binary
+ data
+ 400:
+ description: Value was stored with a codec unavailable over the API
+ 404:
+ description: Extension not found
+ """
+ extension_id = f"{publisher}.{name}"
+ extension = get_extension_or_404(extension_id)
+ if not extension:
+ return self.response_404("Extension not found")
+
+ shared = request.args.get("shared", "false").lower() == "true"
+ raw = ExtensionEphemeralDAO.get_raw(extension_id, key, shared=shared)
+ if raw is None:
+ return self.response(200, result=None)
+ value, codec = raw
+ if codec not in SAFE_CODECS:
+ return self.response_400(
+ f"Value was stored with codec '{codec}', which cannot be "
+ "read over the REST API."
+ )
+
+ result, is_binary = _decoded_result_for_wire(get_codec(codec).decode(value))
+ return self.response(200, result=result, codec=codec, isBinary=is_binary)
+
+ @protect()
+ @safe
+ @expose("/<publisher>/<name>/storage/ephemeral/<key>", methods=("PUT",))
+ def set_ephemeral(
+ self, publisher: str, name: str, key: str, **kwargs: Any
+ ) -> Response:
+ """Set a value in ephemeral state.
+ ---
+ put:
+ summary: Set a value in ephemeral state
+ parameters:
+ - in: path
+ name: publisher
+ schema:
+ type: string
+ required: true
+ description: Extension publisher
+ - in: path
+ name: name
+ schema:
+ type: string
+ required: true
+ description: Extension name
+ - in: path
+ name: key
+ schema:
+ type: string
+ required: true
+ description: Storage key
+ - in: query
+ name: shared
+ schema:
+ type: boolean
+ required: false
+ description: If true, store as shared state visible to all users
+ requestBody:
+ required: true
+ content:
+ application/json:
+ schema:
+ type: object
+ required:
+ - value
+ - ttl
+ properties:
+ value:
+ description: The value to store (must not exceed
+ MAX_VALUE_SIZE bytes once encoded with 'codec').
+ When 'isBinary' is true, this must be a base64
+ string, decoded before being handed to 'codec'.
+ codec:
+ type: string
+ description: Name of the codec used to encode 'value',
+ e.g. "json" (default). Must be one of the codecs
+ allowed over the REST API.
+ isBinary:
+ type: boolean
+ description: Whether 'value' is binary data
+ ttl:
+ type: integer
+ description: Time-to-live in seconds (must be a positive
+ integer not exceeding MAX_TTL)
+ responses:
+ 200:
+ description: Value stored successfully
+ 400:
+ description: Invalid request body, or codec not allowed over the API
+ 404:
+ description: Extension not found
+ """
+ extension_id = f"{publisher}.{name}"
+ extension = get_extension_or_404(extension_id)
+ if not extension:
+ return self.response_404("Extension not found")
+
+ body = request.get_json(silent=True) or {}
+ if "value" not in body:
+ return self.response_400("Request body must contain 'value' field")
+
+ codec = body.get("codec", DEFAULT_CODEC)
+ if codec not in SAFE_CODECS:
+ return self.response_400(
+ f"Codec '{codec}' is not allowed over the REST API."
+ )
Review Comment:
**Suggestion:** If a client sends a non-hashable JSON value for `codec` (for example, an array), the membership check against `SAFE_CODECS` raises `TypeError` and returns a 500 instead of a controlled 400 response; validate that `codec` is a string before checking membership. [type error]
<details>
<summary><b>Severity Level:</b> Major β οΈ</summary>
```mdx
- β Ephemeral storage write can crash on malformed codec.
- β οΈ Harder to debug invalid codec inputs from clients.
```
</details>
<details>
<summary><b>Steps of Reproduction β
</b></summary>
```mdx
1. Start Superset with this PR so `ExtensionStorageRestApi`
(superset/extensions/storage/api.py:88) is registered.
2. Authenticate as a user and issue `PUT
/api/v1/extensions/test_publisher/test_name/storage/ephemeral/test_key` (route defined by
`set_ephemeral` at lines 193β195).
3. Send JSON body `{"value": "test", "codec": ["json"], "ttl": 60}` so
`request.get_json()` (line 265) yields a list for `codec` at line 269.
4. At line 270, `if codec not in SAFE_CODECS:` executes; with `SAFE_CODECS` defined as a
set of codec names in superset/extensions/storage/codecs.py, Python raises `TypeError:
unhashable type: 'list'`, which is not caught and surfaces as an HTTP 500 instead of a 400
input-validation error.
```
</details>
[](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=6b81ca8f0f544acf92ae18224564c86f&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=6b81ca8f0f544acf92ae18224564c86f&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset)
*(Use Cmd/Ctrl + Click for best experience)*
<details>
<summary><b>Prompt for AI Agent π€ </b></summary>
```mdx
This is a comment left during a code review.
**Path:** superset/extensions/storage/api.py
**Line:** 269:273
**Comment:**
*Type Error: If a client sends a non-hashable JSON value for `codec` (for example, an array), the membership check against `SAFE_CODECS` raises `TypeError` and returns a 500 instead of a controlled 400 response; validate that `codec` is a string before checking membership.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F39171&comment_hash=2d0199bd3d38d2f1bb745576f6628b331e5e0d2a2bfeadfbdc635b5fb3baf1f8&reaction=like'>π</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F39171&comment_hash=2d0199bd3d38d2f1bb745576f6628b331e5e0d2a2bfeadfbdc635b5fb3baf1f8&reaction=dislike'>π</a>
##########
superset/extensions/storage/api.py:
##########
@@ -0,0 +1,732 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""
+REST API for extension storage.
+
+Provides HTTP endpoints for frontend extensions to access server-side
+ephemeral storage without direct backend code.
+
+All operations are user-scoped by default. Use `?shared=true` query param
+to access shared state visible to all users.
+"""
+
+from __future__ import annotations
+
+import base64
+from typing import Any
+
+from flask import g, request
+from flask.wrappers import Response
+from flask_appbuilder.api import BaseApi, expose, protect, safe
+
+from superset.extensions.storage.codecs import DEFAULT_CODEC, get_codec, SAFE_CODECS
+from superset.extensions.storage.ephemeral_dao import (
+ ExtensionEphemeralDAO,
+ ExtensionEphemeralTTLInvalid,
+ ExtensionEphemeralValueTooLarge,
+)
+from superset.extensions.storage.persistent_dao import (
+ ExtensionStorageDAO,
+ ExtensionStorageKeyTooLong,
+ ExtensionStorageListPayloadTooLarge,
+ ExtensionStorageQuotaExceeded,
+ ExtensionStorageValueTooLarge,
+)
+from superset.extensions.storage.utils import get_extension_or_404, parse_ttl
+from superset.utils.decorators import transaction
+
+
+def _decoded_result_for_wire(decoded: Any) -> tuple[Any, bool]:
+ """Convert a codec-decoded value into its JSON wire representation.
+
+ JSON has no byte type, so a raw `bytes` value is base64-encoded to a
+ string for the response and flagged as such; every other value is
+ already JSON-representable as-is. Checked on the decoded value's
+ actual type, not the codec's name, so this keeps working for a codec
+ this module didn't define.
+
+ :returns: (wire_value, is_binary)
+ """
+ if isinstance(decoded, bytes):
+ return base64.b64encode(decoded).decode("ascii"), True
+ return decoded, False
+
+
+def _wire_value_for_request(value: Any, is_binary: bool) -> Any:
+ """Convert a request body's JSON `value` into a codec's input type.
+
+ `is_binary` is an explicit flag from the caller: JSON has no byte
+ type, so there is no way to tell, from the JSON `value` alone,
+ whether it is a base64 string that must be decoded to bytes before
+ being handed to the codec's `encode`, or a literal value to pass
+ through as-is. Only the caller knows which it sent.
+
+ :raises ValueError: if `is_binary` is set and `value` is not a valid
+ base64 string.
+ """
+ if is_binary:
+ return base64.b64decode(value, validate=True)
+ return value
+
+
+class ExtensionStorageRestApi(BaseApi):
+ """REST API for extension ephemeral state storage."""
+
Review Comment:
**Suggestion:** The class docstring says this API is only for ephemeral storage, but the class also implements persistent storage endpoints, which makes the contract misleading for maintainers and API consumers; update the docstring to reflect both storage tiers. [docstring mismatch]
<details>
<summary><b>Severity Level:</b> Minor π§Ή</summary>
```mdx
- β οΈ Docstring misleads about API supporting persistent storage.
- β οΈ Maintainers may misunderstand class scope and responsibilities.
```
</details>
<details>
<summary><b>Steps of Reproduction β
</b></summary>
```mdx
1. Open superset/extensions/storage/api.py with this PR applied.
2. Locate `class ExtensionStorageRestApi(BaseApi):` at line 88 and read its docstring at
line 89: "REST API for extension ephemeral state storage."
3. Scroll down to see persistent storage endpoints implemented in the same class:
`list_persistent` (lines 350β481), `get_persistent` (486β565), `set_persistent` (566β679),
and `delete_persistent` (682β732).
4. Observe that the class docstring describes only ephemeral state, which misrepresents
the actual responsibilities and can mislead maintainers or API consumers about the scope
of this REST API.
```
</details>
[](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=786e5242c45b4202acde89c93771fef4&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=786e5242c45b4202acde89c93771fef4&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset)
*(Use Cmd/Ctrl + Click for best experience)*
<details>
<summary><b>Prompt for AI Agent π€ </b></summary>
```mdx
This is a comment left during a code review.
**Path:** superset/extensions/storage/api.py
**Line:** 88:89
**Comment:**
*Docstring Mismatch: The class docstring says this API is only for ephemeral storage, but the class also implements persistent storage endpoints, which makes the contract misleading for maintainers and API consumers; update the docstring to reflect both storage tiers.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F39171&comment_hash=16b29039ac4d7e9ae311aec5a70f295ac573488af2cc42c14e37332195044ba4&reaction=like'>π</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F39171&comment_hash=16b29039ac4d7e9ae311aec5a70f295ac573488af2cc42c14e37332195044ba4&reaction=dislike'>π</a>
##########
superset/extensions/storage/api.py:
##########
@@ -0,0 +1,732 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""
+REST API for extension storage.
+
+Provides HTTP endpoints for frontend extensions to access server-side
+ephemeral storage without direct backend code.
+
+All operations are user-scoped by default. Use `?shared=true` query param
+to access shared state visible to all users.
+"""
+
+from __future__ import annotations
+
+import base64
+from typing import Any
+
+from flask import g, request
+from flask.wrappers import Response
+from flask_appbuilder.api import BaseApi, expose, protect, safe
+
+from superset.extensions.storage.codecs import DEFAULT_CODEC, get_codec, SAFE_CODECS
+from superset.extensions.storage.ephemeral_dao import (
+ ExtensionEphemeralDAO,
+ ExtensionEphemeralTTLInvalid,
+ ExtensionEphemeralValueTooLarge,
+)
+from superset.extensions.storage.persistent_dao import (
+ ExtensionStorageDAO,
+ ExtensionStorageKeyTooLong,
+ ExtensionStorageListPayloadTooLarge,
+ ExtensionStorageQuotaExceeded,
+ ExtensionStorageValueTooLarge,
+)
+from superset.extensions.storage.utils import get_extension_or_404, parse_ttl
+from superset.utils.decorators import transaction
+
+
+def _decoded_result_for_wire(decoded: Any) -> tuple[Any, bool]:
+ """Convert a codec-decoded value into its JSON wire representation.
+
+ JSON has no byte type, so a raw `bytes` value is base64-encoded to a
+ string for the response and flagged as such; every other value is
+ already JSON-representable as-is. Checked on the decoded value's
+ actual type, not the codec's name, so this keeps working for a codec
+ this module didn't define.
+
+ :returns: (wire_value, is_binary)
+ """
+ if isinstance(decoded, bytes):
+ return base64.b64encode(decoded).decode("ascii"), True
+ return decoded, False
+
+
+def _wire_value_for_request(value: Any, is_binary: bool) -> Any:
+ """Convert a request body's JSON `value` into a codec's input type.
+
+ `is_binary` is an explicit flag from the caller: JSON has no byte
+ type, so there is no way to tell, from the JSON `value` alone,
+ whether it is a base64 string that must be decoded to bytes before
+ being handed to the codec's `encode`, or a literal value to pass
+ through as-is. Only the caller knows which it sent.
+
+ :raises ValueError: if `is_binary` is set and `value` is not a valid
+ base64 string.
+ """
+ if is_binary:
+ return base64.b64decode(value, validate=True)
+ return value
+
+
+class ExtensionStorageRestApi(BaseApi):
+ """REST API for extension ephemeral state storage."""
+
+ allow_browser_login = True
+ route_base = "/api/v1/extensions"
+
+ def response(self, status_code: int, **kwargs: Any) -> Response:
+ """Helper method to create JSON responses."""
+ from flask import jsonify
+
+ return jsonify(kwargs), status_code
+
+ def response_404(self, message: str = "Not found") -> Response:
+ """Helper method to create 404 responses."""
+ from flask import jsonify
+
+ return jsonify({"message": message}), 404
+
+ def response_400(self, message: str) -> Response:
+ """Helper method to create 400 responses."""
+ from flask import jsonify
+
+ return jsonify({"message": message}), 400
+
+ @protect()
+ @safe
+ @expose("/<publisher>/<name>/storage/ephemeral/<key>", methods=("GET",))
+ def get_ephemeral(
+ self, publisher: str, name: str, key: str, **kwargs: Any
+ ) -> Response:
+ """Get a value from ephemeral state.
+ ---
+ get:
+ summary: Get a value from ephemeral state
+ parameters:
+ - in: path
+ name: publisher
+ schema:
+ type: string
+ required: true
+ description: Extension publisher
+ - in: path
+ name: name
+ schema:
+ type: string
+ required: true
+ description: Extension name
+ - in: path
+ name: key
+ schema:
+ type: string
+ required: true
+ description: Storage key
+ - in: query
+ name: shared
+ schema:
+ type: boolean
+ required: false
+ description: If true, read from shared state visible to all users
+ responses:
+ 200:
+ description: Value retrieved successfully
+ content:
+ application/json:
+ schema:
+ type: object
+ properties:
+ result:
+ description: The stored value. When 'isBinary' is
+ true, this is a base64 string that must be
+ decoded to get the actual value.
+ codec:
+ type: string
+ description: Name of the codec 'result' was encoded
+ with, e.g. "json" (default) or "binary"
+ isBinary:
+ type: boolean
+ description: Whether the stored value is binary
+ data
+ 400:
+ description: Value was stored with a codec unavailable over the API
+ 404:
+ description: Extension not found
+ """
+ extension_id = f"{publisher}.{name}"
+ extension = get_extension_or_404(extension_id)
+ if not extension:
+ return self.response_404("Extension not found")
+
+ shared = request.args.get("shared", "false").lower() == "true"
+ raw = ExtensionEphemeralDAO.get_raw(extension_id, key, shared=shared)
+ if raw is None:
+ return self.response(200, result=None)
+ value, codec = raw
+ if codec not in SAFE_CODECS:
+ return self.response_400(
+ f"Value was stored with codec '{codec}', which cannot be "
+ "read over the REST API."
+ )
+
+ result, is_binary = _decoded_result_for_wire(get_codec(codec).decode(value))
+ return self.response(200, result=result, codec=codec, isBinary=is_binary)
+
+ @protect()
+ @safe
+ @expose("/<publisher>/<name>/storage/ephemeral/<key>", methods=("PUT",))
+ def set_ephemeral(
+ self, publisher: str, name: str, key: str, **kwargs: Any
+ ) -> Response:
+ """Set a value in ephemeral state.
+ ---
+ put:
+ summary: Set a value in ephemeral state
+ parameters:
+ - in: path
+ name: publisher
+ schema:
+ type: string
+ required: true
+ description: Extension publisher
+ - in: path
+ name: name
+ schema:
+ type: string
+ required: true
+ description: Extension name
+ - in: path
+ name: key
+ schema:
+ type: string
+ required: true
+ description: Storage key
+ - in: query
+ name: shared
+ schema:
+ type: boolean
+ required: false
+ description: If true, store as shared state visible to all users
+ requestBody:
+ required: true
+ content:
+ application/json:
+ schema:
+ type: object
+ required:
+ - value
+ - ttl
+ properties:
+ value:
+ description: The value to store (must not exceed
+ MAX_VALUE_SIZE bytes once encoded with 'codec').
+ When 'isBinary' is true, this must be a base64
+ string, decoded before being handed to 'codec'.
+ codec:
+ type: string
+ description: Name of the codec used to encode 'value',
+ e.g. "json" (default). Must be one of the codecs
+ allowed over the REST API.
+ isBinary:
+ type: boolean
+ description: Whether 'value' is binary data
+ ttl:
+ type: integer
+ description: Time-to-live in seconds (must be a positive
+ integer not exceeding MAX_TTL)
+ responses:
+ 200:
+ description: Value stored successfully
+ 400:
+ description: Invalid request body, or codec not allowed over the API
+ 404:
+ description: Extension not found
+ """
+ extension_id = f"{publisher}.{name}"
+ extension = get_extension_or_404(extension_id)
+ if not extension:
+ return self.response_404("Extension not found")
+
+ body = request.get_json(silent=True) or {}
+ if "value" not in body:
+ return self.response_400("Request body must contain 'value' field")
+
+ codec = body.get("codec", DEFAULT_CODEC)
+ if codec not in SAFE_CODECS:
+ return self.response_400(
+ f"Codec '{codec}' is not allowed over the REST API."
+ )
+
+ is_binary = bool(body.get("isBinary", False))
+ try:
+ value = _wire_value_for_request(body["value"], is_binary)
+ except (ValueError, TypeError):
+ return self.response_400(
+ "Value must be a valid base64 string when 'isBinary' is true."
+ )
+ ttl, error = parse_ttl(body)
+ if error:
+ return self.response_400(error)
+
+ shared = request.args.get("shared", "false").lower() == "true"
+ try:
+ ExtensionEphemeralDAO.set(
+ extension_id, key, value, ttl, codec=codec, shared=shared
+ )
+ except (ExtensionEphemeralTTLInvalid, ExtensionEphemeralValueTooLarge) as ex:
+ return self.response_400(ex.message)
+
+ return self.response(200, message="Value stored successfully")
+
+ @protect()
+ @safe
+ @expose("/<publisher>/<name>/storage/ephemeral/<key>", methods=("DELETE",))
+ def delete_ephemeral(
+ self, publisher: str, name: str, key: str, **kwargs: Any
+ ) -> Response:
+ """Delete a value from ephemeral state.
+ ---
+ delete:
+ summary: Delete a value from ephemeral state
+ parameters:
+ - in: path
+ name: publisher
+ schema:
+ type: string
+ required: true
+ description: Extension publisher
+ - in: path
+ name: name
+ schema:
+ type: string
+ required: true
+ description: Extension name
+ - in: path
+ name: key
+ schema:
+ type: string
+ required: true
+ description: Storage key
+ - in: query
+ name: shared
+ schema:
+ type: boolean
+ required: false
+ description: If true, delete from shared state
+ responses:
+ 200:
+ description: Value deleted successfully
+ 404:
+ description: Extension not found
+ """
+ extension_id = f"{publisher}.{name}"
+ extension = get_extension_or_404(extension_id)
+ if not extension:
+ return self.response_404("Extension not found")
+
+ shared = request.args.get("shared", "false").lower() == "true"
+ ExtensionEphemeralDAO.delete(extension_id, key, shared=shared)
+
+ return self.response(200, message="Value deleted successfully")
+
+ @protect()
+ @safe
+ @expose("/<publisher>/<name>/storage/persistent", methods=("GET",))
+ def list_persistent(self, publisher: str, name: str, **kwargs: Any) -> Response:
+ """List entries in persistent state.
+ ---
+ get:
+ summary: List entries in persistent state
+ parameters:
+ - in: path
+ name: publisher
+ schema:
+ type: string
+ required: true
+ description: Extension publisher
+ - in: path
+ name: name
+ schema:
+ type: string
+ required: true
+ description: Extension name
+ - in: query
+ name: shared
+ schema:
+ type: boolean
+ required: false
+ description: If true, list shared state visible to all users
+ - in: query
+ name: resource_type
+ schema:
+ type: string
+ required: false
+ description: Filter by resource type
+ - in: query
+ name: resource_uuid
+ schema:
+ type: string
+ required: false
+ description: Filter by resource UUID (requires resource_type)
+ - in: query
+ name: page
+ schema:
+ type: integer
+ required: false
+ description: Zero-indexed page number. Defaults to 0.
+ - in: query
+ name: page_size
+ schema:
+ type: integer
+ required: false
+ description: Number of entries per page. Defaults to 10. There
+ is no fixed ceiling, but a page whose combined value size
+ exceeds MAX_LIST_PAYLOAD_SIZE is rejected β reduce page_size
+ and retry if that happens.
+ responses:
+ 200:
+ description: Entries retrieved successfully
+ content:
+ application/json:
+ schema:
+ type: object
+ properties:
+ result:
+ type: array
+ items:
+ type: object
+ properties:
+ key:
+ type: string
+ value:
+ description: The stored value, or null if its
+ codec cannot be read over the REST API.
+ When 'isBinary' is true, this is a base64
+ string that must be decoded to get the
+ actual value.
+ codec:
+ type: string
+ isBinary:
+ type: boolean
+ description: Whether the stored value is
+ binary data
+ count:
+ type: integer
+ description: Total number of entries matching the
+ given scope/filters, across all pages
+ 400:
+ description: The requested page's combined value size exceeds
+ MAX_LIST_PAYLOAD_SIZE
+ 404:
+ description: Extension not found
+ """
+ extension_id = f"{publisher}.{name}"
+ extension = get_extension_or_404(extension_id)
+ if not extension:
+ return self.response_404("Extension not found")
+
+ shared = request.args.get("shared", "false").lower() == "true"
+ user_fk = None if shared else g.user.id
+ resource_type = request.args.get("resource_type")
+ resource_uuid = request.args.get("resource_uuid")
+ try:
+ page = int(request.args.get("page", 0))
+ page_size = int(request.args.get("page_size", 10))
+ except (TypeError, ValueError):
+ return self.response_400("'page' and 'page_size' must be integers")
Review Comment:
**Suggestion:** `page` and `page_size` are only parsed as integers but not validated for non-negative/positive bounds, so negative values can propagate to DAO pagination and cause backend query errors or incorrect paging behavior; enforce valid ranges before calling the DAO. [logic error]
<details>
<summary><b>Severity Level:</b> Major β οΈ</summary>
```mdx
- β Negative paging can produce incorrect or failing queries.
- β οΈ Malicious clients can stress storage listing behavior.
```
</details>
<details>
<summary><b>Steps of Reproduction β
</b></summary>
```mdx
1. Start Superset with this PR so `list_persistent` is exposed
(superset/extensions/storage/api.py:350β481).
2. Authenticate and call `GET
/api/v1/extensions/test_publisher/test_name/storage/persistent?page=-1&page_size=-5`
(route at lines 349β350).
3. In `list_persistent`, `page` and `page_size` are parsed as integers at lines 448β449;
the `except` block at line 450 only rejects non-integer input, so negative values are
accepted.
4. These negative `page` and `page_size` values are passed unchanged to
`ExtensionStorageDAO.list_entries` (lines 455β461 in superset/extensions/storage/api.py),
where they can cause incorrect offsets, unexpected query behavior, or backend errors
depending on the DAOβs pagination logic in superset/extensions/storage/persistent_dao.py.
```
</details>
[](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=5ab2f5954f4e44a99568ce9dd6659607&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=5ab2f5954f4e44a99568ce9dd6659607&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset)
*(Use Cmd/Ctrl + Click for best experience)*
<details>
<summary><b>Prompt for AI Agent π€ </b></summary>
```mdx
This is a comment left during a code review.
**Path:** superset/extensions/storage/api.py
**Line:** 448:451
**Comment:**
*Logic Error: `page` and `page_size` are only parsed as integers but not validated for non-negative/positive bounds, so negative values can propagate to DAO pagination and cause backend query errors or incorrect paging behavior; enforce valid ranges before calling the DAO.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F39171&comment_hash=1e43d6468272a07d4a4f98c9bacdd1593bc7cb68961e9cc41e2258837175e6fd&reaction=like'>π</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F39171&comment_hash=1e43d6468272a07d4a4f98c9bacdd1593bc7cb68961e9cc41e2258837175e6fd&reaction=dislike'>π</a>
##########
superset/extensions/storage/api.py:
##########
@@ -0,0 +1,732 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""
+REST API for extension storage.
+
+Provides HTTP endpoints for frontend extensions to access server-side
+ephemeral storage without direct backend code.
+
+All operations are user-scoped by default. Use `?shared=true` query param
+to access shared state visible to all users.
+"""
+
+from __future__ import annotations
+
+import base64
+from typing import Any
+
+from flask import g, request
+from flask.wrappers import Response
+from flask_appbuilder.api import BaseApi, expose, protect, safe
+
+from superset.extensions.storage.codecs import DEFAULT_CODEC, get_codec, SAFE_CODECS
+from superset.extensions.storage.ephemeral_dao import (
+ ExtensionEphemeralDAO,
+ ExtensionEphemeralTTLInvalid,
+ ExtensionEphemeralValueTooLarge,
+)
+from superset.extensions.storage.persistent_dao import (
+ ExtensionStorageDAO,
+ ExtensionStorageKeyTooLong,
+ ExtensionStorageListPayloadTooLarge,
+ ExtensionStorageQuotaExceeded,
+ ExtensionStorageValueTooLarge,
+)
+from superset.extensions.storage.utils import get_extension_or_404, parse_ttl
+from superset.utils.decorators import transaction
+
+
+def _decoded_result_for_wire(decoded: Any) -> tuple[Any, bool]:
+ """Convert a codec-decoded value into its JSON wire representation.
+
+ JSON has no byte type, so a raw `bytes` value is base64-encoded to a
+ string for the response and flagged as such; every other value is
+ already JSON-representable as-is. Checked on the decoded value's
+ actual type, not the codec's name, so this keeps working for a codec
+ this module didn't define.
+
+ :returns: (wire_value, is_binary)
+ """
+ if isinstance(decoded, bytes):
+ return base64.b64encode(decoded).decode("ascii"), True
+ return decoded, False
+
+
+def _wire_value_for_request(value: Any, is_binary: bool) -> Any:
+ """Convert a request body's JSON `value` into a codec's input type.
+
+ `is_binary` is an explicit flag from the caller: JSON has no byte
+ type, so there is no way to tell, from the JSON `value` alone,
+ whether it is a base64 string that must be decoded to bytes before
+ being handed to the codec's `encode`, or a literal value to pass
+ through as-is. Only the caller knows which it sent.
+
+ :raises ValueError: if `is_binary` is set and `value` is not a valid
+ base64 string.
+ """
+ if is_binary:
+ return base64.b64decode(value, validate=True)
+ return value
+
+
+class ExtensionStorageRestApi(BaseApi):
+ """REST API for extension ephemeral state storage."""
+
+ allow_browser_login = True
+ route_base = "/api/v1/extensions"
+
+ def response(self, status_code: int, **kwargs: Any) -> Response:
+ """Helper method to create JSON responses."""
+ from flask import jsonify
+
+ return jsonify(kwargs), status_code
+
+ def response_404(self, message: str = "Not found") -> Response:
+ """Helper method to create 404 responses."""
+ from flask import jsonify
+
+ return jsonify({"message": message}), 404
+
+ def response_400(self, message: str) -> Response:
+ """Helper method to create 400 responses."""
+ from flask import jsonify
+
+ return jsonify({"message": message}), 400
+
+ @protect()
+ @safe
+ @expose("/<publisher>/<name>/storage/ephemeral/<key>", methods=("GET",))
+ def get_ephemeral(
+ self, publisher: str, name: str, key: str, **kwargs: Any
+ ) -> Response:
+ """Get a value from ephemeral state.
+ ---
+ get:
+ summary: Get a value from ephemeral state
+ parameters:
+ - in: path
+ name: publisher
+ schema:
+ type: string
+ required: true
+ description: Extension publisher
+ - in: path
+ name: name
+ schema:
+ type: string
+ required: true
+ description: Extension name
+ - in: path
+ name: key
+ schema:
+ type: string
+ required: true
+ description: Storage key
+ - in: query
+ name: shared
+ schema:
+ type: boolean
+ required: false
+ description: If true, read from shared state visible to all users
+ responses:
+ 200:
+ description: Value retrieved successfully
+ content:
+ application/json:
+ schema:
+ type: object
+ properties:
+ result:
+ description: The stored value. When 'isBinary' is
+ true, this is a base64 string that must be
+ decoded to get the actual value.
+ codec:
+ type: string
+ description: Name of the codec 'result' was encoded
+ with, e.g. "json" (default) or "binary"
+ isBinary:
+ type: boolean
+ description: Whether the stored value is binary
+ data
+ 400:
+ description: Value was stored with a codec unavailable over the API
+ 404:
+ description: Extension not found
+ """
+ extension_id = f"{publisher}.{name}"
+ extension = get_extension_or_404(extension_id)
+ if not extension:
+ return self.response_404("Extension not found")
+
+ shared = request.args.get("shared", "false").lower() == "true"
+ raw = ExtensionEphemeralDAO.get_raw(extension_id, key, shared=shared)
+ if raw is None:
+ return self.response(200, result=None)
+ value, codec = raw
+ if codec not in SAFE_CODECS:
+ return self.response_400(
+ f"Value was stored with codec '{codec}', which cannot be "
+ "read over the REST API."
+ )
+
+ result, is_binary = _decoded_result_for_wire(get_codec(codec).decode(value))
+ return self.response(200, result=result, codec=codec, isBinary=is_binary)
+
+ @protect()
+ @safe
+ @expose("/<publisher>/<name>/storage/ephemeral/<key>", methods=("PUT",))
+ def set_ephemeral(
+ self, publisher: str, name: str, key: str, **kwargs: Any
+ ) -> Response:
+ """Set a value in ephemeral state.
+ ---
+ put:
+ summary: Set a value in ephemeral state
+ parameters:
+ - in: path
+ name: publisher
+ schema:
+ type: string
+ required: true
+ description: Extension publisher
+ - in: path
+ name: name
+ schema:
+ type: string
+ required: true
+ description: Extension name
+ - in: path
+ name: key
+ schema:
+ type: string
+ required: true
+ description: Storage key
+ - in: query
+ name: shared
+ schema:
+ type: boolean
+ required: false
+ description: If true, store as shared state visible to all users
+ requestBody:
+ required: true
+ content:
+ application/json:
+ schema:
+ type: object
+ required:
+ - value
+ - ttl
+ properties:
+ value:
+ description: The value to store (must not exceed
+ MAX_VALUE_SIZE bytes once encoded with 'codec').
+ When 'isBinary' is true, this must be a base64
+ string, decoded before being handed to 'codec'.
+ codec:
+ type: string
+ description: Name of the codec used to encode 'value',
+ e.g. "json" (default). Must be one of the codecs
+ allowed over the REST API.
+ isBinary:
+ type: boolean
+ description: Whether 'value' is binary data
+ ttl:
+ type: integer
+ description: Time-to-live in seconds (must be a positive
+ integer not exceeding MAX_TTL)
+ responses:
+ 200:
+ description: Value stored successfully
+ 400:
+ description: Invalid request body, or codec not allowed over the API
+ 404:
+ description: Extension not found
+ """
+ extension_id = f"{publisher}.{name}"
+ extension = get_extension_or_404(extension_id)
+ if not extension:
+ return self.response_404("Extension not found")
+
+ body = request.get_json(silent=True) or {}
+ if "value" not in body:
+ return self.response_400("Request body must contain 'value' field")
+
+ codec = body.get("codec", DEFAULT_CODEC)
+ if codec not in SAFE_CODECS:
+ return self.response_400(
+ f"Codec '{codec}' is not allowed over the REST API."
+ )
+
+ is_binary = bool(body.get("isBinary", False))
+ try:
+ value = _wire_value_for_request(body["value"], is_binary)
+ except (ValueError, TypeError):
+ return self.response_400(
+ "Value must be a valid base64 string when 'isBinary' is true."
+ )
+ ttl, error = parse_ttl(body)
+ if error:
+ return self.response_400(error)
+
+ shared = request.args.get("shared", "false").lower() == "true"
+ try:
+ ExtensionEphemeralDAO.set(
+ extension_id, key, value, ttl, codec=codec, shared=shared
+ )
+ except (ExtensionEphemeralTTLInvalid, ExtensionEphemeralValueTooLarge) as ex:
+ return self.response_400(ex.message)
+
+ return self.response(200, message="Value stored successfully")
+
+ @protect()
+ @safe
+ @expose("/<publisher>/<name>/storage/ephemeral/<key>", methods=("DELETE",))
+ def delete_ephemeral(
+ self, publisher: str, name: str, key: str, **kwargs: Any
+ ) -> Response:
+ """Delete a value from ephemeral state.
+ ---
+ delete:
+ summary: Delete a value from ephemeral state
+ parameters:
+ - in: path
+ name: publisher
+ schema:
+ type: string
+ required: true
+ description: Extension publisher
+ - in: path
+ name: name
+ schema:
+ type: string
+ required: true
+ description: Extension name
+ - in: path
+ name: key
+ schema:
+ type: string
+ required: true
+ description: Storage key
+ - in: query
+ name: shared
+ schema:
+ type: boolean
+ required: false
+ description: If true, delete from shared state
+ responses:
+ 200:
+ description: Value deleted successfully
+ 404:
+ description: Extension not found
+ """
+ extension_id = f"{publisher}.{name}"
+ extension = get_extension_or_404(extension_id)
+ if not extension:
+ return self.response_404("Extension not found")
+
+ shared = request.args.get("shared", "false").lower() == "true"
+ ExtensionEphemeralDAO.delete(extension_id, key, shared=shared)
+
+ return self.response(200, message="Value deleted successfully")
+
+ @protect()
+ @safe
+ @expose("/<publisher>/<name>/storage/persistent", methods=("GET",))
+ def list_persistent(self, publisher: str, name: str, **kwargs: Any) -> Response:
+ """List entries in persistent state.
+ ---
+ get:
+ summary: List entries in persistent state
+ parameters:
+ - in: path
+ name: publisher
+ schema:
+ type: string
+ required: true
+ description: Extension publisher
+ - in: path
+ name: name
+ schema:
+ type: string
+ required: true
+ description: Extension name
+ - in: query
+ name: shared
+ schema:
+ type: boolean
+ required: false
+ description: If true, list shared state visible to all users
+ - in: query
+ name: resource_type
+ schema:
+ type: string
+ required: false
+ description: Filter by resource type
+ - in: query
+ name: resource_uuid
+ schema:
+ type: string
+ required: false
+ description: Filter by resource UUID (requires resource_type)
+ - in: query
+ name: page
+ schema:
+ type: integer
+ required: false
+ description: Zero-indexed page number. Defaults to 0.
+ - in: query
+ name: page_size
+ schema:
+ type: integer
+ required: false
+ description: Number of entries per page. Defaults to 10. There
+ is no fixed ceiling, but a page whose combined value size
+ exceeds MAX_LIST_PAYLOAD_SIZE is rejected β reduce page_size
+ and retry if that happens.
+ responses:
+ 200:
+ description: Entries retrieved successfully
+ content:
+ application/json:
+ schema:
+ type: object
+ properties:
+ result:
+ type: array
+ items:
+ type: object
+ properties:
+ key:
+ type: string
+ value:
+ description: The stored value, or null if its
+ codec cannot be read over the REST API.
+ When 'isBinary' is true, this is a base64
+ string that must be decoded to get the
+ actual value.
+ codec:
+ type: string
+ isBinary:
+ type: boolean
+ description: Whether the stored value is
+ binary data
+ count:
+ type: integer
+ description: Total number of entries matching the
+ given scope/filters, across all pages
+ 400:
+ description: The requested page's combined value size exceeds
+ MAX_LIST_PAYLOAD_SIZE
+ 404:
+ description: Extension not found
+ """
+ extension_id = f"{publisher}.{name}"
+ extension = get_extension_or_404(extension_id)
+ if not extension:
+ return self.response_404("Extension not found")
+
+ shared = request.args.get("shared", "false").lower() == "true"
+ user_fk = None if shared else g.user.id
+ resource_type = request.args.get("resource_type")
+ resource_uuid = request.args.get("resource_uuid")
+ try:
+ page = int(request.args.get("page", 0))
+ page_size = int(request.args.get("page_size", 10))
+ except (TypeError, ValueError):
+ return self.response_400("'page' and 'page_size' must be integers")
+
+ try:
+ entries, count = ExtensionStorageDAO.list_entries(
+ extension_id,
+ user_fk=user_fk,
+ resource_type=resource_type,
+ resource_uuid=resource_uuid,
+ page=page,
+ page_size=page_size,
+ )
+ except ExtensionStorageListPayloadTooLarge as ex:
+ return self.response(ex.status, message=ex.message)
+
+ result = []
+ for entry in entries:
+ value: Any = None
+ is_binary = False
+ if entry.codec in SAFE_CODECS and entry.value is not None:
+ decoded = get_codec(entry.codec).decode(entry.value)
+ value, is_binary = _decoded_result_for_wire(decoded)
+ result.append(
+ {
+ "key": entry.key,
+ "value": value,
+ "codec": entry.codec,
+ "isBinary": is_binary,
+ }
+ )
+
+ return self.response(200, result=result, count=count)
+
+ @protect()
+ @safe
+ @expose("/<publisher>/<name>/storage/persistent/<key>", methods=("GET",))
+ def get_persistent(
+ self, publisher: str, name: str, key: str, **kwargs: Any
+ ) -> Response:
+ """Get a value from persistent state.
+ ---
+ get:
+ summary: Get a value from persistent state
+ parameters:
+ - in: path
+ name: publisher
+ schema:
+ type: string
+ required: true
+ description: Extension publisher
+ - in: path
+ name: name
+ schema:
+ type: string
+ required: true
+ description: Extension name
+ - in: path
+ name: key
+ schema:
+ type: string
+ required: true
+ description: Storage key
+ - in: query
+ name: shared
+ schema:
+ type: boolean
+ required: false
+ description: If true, read from shared state visible to all users
+ responses:
+ 200:
+ description: Value retrieved successfully
+ content:
+ application/json:
+ schema:
+ type: object
+ properties:
+ result:
+ description: The stored value. When 'isBinary' is
+ true, this is a base64 string that must be
+ decoded to get the actual value.
+ codec:
+ type: string
+ description: Name of the codec 'result' was encoded
+ with, e.g. "json" (default) or "binary"
+ isBinary:
+ type: boolean
+ description: Whether the stored value is binary
+ data
+ 400:
+ description: Value was stored with a codec unavailable over the API
+ 404:
+ description: Extension not found
+ """
+ extension_id = f"{publisher}.{name}"
+ extension = get_extension_or_404(extension_id)
+ if not extension:
+ return self.response_404("Extension not found")
+
+ shared = request.args.get("shared", "false").lower() == "true"
+ user_fk = None if shared else g.user.id
+ entry = ExtensionStorageDAO.get(extension_id, key, user_fk=user_fk)
+ if entry is None:
+ return self.response(200, result=None)
+ if entry.codec not in SAFE_CODECS:
+ return self.response_400(
+ f"Value was stored with codec '{entry.codec}', which cannot be "
+ "read over the REST API."
+ )
+
+ decoded = ExtensionStorageDAO.get_decoded_value(
+ extension_id, key, user_fk=user_fk
+ )
+ result, is_binary = _decoded_result_for_wire(decoded)
+
+ return self.response(200, result=result, codec=entry.codec, isBinary=is_binary)
+
+ @protect()
+ @safe
+ @expose("/<publisher>/<name>/storage/persistent/<key>", methods=("PUT",))
+ @transaction()
+ def set_persistent(
+ self, publisher: str, name: str, key: str, **kwargs: Any
+ ) -> Response:
+ """Set a value in persistent state.
+ ---
+ put:
+ summary: Set a value in persistent state
+ parameters:
+ - in: path
+ name: publisher
+ schema:
+ type: string
+ required: true
+ description: Extension publisher
+ - in: path
+ name: name
+ schema:
+ type: string
+ required: true
+ description: Extension name
+ - in: path
+ name: key
+ schema:
+ type: string
+ required: true
+ description: Storage key
+ - in: query
+ name: shared
+ schema:
+ type: boolean
+ required: false
+ description: If true, store as shared state visible to all users
+ requestBody:
+ required: true
+ content:
+ application/json:
+ schema:
+ type: object
+ required:
+ - value
+ properties:
+ value:
+ description: The value to store. When 'isBinary' is
+ true, this must be a base64 string, decoded
+ before being handed to 'codec'.
+ codec:
+ type: string
+ description: Name of the codec used to encode 'value',
+ e.g. "json" (default). Must be one of the codecs
+ allowed over the REST API.
+ isBinary:
+ type: boolean
+ description: Whether 'value' is binary data
+ encrypt:
+ type: boolean
+ description: If true, the value is encrypted at rest
+ responses:
+ 200:
+ description: Value stored successfully
+ 400:
+ description: Invalid request body, codec not allowed over the
+ API, or value exceeds MAX_VALUE_SIZE
+ 404:
+ description: Extension not found
+ 413:
+ description: Extension persistent storage quota exceeded
+ """
+ extension_id = f"{publisher}.{name}"
+ extension = get_extension_or_404(extension_id)
+ if not extension:
+ return self.response_404("Extension not found")
+
+ body = request.get_json(silent=True) or {}
+ if "value" not in body:
+ return self.response_400("Request body must contain 'value' field")
+
+ codec = body.get("codec", DEFAULT_CODEC)
+ if codec not in SAFE_CODECS:
+ return self.response_400(
+ f"Codec '{codec}' is not allowed over the REST API."
+ )
Review Comment:
**Suggestion:** The same unhashable `codec` issue exists in persistent writes: sending a list/object for `codec` causes `TypeError` in the set-membership check and returns a 500; reject non-string codec values before checking allowed codecs. [type error]
<details>
<summary><b>Severity Level:</b> Major β οΈ</summary>
```mdx
- β Persistent storage write can crash on malformed codec.
- β οΈ Persistent API less robust to invalid client input.
```
</details>
<details>
<summary><b>Steps of Reproduction β
</b></summary>
```mdx
1. Start Superset with this PR, ensuring `ExtensionStorageRestApi`
(superset/extensions/storage/api.py:88) is loaded.
2. Authenticate and call `PUT
/api/v1/extensions/test_publisher/test_name/storage/persistent/test_key` (handled by
`set_persistent` at lines 566β572).
3. Use a JSON body like `{"value": "test", "codec": ["json"], "encrypt": false}` so
`request.get_json()` (line 642) returns a list for `codec` at line 646.
4. At line 647, `if codec not in SAFE_CODECS:` runs; since `SAFE_CODECS` is a set of
allowed codec names (superset/extensions/storage/codecs.py), membership with a list raises
`TypeError: unhashable type: 'list'`, which is uncaught and results in a 500 instead of a
400 describing the invalid codec.
```
</details>
[](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=f57d080065c1435c8957fe52090fa46c&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=f57d080065c1435c8957fe52090fa46c&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset)
*(Use Cmd/Ctrl + Click for best experience)*
<details>
<summary><b>Prompt for AI Agent π€ </b></summary>
```mdx
This is a comment left during a code review.
**Path:** superset/extensions/storage/api.py
**Line:** 646:650
**Comment:**
*Type Error: The same unhashable `codec` issue exists in persistent writes: sending a list/object for `codec` causes `TypeError` in the set-membership check and returns a 500; reject non-string codec values before checking allowed codecs.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F39171&comment_hash=b1fbb15b2d99341cbc40c8ceec8112156dc9fd81be8e239721c6992b7bc0bcfd&reaction=like'>π</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F39171&comment_hash=b1fbb15b2d99341cbc40c8ceec8112156dc9fd81be8e239721c6992b7bc0bcfd&reaction=dislike'>π</a>
##########
superset/extensions/storage/api.py:
##########
@@ -0,0 +1,732 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""
+REST API for extension storage.
+
+Provides HTTP endpoints for frontend extensions to access server-side
+ephemeral storage without direct backend code.
+
+All operations are user-scoped by default. Use `?shared=true` query param
+to access shared state visible to all users.
+"""
+
+from __future__ import annotations
+
+import base64
+from typing import Any
+
+from flask import g, request
+from flask.wrappers import Response
+from flask_appbuilder.api import BaseApi, expose, protect, safe
+
+from superset.extensions.storage.codecs import DEFAULT_CODEC, get_codec, SAFE_CODECS
+from superset.extensions.storage.ephemeral_dao import (
+ ExtensionEphemeralDAO,
+ ExtensionEphemeralTTLInvalid,
+ ExtensionEphemeralValueTooLarge,
+)
+from superset.extensions.storage.persistent_dao import (
+ ExtensionStorageDAO,
+ ExtensionStorageKeyTooLong,
+ ExtensionStorageListPayloadTooLarge,
+ ExtensionStorageQuotaExceeded,
+ ExtensionStorageValueTooLarge,
+)
+from superset.extensions.storage.utils import get_extension_or_404, parse_ttl
+from superset.utils.decorators import transaction
+
+
+def _decoded_result_for_wire(decoded: Any) -> tuple[Any, bool]:
+ """Convert a codec-decoded value into its JSON wire representation.
+
+ JSON has no byte type, so a raw `bytes` value is base64-encoded to a
+ string for the response and flagged as such; every other value is
+ already JSON-representable as-is. Checked on the decoded value's
+ actual type, not the codec's name, so this keeps working for a codec
+ this module didn't define.
+
+ :returns: (wire_value, is_binary)
+ """
+ if isinstance(decoded, bytes):
+ return base64.b64encode(decoded).decode("ascii"), True
+ return decoded, False
+
+
+def _wire_value_for_request(value: Any, is_binary: bool) -> Any:
+ """Convert a request body's JSON `value` into a codec's input type.
+
+ `is_binary` is an explicit flag from the caller: JSON has no byte
+ type, so there is no way to tell, from the JSON `value` alone,
+ whether it is a base64 string that must be decoded to bytes before
+ being handed to the codec's `encode`, or a literal value to pass
+ through as-is. Only the caller knows which it sent.
+
+ :raises ValueError: if `is_binary` is set and `value` is not a valid
+ base64 string.
+ """
+ if is_binary:
+ return base64.b64decode(value, validate=True)
+ return value
+
+
+class ExtensionStorageRestApi(BaseApi):
+ """REST API for extension ephemeral state storage."""
+
+ allow_browser_login = True
+ route_base = "/api/v1/extensions"
+
+ def response(self, status_code: int, **kwargs: Any) -> Response:
+ """Helper method to create JSON responses."""
+ from flask import jsonify
+
+ return jsonify(kwargs), status_code
+
+ def response_404(self, message: str = "Not found") -> Response:
+ """Helper method to create 404 responses."""
+ from flask import jsonify
+
+ return jsonify({"message": message}), 404
+
+ def response_400(self, message: str) -> Response:
+ """Helper method to create 400 responses."""
+ from flask import jsonify
+
+ return jsonify({"message": message}), 400
+
+ @protect()
+ @safe
+ @expose("/<publisher>/<name>/storage/ephemeral/<key>", methods=("GET",))
+ def get_ephemeral(
+ self, publisher: str, name: str, key: str, **kwargs: Any
+ ) -> Response:
+ """Get a value from ephemeral state.
+ ---
+ get:
+ summary: Get a value from ephemeral state
+ parameters:
+ - in: path
+ name: publisher
+ schema:
+ type: string
+ required: true
+ description: Extension publisher
+ - in: path
+ name: name
+ schema:
+ type: string
+ required: true
+ description: Extension name
+ - in: path
+ name: key
+ schema:
+ type: string
+ required: true
+ description: Storage key
+ - in: query
+ name: shared
+ schema:
+ type: boolean
+ required: false
+ description: If true, read from shared state visible to all users
+ responses:
+ 200:
+ description: Value retrieved successfully
+ content:
+ application/json:
+ schema:
+ type: object
+ properties:
+ result:
+ description: The stored value. When 'isBinary' is
+ true, this is a base64 string that must be
+ decoded to get the actual value.
+ codec:
+ type: string
+ description: Name of the codec 'result' was encoded
+ with, e.g. "json" (default) or "binary"
+ isBinary:
+ type: boolean
+ description: Whether the stored value is binary
+ data
+ 400:
+ description: Value was stored with a codec unavailable over the API
+ 404:
+ description: Extension not found
+ """
+ extension_id = f"{publisher}.{name}"
+ extension = get_extension_or_404(extension_id)
+ if not extension:
+ return self.response_404("Extension not found")
+
+ shared = request.args.get("shared", "false").lower() == "true"
+ raw = ExtensionEphemeralDAO.get_raw(extension_id, key, shared=shared)
+ if raw is None:
+ return self.response(200, result=None)
+ value, codec = raw
+ if codec not in SAFE_CODECS:
+ return self.response_400(
+ f"Value was stored with codec '{codec}', which cannot be "
+ "read over the REST API."
+ )
+
+ result, is_binary = _decoded_result_for_wire(get_codec(codec).decode(value))
+ return self.response(200, result=result, codec=codec, isBinary=is_binary)
+
+ @protect()
+ @safe
+ @expose("/<publisher>/<name>/storage/ephemeral/<key>", methods=("PUT",))
+ def set_ephemeral(
+ self, publisher: str, name: str, key: str, **kwargs: Any
+ ) -> Response:
+ """Set a value in ephemeral state.
+ ---
+ put:
+ summary: Set a value in ephemeral state
+ parameters:
+ - in: path
+ name: publisher
+ schema:
+ type: string
+ required: true
+ description: Extension publisher
+ - in: path
+ name: name
+ schema:
+ type: string
+ required: true
+ description: Extension name
+ - in: path
+ name: key
+ schema:
+ type: string
+ required: true
+ description: Storage key
+ - in: query
+ name: shared
+ schema:
+ type: boolean
+ required: false
+ description: If true, store as shared state visible to all users
+ requestBody:
+ required: true
+ content:
+ application/json:
+ schema:
+ type: object
+ required:
+ - value
+ - ttl
+ properties:
+ value:
+ description: The value to store (must not exceed
+ MAX_VALUE_SIZE bytes once encoded with 'codec').
+ When 'isBinary' is true, this must be a base64
+ string, decoded before being handed to 'codec'.
+ codec:
+ type: string
+ description: Name of the codec used to encode 'value',
+ e.g. "json" (default). Must be one of the codecs
+ allowed over the REST API.
+ isBinary:
+ type: boolean
+ description: Whether 'value' is binary data
+ ttl:
+ type: integer
+ description: Time-to-live in seconds (must be a positive
+ integer not exceeding MAX_TTL)
+ responses:
+ 200:
+ description: Value stored successfully
+ 400:
+ description: Invalid request body, or codec not allowed over the API
+ 404:
+ description: Extension not found
+ """
+ extension_id = f"{publisher}.{name}"
+ extension = get_extension_or_404(extension_id)
+ if not extension:
+ return self.response_404("Extension not found")
+
+ body = request.get_json(silent=True) or {}
+ if "value" not in body:
+ return self.response_400("Request body must contain 'value' field")
+
+ codec = body.get("codec", DEFAULT_CODEC)
+ if codec not in SAFE_CODECS:
+ return self.response_400(
+ f"Codec '{codec}' is not allowed over the REST API."
+ )
+
+ is_binary = bool(body.get("isBinary", False))
+ try:
+ value = _wire_value_for_request(body["value"], is_binary)
+ except (ValueError, TypeError):
+ return self.response_400(
+ "Value must be a valid base64 string when 'isBinary' is true."
+ )
+ ttl, error = parse_ttl(body)
+ if error:
+ return self.response_400(error)
+
+ shared = request.args.get("shared", "false").lower() == "true"
+ try:
+ ExtensionEphemeralDAO.set(
+ extension_id, key, value, ttl, codec=codec, shared=shared
+ )
+ except (ExtensionEphemeralTTLInvalid, ExtensionEphemeralValueTooLarge) as ex:
+ return self.response_400(ex.message)
+
+ return self.response(200, message="Value stored successfully")
+
+ @protect()
+ @safe
+ @expose("/<publisher>/<name>/storage/ephemeral/<key>", methods=("DELETE",))
+ def delete_ephemeral(
+ self, publisher: str, name: str, key: str, **kwargs: Any
+ ) -> Response:
+ """Delete a value from ephemeral state.
+ ---
+ delete:
+ summary: Delete a value from ephemeral state
+ parameters:
+ - in: path
+ name: publisher
+ schema:
+ type: string
+ required: true
+ description: Extension publisher
+ - in: path
+ name: name
+ schema:
+ type: string
+ required: true
+ description: Extension name
+ - in: path
+ name: key
+ schema:
+ type: string
+ required: true
+ description: Storage key
+ - in: query
+ name: shared
+ schema:
+ type: boolean
+ required: false
+ description: If true, delete from shared state
+ responses:
+ 200:
+ description: Value deleted successfully
+ 404:
+ description: Extension not found
+ """
+ extension_id = f"{publisher}.{name}"
+ extension = get_extension_or_404(extension_id)
+ if not extension:
+ return self.response_404("Extension not found")
+
+ shared = request.args.get("shared", "false").lower() == "true"
+ ExtensionEphemeralDAO.delete(extension_id, key, shared=shared)
+
+ return self.response(200, message="Value deleted successfully")
+
+ @protect()
+ @safe
+ @expose("/<publisher>/<name>/storage/persistent", methods=("GET",))
+ def list_persistent(self, publisher: str, name: str, **kwargs: Any) -> Response:
+ """List entries in persistent state.
+ ---
+ get:
+ summary: List entries in persistent state
+ parameters:
+ - in: path
+ name: publisher
+ schema:
+ type: string
+ required: true
+ description: Extension publisher
+ - in: path
+ name: name
+ schema:
+ type: string
+ required: true
+ description: Extension name
+ - in: query
+ name: shared
+ schema:
+ type: boolean
+ required: false
+ description: If true, list shared state visible to all users
+ - in: query
+ name: resource_type
+ schema:
+ type: string
+ required: false
+ description: Filter by resource type
+ - in: query
+ name: resource_uuid
+ schema:
+ type: string
+ required: false
+ description: Filter by resource UUID (requires resource_type)
+ - in: query
+ name: page
+ schema:
+ type: integer
+ required: false
+ description: Zero-indexed page number. Defaults to 0.
+ - in: query
+ name: page_size
+ schema:
+ type: integer
+ required: false
+ description: Number of entries per page. Defaults to 10. There
+ is no fixed ceiling, but a page whose combined value size
+ exceeds MAX_LIST_PAYLOAD_SIZE is rejected β reduce page_size
+ and retry if that happens.
+ responses:
+ 200:
+ description: Entries retrieved successfully
+ content:
+ application/json:
+ schema:
+ type: object
+ properties:
+ result:
+ type: array
+ items:
+ type: object
+ properties:
+ key:
+ type: string
+ value:
+ description: The stored value, or null if its
+ codec cannot be read over the REST API.
+ When 'isBinary' is true, this is a base64
+ string that must be decoded to get the
+ actual value.
+ codec:
+ type: string
+ isBinary:
+ type: boolean
+ description: Whether the stored value is
+ binary data
+ count:
+ type: integer
+ description: Total number of entries matching the
+ given scope/filters, across all pages
+ 400:
+ description: The requested page's combined value size exceeds
+ MAX_LIST_PAYLOAD_SIZE
+ 404:
+ description: Extension not found
+ """
+ extension_id = f"{publisher}.{name}"
+ extension = get_extension_or_404(extension_id)
+ if not extension:
+ return self.response_404("Extension not found")
+
+ shared = request.args.get("shared", "false").lower() == "true"
+ user_fk = None if shared else g.user.id
+ resource_type = request.args.get("resource_type")
+ resource_uuid = request.args.get("resource_uuid")
+ try:
+ page = int(request.args.get("page", 0))
+ page_size = int(request.args.get("page_size", 10))
+ except (TypeError, ValueError):
+ return self.response_400("'page' and 'page_size' must be integers")
+
+ try:
+ entries, count = ExtensionStorageDAO.list_entries(
+ extension_id,
+ user_fk=user_fk,
+ resource_type=resource_type,
+ resource_uuid=resource_uuid,
+ page=page,
+ page_size=page_size,
+ )
+ except ExtensionStorageListPayloadTooLarge as ex:
+ return self.response(ex.status, message=ex.message)
+
+ result = []
+ for entry in entries:
+ value: Any = None
+ is_binary = False
+ if entry.codec in SAFE_CODECS and entry.value is not None:
+ decoded = get_codec(entry.codec).decode(entry.value)
+ value, is_binary = _decoded_result_for_wire(decoded)
+ result.append(
+ {
+ "key": entry.key,
+ "value": value,
+ "codec": entry.codec,
+ "isBinary": is_binary,
+ }
+ )
+
+ return self.response(200, result=result, count=count)
+
+ @protect()
+ @safe
+ @expose("/<publisher>/<name>/storage/persistent/<key>", methods=("GET",))
+ def get_persistent(
+ self, publisher: str, name: str, key: str, **kwargs: Any
+ ) -> Response:
+ """Get a value from persistent state.
+ ---
+ get:
+ summary: Get a value from persistent state
+ parameters:
+ - in: path
+ name: publisher
+ schema:
+ type: string
+ required: true
+ description: Extension publisher
+ - in: path
+ name: name
+ schema:
+ type: string
+ required: true
+ description: Extension name
+ - in: path
+ name: key
+ schema:
+ type: string
+ required: true
+ description: Storage key
+ - in: query
+ name: shared
+ schema:
+ type: boolean
+ required: false
+ description: If true, read from shared state visible to all users
+ responses:
+ 200:
+ description: Value retrieved successfully
+ content:
+ application/json:
+ schema:
+ type: object
+ properties:
+ result:
+ description: The stored value. When 'isBinary' is
+ true, this is a base64 string that must be
+ decoded to get the actual value.
+ codec:
+ type: string
+ description: Name of the codec 'result' was encoded
+ with, e.g. "json" (default) or "binary"
+ isBinary:
+ type: boolean
+ description: Whether the stored value is binary
+ data
+ 400:
+ description: Value was stored with a codec unavailable over the API
+ 404:
+ description: Extension not found
+ """
+ extension_id = f"{publisher}.{name}"
+ extension = get_extension_or_404(extension_id)
+ if not extension:
+ return self.response_404("Extension not found")
+
+ shared = request.args.get("shared", "false").lower() == "true"
+ user_fk = None if shared else g.user.id
+ entry = ExtensionStorageDAO.get(extension_id, key, user_fk=user_fk)
+ if entry is None:
+ return self.response(200, result=None)
+ if entry.codec not in SAFE_CODECS:
+ return self.response_400(
+ f"Value was stored with codec '{entry.codec}', which cannot be "
+ "read over the REST API."
+ )
+
+ decoded = ExtensionStorageDAO.get_decoded_value(
+ extension_id, key, user_fk=user_fk
+ )
+ result, is_binary = _decoded_result_for_wire(decoded)
+
+ return self.response(200, result=result, codec=entry.codec, isBinary=is_binary)
+
+ @protect()
+ @safe
+ @expose("/<publisher>/<name>/storage/persistent/<key>", methods=("PUT",))
+ @transaction()
+ def set_persistent(
+ self, publisher: str, name: str, key: str, **kwargs: Any
+ ) -> Response:
+ """Set a value in persistent state.
+ ---
+ put:
+ summary: Set a value in persistent state
+ parameters:
+ - in: path
+ name: publisher
+ schema:
+ type: string
+ required: true
+ description: Extension publisher
+ - in: path
+ name: name
+ schema:
+ type: string
+ required: true
+ description: Extension name
+ - in: path
+ name: key
+ schema:
+ type: string
+ required: true
+ description: Storage key
+ - in: query
+ name: shared
+ schema:
+ type: boolean
+ required: false
+ description: If true, store as shared state visible to all users
+ requestBody:
+ required: true
+ content:
+ application/json:
+ schema:
+ type: object
+ required:
+ - value
+ properties:
+ value:
+ description: The value to store. When 'isBinary' is
+ true, this must be a base64 string, decoded
+ before being handed to 'codec'.
+ codec:
+ type: string
+ description: Name of the codec used to encode 'value',
+ e.g. "json" (default). Must be one of the codecs
+ allowed over the REST API.
+ isBinary:
+ type: boolean
+ description: Whether 'value' is binary data
+ encrypt:
+ type: boolean
+ description: If true, the value is encrypted at rest
+ responses:
+ 200:
+ description: Value stored successfully
+ 400:
+ description: Invalid request body, codec not allowed over the
+ API, or value exceeds MAX_VALUE_SIZE
+ 404:
+ description: Extension not found
+ 413:
+ description: Extension persistent storage quota exceeded
+ """
+ extension_id = f"{publisher}.{name}"
+ extension = get_extension_or_404(extension_id)
+ if not extension:
+ return self.response_404("Extension not found")
+
+ body = request.get_json(silent=True) or {}
+ if "value" not in body:
+ return self.response_400("Request body must contain 'value' field")
+
+ codec = body.get("codec", DEFAULT_CODEC)
+ if codec not in SAFE_CODECS:
+ return self.response_400(
+ f"Codec '{codec}' is not allowed over the REST API."
+ )
+
+ is_binary = bool(body.get("isBinary", False))
+ encrypt = bool(body.get("encrypt", False))
+ shared = request.args.get("shared", "false").lower() == "true"
+ user_fk = None if shared else g.user.id
+ try:
+ wire_value = _wire_value_for_request(body["value"], is_binary)
+ except (ValueError, TypeError):
+ return self.response_400(
+ "Value must be a valid base64 string when 'isBinary' is true."
+ )
+ value_bytes = get_codec(codec).encode(wire_value)
Review Comment:
**Suggestion:** Encoding user-provided values is done outside the exception handling block, so serialization failures (for example non-JSON-serializable values when `codec` is `json`) will bubble up as 500 errors; catch encode/decode exceptions and return a 400 describing invalid payload for the selected codec. [possible bug]
<details>
<summary><b>Severity Level:</b> Major β οΈ</summary>
```mdx
- β Persistent writes can 500 on codec encoding errors.
- β οΈ Clients receive opaque failures for bad codec payloads.
```
</details>
<details>
<summary><b>Steps of Reproduction β
</b></summary>
```mdx
1. Start Superset with this PR; confirm `ExtensionStorageRestApi.set_persistent` is
available (superset/extensions/storage/api.py:566β679).
2. Send `PUT /api/v1/extensions/test_publisher/test_name/storage/persistent/test_key` with
JSON body that is incompatible with the chosen codec, e.g. `{"value": "not-bytes",
"codec": "binary", "isBinary": false}`.
3. Inside `set_persistent`, `wire_value = _wire_value_for_request(body["value"],
is_binary)` is computed at lines 652β657, and then `value_bytes =
get_codec(codec).encode(wire_value)` at line 662 calls the codecβs `encode` on an invalid
type.
4. The codecβs `encode` implementation (from superset/extensions/storage/codecs.py) raises
an exception (e.g. `TypeError` for wrong type), which is not caught by the surrounding
`try`/`except` that only wraps `ExtensionStorageDAO.set` (lines 664β677), so the request
returns HTTP 500 instead of a 400 explaining the invalid payload for the selected codec.
```
</details>
[](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=933b8e6a6e8a4475bcfcddac69a439c6&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=933b8e6a6e8a4475bcfcddac69a439c6&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset)
*(Use Cmd/Ctrl + Click for best experience)*
<details>
<summary><b>Prompt for AI Agent π€ </b></summary>
```mdx
This is a comment left during a code review.
**Path:** superset/extensions/storage/api.py
**Line:** 662:662
**Comment:**
*Possible Bug: Encoding user-provided values is done outside the exception handling block, so serialization failures (for example non-JSON-serializable values when `codec` is `json`) will bubble up as 500 errors; catch encode/decode exceptions and return a 400 describing invalid payload for the selected codec.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F39171&comment_hash=56003ae43da8be899976921af155eaa0c7a75bef76e48d87d2c3137bf578fe2e&reaction=like'>π</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F39171&comment_hash=56003ae43da8be899976921af155eaa0c7a75bef76e48d87d2c3137bf578fe2e&reaction=dislike'>π</a>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(extensions): add a Storage API for extensions [superset]
Posted by "codeant-ai-for-open-source[bot] (via GitHub)" <gi...@apache.org>.
codeant-ai-for-open-source[bot] commented on code in PR #39171:
URL: https://github.com/apache/superset/pull/39171#discussion_r3624212612
##########
superset/extensions/storage/api.py:
##########
@@ -0,0 +1,740 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""
+REST API for extension storage.
+
+Provides HTTP endpoints for frontend extensions to access server-side
+ephemeral storage without direct backend code.
+
+All operations are user-scoped by default. Use `?shared=true` query param
+to access shared state visible to all users.
+"""
+
+from __future__ import annotations
+
+import base64
+from typing import Any
+
+from flask import g, request
+from flask.wrappers import Response
+from flask_appbuilder.api import BaseApi, expose, protect, safe
+
+from superset.extensions.storage.codecs import DEFAULT_CODEC, get_codec, SAFE_CODECS
+from superset.extensions.storage.ephemeral_dao import (
+ ExtensionEphemeralDAO,
+ ExtensionEphemeralTTLInvalid,
+ ExtensionEphemeralValueTooLarge,
+)
+from superset.extensions.storage.persistent_dao import (
+ ExtensionStorageDAO,
+ ExtensionStorageKeyTooLong,
+ ExtensionStorageListPayloadTooLarge,
+ ExtensionStorageQuotaExceeded,
+ ExtensionStorageValueTooLarge,
+)
+from superset.extensions.storage.utils import get_extension_or_404, parse_ttl
+from superset.key_value.exceptions import KeyValueCodecEncodeException
+from superset.utils.decorators import transaction
+
+
+def _decoded_result_for_wire(decoded: Any) -> tuple[Any, bool]:
+ """Convert a codec-decoded value into its JSON wire representation.
+
+ JSON has no byte type, so a raw `bytes` value is base64-encoded to a
+ string for the response and flagged as such; every other value is
+ already JSON-representable as-is. Checked on the decoded value's
+ actual type, not the codec's name, so this keeps working for a codec
+ this module didn't define.
+
+ :returns: (wire_value, is_binary)
+ """
+ if isinstance(decoded, bytes):
+ return base64.b64encode(decoded).decode("ascii"), True
+ return decoded, False
+
+
+def _wire_value_for_request(value: Any, is_binary: bool) -> Any:
+ """Convert a request body's JSON `value` into a codec's input type.
+
+ `is_binary` is an explicit flag from the caller: JSON has no byte
+ type, so there is no way to tell, from the JSON `value` alone,
+ whether it is a base64 string that must be decoded to bytes before
+ being handed to the codec's `encode`, or a literal value to pass
+ through as-is. Only the caller knows which it sent.
+
+ :raises ValueError: if `is_binary` is set and `value` is not a valid
+ base64 string.
+ """
+ if is_binary:
+ return base64.b64decode(value, validate=True)
+ return value
Review Comment:
**Suggestion:** Invalid base64 payloads can raise `binascii.Error`, but the callers only catch `ValueError`/`TypeError`; this can leak a 500 instead of returning the intended 400 error. Catch the base64-specific exception in the request parsing path. [possible bug]
<details>
<summary><b>Severity Level:</b> Major β οΈ</summary>
```mdx
- β Ephemeral storage PUT with bad base64 returns server error.
- β οΈ Persistent storage PUT binary writes less robust to misuse.
```
</details>
<details>
<summary><b>Steps of Reproduction β
</b></summary>
```mdx
1. Start a Superset instance that includes `ExtensionStorageRestApi` from
`superset/extensions/storage/api.py`.
2. Send an HTTP PUT request to the ephemeral storage endpoint
`/api/v1/extensions/test_publisher/test_ext/storage/ephemeral/test_key` implemented by
`set_ephemeral` at `superset/extensions/storage/api.py:194-299`, with JSON body `{"value":
"not-base64!", "isBinary": true, "ttl": 60}`.
3. In `set_ephemeral` (`api.py:266-283`), the body is parsed, `is_binary` is set to True
(line 276), and `_wire_value_for_request` (`api.py:71-85`) is called with the string
`"not-base64!"` and `True`.
4. Inside `_wire_value_for_request` (line 83), `base64.b64decode(value, validate=True)`
raises `binascii.Error` for the invalid payload; this exception is not caught by the
surrounding `except (ValueError, TypeError)` in `set_ephemeral` (lines 279-282), so the
view raises and `@safe` returns an HTTP 500 instead of the intended 400 error.
```
</details>
[](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=e5189e1f4e3d4b3db4c18c4d3500dc75&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=e5189e1f4e3d4b3db4c18c4d3500dc75&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset)
*(Use Cmd/Ctrl + Click for best experience)*
<details>
<summary><b>Prompt for AI Agent π€ </b></summary>
```mdx
This is a comment left during a code review.
**Path:** superset/extensions/storage/api.py
**Line:** 71:85
**Comment:**
*Possible Bug: Invalid base64 payloads can raise `binascii.Error`, but the callers only catch `ValueError`/`TypeError`; this can leak a 500 instead of returning the intended 400 error. Catch the base64-specific exception in the request parsing path.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F39171&comment_hash=bc07bb5e8fed17e54371a7d2479d0df19532b347c6eb79d26ea949307e6b975b&reaction=like'>π</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F39171&comment_hash=bc07bb5e8fed17e54371a7d2479d0df19532b347c6eb79d26ea949307e6b975b&reaction=dislike'>π</a>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(extensions): add Tier 1 and Tier 2 storage APIs [superset]
Posted by "michael-s-molina (via GitHub)" <gi...@apache.org>.
michael-s-molina commented on code in PR #39171:
URL: https://github.com/apache/superset/pull/39171#discussion_r3065712736
##########
superset-frontend/packages/superset-core/src/storage/ephemeralState.ts:
##########
@@ -0,0 +1,160 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import type { JsonValue, StorageAccessor } from './types';
+
+/**
+ * @fileoverview Ephemeral State API for Superset extensions (Tier 2 Storage).
+ *
+ * Provides short-lived KV storage backed by the configured server-side cache
+ * backend (Redis, Memcached, or filesystem). Automatically expires based on TTL.
+ * Not guaranteed to survive server restarts.
+ *
+ * By default, all operations are user-scoped (private to the current user).
+ * Use `shared` to access state that is visible to all users.
+ *
+ * Cache keys are namespaced automatically:
+ * - User-scoped (default): superset-ext:{extension_id}:user:{user_id}:{key}
+ * - Shared (global): superset-ext:{extension_id}:{key}
+ *
+ * @example
+ * ```typescript
+ * import { ephemeralState } from '@apache-superset/core/storage';
+ *
+ * // User-scoped state (default - private to current user)
+ * const progress = await ephemeralState.get('job_progress');
+ * await ephemeralState.set('job_progress', { pct: 42 }, { ttl: 3600 });
+ * await ephemeralState.remove('job_progress');
+ *
+ * // Shared state (explicit opt-in - visible to all users)
+ * const result = await ephemeralState.shared.get('shared_result');
+ * await ephemeralState.shared.set('shared_result', { data: [1, 2, 3] });
+ * await ephemeralState.shared.remove('shared_result');
+ * ```
+ */
+
+/**
+ * Default TTL in seconds (1 hour).
+ */
+export const DEFAULT_TTL = 3600;
Review Comment:
Let's:
- Make `ttl` a required client parameter
- Change `CACHE_DEFAULT_TIMEOUT` to `MAX_TIMEOUT` = 7 days to give admins more control.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(extensions): add a Storage API for extensions [superset]
Posted by "villebro (via GitHub)" <gi...@apache.org>.
villebro commented on PR #39171:
URL: https://github.com/apache/superset/pull/39171#issuecomment-5024474411
> Nice work here, Michael, LGTM!
>
> One thing I'm curious about, though. `ExtensionStorageDAO.set` does a select-then-insert/update rather than an atomic upsert, and the unique constraint on `extension_storage` seeminly won't catch two concurrent writes to the same shared key since `user_fk` is NULL there. Is this a race condition we care about, or edge case we can live with?
That's a good point. I think we should use the distributed lock for this as we do in other similar cases.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(extensions): add a Storage API for extensions [superset]
Posted by "codeant-ai-for-open-source[bot] (via GitHub)" <gi...@apache.org>.
codeant-ai-for-open-source[bot] commented on code in PR #39171:
URL: https://github.com/apache/superset/pull/39171#discussion_r3561065485
##########
superset-frontend/src/extensions/ExtensionsLoader.test.ts:
##########
@@ -48,9 +47,53 @@ function createMockExtension(overrides: Partial<Extension> = {}): Extension {
};
}
+// Simulate what real webpack does when __webpack_init_sharing__ runs: it
+// registers the host's own eagerly-provided @apache-superset/core versions
+// into the share scope.
+function mockWebpackSharing(coreVersions: Record<string, unknown> = {}) {
+ (globalThis as any).__webpack_init_sharing__ = jest
+ .fn()
+ .mockImplementation(() => {
+ (globalThis as any).__webpack_share_scopes__.default[
+ '@apache-superset/core'
+ ] = coreVersions;
+ });
+ (globalThis as any).__webpack_share_scopes__ = { default: {} };
Review Comment:
**Suggestion:** Type the webpack share scopes object explicitly and remove the `any` cast from this assignment. [custom_rule]
**Severity Level:** Minor π§Ή
<details>
<summary><b>Why it matters? β </b></summary>
This is another direct `any` cast in TypeScript code, so the no-`any` rule is violated here as well.
</details>
[](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=5e4c1403642a47a2ab1afb0fd5a8443d&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=5e4c1403642a47a2ab1afb0fd5a8443d&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset)
*(Use Cmd/Ctrl + Click for best experience)*
<details>
<summary><b>Prompt for AI Agent π€ </b></summary>
```mdx
This is a comment left during a code review.
**Path:** superset-frontend/src/extensions/ExtensionsLoader.test.ts
**Line:** 61:61
**Comment:**
*Custom Rule: Type the webpack share scopes object explicitly and remove the `any` cast from this assignment.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F39171&comment_hash=ed4bf1ee0f8ec5dc25c65aba5b1daf36cc228859328fc2379638892b7d311198&reaction=like'>π</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F39171&comment_hash=ed4bf1ee0f8ec5dc25c65aba5b1daf36cc228859328fc2379638892b7d311198&reaction=dislike'>π</a>
##########
superset-frontend/src/extensions/ExtensionsLoader.test.ts:
##########
@@ -48,9 +47,53 @@ function createMockExtension(overrides: Partial<Extension> = {}): Extension {
};
}
+// Simulate what real webpack does when __webpack_init_sharing__ runs: it
+// registers the host's own eagerly-provided @apache-superset/core versions
+// into the share scope.
+function mockWebpackSharing(coreVersions: Record<string, unknown> = {}) {
+ (globalThis as any).__webpack_init_sharing__ = jest
+ .fn()
+ .mockImplementation(() => {
+ (globalThis as any).__webpack_share_scopes__.default[
+ '@apache-superset/core'
+ ] = coreVersions;
+ });
+ (globalThis as any).__webpack_share_scopes__ = { default: {} };
+}
+
+function cleanupWebpackSharing() {
+ delete (globalThis as any).__webpack_init_sharing__;
+ delete (globalThis as any).__webpack_share_scopes__;
+}
+
+// Intercept script element creation so onload fires synchronously in jsdom.
+function mockRemoteEntryLoad() {
+ return jest
+ .spyOn(document.head, 'appendChild')
+ .mockImplementation((el: Node) => {
+ if (el instanceof HTMLScriptElement && el.onload) {
+ (el as HTMLScriptElement).onload!(new Event('load'));
+ }
+ return el;
+ });
+}
+
beforeEach(() => {
(ExtensionsLoader as any).instance = undefined;
mockApplicationRoot.mockReturnValue('');
+ // Reset window.superset to a base object before each test, including a
+ // shared singleton (commands) to verify identity is preserved across
+ // per-extension scoped copies.
+ (window as any).superset = {
Review Comment:
**Suggestion:** Introduce a typed shape for `window.superset` in tests and assign through that type instead of casting `window` to `any`. [custom_rule]
**Severity Level:** Minor π§Ή
<details>
<summary><b>Why it matters? β </b></summary>
The code casts `window` to `any` to assign `superset`, which is a concrete instance of the banned `any` usage.
</details>
[](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=75ce581413a7423789d2acb975d6fc56&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=75ce581413a7423789d2acb975d6fc56&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset)
*(Use Cmd/Ctrl + Click for best experience)*
<details>
<summary><b>Prompt for AI Agent π€ </b></summary>
```mdx
This is a comment left during a code review.
**Path:** superset-frontend/src/extensions/ExtensionsLoader.test.ts
**Line:** 87:87
**Comment:**
*Custom Rule: Introduce a typed shape for `window.superset` in tests and assign through that type instead of casting `window` to `any`.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F39171&comment_hash=26b689b38f334625b3a5aef302bc7d066b456b1752906cd4fa19f16211582939&reaction=like'>π</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F39171&comment_hash=26b689b38f334625b3a5aef302bc7d066b456b1752906cd4fa19f16211582939&reaction=dislike'>π</a>
##########
superset-frontend/src/extensions/ExtensionsLoader.test.ts:
##########
@@ -170,6 +213,212 @@ test('logs success after initializeExtensions completes', async () => {
infoSpy.mockRestore();
});
+test('logs partial failure count when some extensions fail to initialize', async () => {
+ const loader = ExtensionsLoader.getInstance();
+ const infoSpy = jest.spyOn(logging, 'info').mockImplementation();
+ jest.spyOn(SupersetClient, 'get').mockResolvedValue({
+ json: {
+ result: [
+ createMockExtension({ id: 'good-ext', remoteEntry: '' }),
+ createMockExtension({
+ id: 'broken-ext',
+ remoteEntry: 'http://broken-url/remoteEntry.js',
+ }),
+ ],
+ },
+ } as any);
+ const appendChildSpy = jest
+ .spyOn(document.head, 'appendChild')
+ .mockImplementation((element: Node) => {
+ if (element instanceof HTMLScriptElement && element.onerror) {
+ setTimeout(() => {
+ (element.onerror as any)('Script load error');
+ }, 0);
+ }
+ return element;
+ });
+
+ await loader.initializeExtensions();
+
+ expect(infoSpy).toHaveBeenCalledWith(
+ expect.stringContaining('1 of 2 extension(s) failing'),
+ );
+ expect(loader.getExtension('good-ext')).toBeDefined();
+ expect(loader.getExtension('broken-ext')).toBeUndefined();
+
+ infoSpy.mockRestore();
+ appendChildSpy.mockRestore();
+});
+
+test('each extension gets an isolated getContext via module federation custom scope', async () => {
+ const loader = ExtensionsLoader.getInstance();
+
+ // Capture the scoped @apache-superset/core instance each container receives
+ const capturedCoreModules: Array<any> = [];
Review Comment:
**Suggestion:** Replace the `Array<any>` declaration with a specific module/interface type that matches the captured core module shape. [custom_rule]
**Severity Level:** Minor π§Ή
<details>
<summary><b>Why it matters? β </b></summary>
`Array<any>` is explicitly using `any` in a new TypeScript declaration, so it violates the rule.
</details>
[](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=179d02a8697a47949534a10869d84c44&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=179d02a8697a47949534a10869d84c44&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset)
*(Use Cmd/Ctrl + Click for best experience)*
<details>
<summary><b>Prompt for AI Agent π€ </b></summary>
```mdx
This is a comment left during a code review.
**Path:** superset-frontend/src/extensions/ExtensionsLoader.test.ts
**Line:** 257:257
**Comment:**
*Custom Rule: Replace the `Array<any>` declaration with a specific module/interface type that matches the captured core module shape.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F39171&comment_hash=9550f2e19c4abe29a972ceb1f20a77f6bc409168283c7051fb366785a036c819&reaction=like'>π</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F39171&comment_hash=9550f2e19c4abe29a972ceb1f20a77f6bc409168283c7051fb366785a036c819&reaction=dislike'>π</a>
##########
superset-frontend/src/extensions/ExtensionsLoader.test.ts:
##########
@@ -48,9 +47,53 @@ function createMockExtension(overrides: Partial<Extension> = {}): Extension {
};
}
+// Simulate what real webpack does when __webpack_init_sharing__ runs: it
+// registers the host's own eagerly-provided @apache-superset/core versions
+// into the share scope.
+function mockWebpackSharing(coreVersions: Record<string, unknown> = {}) {
+ (globalThis as any).__webpack_init_sharing__ = jest
Review Comment:
**Suggestion:** Replace the untyped global cast with a dedicated typed interface for the webpack sharing globals instead of using `any`. [custom_rule]
**Severity Level:** Minor π§Ή
<details>
<summary><b>Why it matters? β </b></summary>
The file uses `any` in a TypeScript test via a cast on `globalThis`, which directly violates the no-`any` rule.
</details>
[](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=232b9cca86164a15a6f002497a871ce9&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=232b9cca86164a15a6f002497a871ce9&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset)
*(Use Cmd/Ctrl + Click for best experience)*
<details>
<summary><b>Prompt for AI Agent π€ </b></summary>
```mdx
This is a comment left during a code review.
**Path:** superset-frontend/src/extensions/ExtensionsLoader.test.ts
**Line:** 54:54
**Comment:**
*Custom Rule: Replace the untyped global cast with a dedicated typed interface for the webpack sharing globals instead of using `any`.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F39171&comment_hash=e1065c815dddda2652f35488939ec8efbd9f0d3d4e048330dfe01a22cd0b95ef&reaction=like'>π</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F39171&comment_hash=e1065c815dddda2652f35488939ec8efbd9f0d3d4e048330dfe01a22cd0b95ef&reaction=dislike'>π</a>
##########
superset-frontend/src/extensions/ExtensionsLoader.test.ts:
##########
@@ -170,6 +213,212 @@ test('logs success after initializeExtensions completes', async () => {
infoSpy.mockRestore();
});
+test('logs partial failure count when some extensions fail to initialize', async () => {
+ const loader = ExtensionsLoader.getInstance();
+ const infoSpy = jest.spyOn(logging, 'info').mockImplementation();
+ jest.spyOn(SupersetClient, 'get').mockResolvedValue({
+ json: {
+ result: [
+ createMockExtension({ id: 'good-ext', remoteEntry: '' }),
+ createMockExtension({
+ id: 'broken-ext',
+ remoteEntry: 'http://broken-url/remoteEntry.js',
+ }),
+ ],
+ },
+ } as any);
+ const appendChildSpy = jest
+ .spyOn(document.head, 'appendChild')
+ .mockImplementation((element: Node) => {
+ if (element instanceof HTMLScriptElement && element.onerror) {
+ setTimeout(() => {
+ (element.onerror as any)('Script load error');
+ }, 0);
+ }
+ return element;
+ });
+
+ await loader.initializeExtensions();
+
+ expect(infoSpy).toHaveBeenCalledWith(
+ expect.stringContaining('1 of 2 extension(s) failing'),
+ );
+ expect(loader.getExtension('good-ext')).toBeDefined();
+ expect(loader.getExtension('broken-ext')).toBeUndefined();
+
+ infoSpy.mockRestore();
+ appendChildSpy.mockRestore();
+});
+
+test('each extension gets an isolated getContext via module federation custom scope', async () => {
+ const loader = ExtensionsLoader.getInstance();
+
+ // Capture the scoped @apache-superset/core instance each container receives
+ const capturedCoreModules: Array<any> = [];
+
+ const makeContainer = () => ({
+ init: jest.fn().mockImplementation(async (scope: any) => {
Review Comment:
**Suggestion:** Type the `scope` argument with an explicit module-federation scope interface rather than `any`. [custom_rule]
**Severity Level:** Minor π§Ή
<details>
<summary><b>Why it matters? β </b></summary>
The callback parameter is typed as `any`, which is exactly what the custom rule disallows in changed TypeScript code.
</details>
[](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=80b8137f6347478eb35b976330d3d57b&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=80b8137f6347478eb35b976330d3d57b&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset)
*(Use Cmd/Ctrl + Click for best experience)*
<details>
<summary><b>Prompt for AI Agent π€ </b></summary>
```mdx
This is a comment left during a code review.
**Path:** superset-frontend/src/extensions/ExtensionsLoader.test.ts
**Line:** 260:260
**Comment:**
*Custom Rule: Type the `scope` argument with an explicit module-federation scope interface rather than `any`.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F39171&comment_hash=31873b271f1a0119ac0e45ee07196eeefd2e593e4d664b9d4711c312b9b015c0&reaction=like'>π</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F39171&comment_hash=31873b271f1a0119ac0e45ee07196eeefd2e593e4d664b9d4711c312b9b015c0&reaction=dislike'>π</a>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(extensions): add a Storage API for extensions [superset]
Posted by "michael-s-molina (via GitHub)" <gi...@apache.org>.
michael-s-molina commented on code in PR #39171:
URL: https://github.com/apache/superset/pull/39171#discussion_r3560955503
##########
docs/developer_docs/extensions/storage.md:
##########
@@ -0,0 +1,348 @@
+---
+title: Storage
+sidebar_position: 8
+---
+
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Storage
+
+Superset Extensions have access to a managed storage API for persisting data. The storage system provides multiple tiers with different persistence characteristics, allowing extensions to choose the right storage for their needs.
+
+Each extension receives its own isolated storage namespace. When Superset loads your extension, it binds storage to your extension's unique identifier, ensuring data privacyβtwo extensions using the same key will never collide, and extensions cannot access each other's data.
+
+Storage is always accessed through the extension context via `getContext()`. This binding to the context is what ties every operation to the current extension's namespace.
+
+## Storage Tiers
+
+| Tier | Storage Type | Context Property | Use Case |
+| ---- | ----------------- | ------------------------------------------ | -------------------------------------- |
+| 1 | Browser storage | `ctx.storage.local`, `ctx.storage.session` | UI state, wizard progress, draft forms |
+| 2 | Server-side cache | `ctx.storage.ephemeral` | Job progress, temporary results |
+| 3 | Database | `ctx.storage.persistent` | Saved artifacts, durable config |
+
+## Tier 1: Local State
+
+Browser-based storage that persists on the user's device. Use this for UI state and settings that don't need to sync across devices.
+
+### Why Use the API Instead of localStorage Directly?
+
+You might wonder why extensions should use `ctx.storage.local` instead of directly accessing `window.localStorage`. The managed API provides several benefits:
+
+- **Automatic namespacing**: Each extension's data is isolated. Two extensions using the same key name won't collide.
+- **User isolation**: By default, data is scoped to the current user, preventing data leakage between users on shared devices.
+- **Clean uninstall**: When an extension is uninstalled, all its data can be cleanly removed using prefix-based deletion.
+- **Future sandboxing**: The async API is designed for a future sandboxed execution model where extensions run in isolated contexts without direct DOM access.
+- **Consistent patterns**: The same API shape works across all storage tiers, making it easy to switch between them.
+
+### localState
+
+Data persists across browser sessions until explicitly deleted or the user clears browser storage.
+
+```typescript
+import { extensions } from '@apache-superset/core';
+
+const ctx = extensions.getContext();
+
+// Save sidebar state
+await ctx.storage.local.set('sidebar_collapsed', true);
+
+// Retrieve it later
+const isCollapsed = await ctx.storage.local.get('sidebar_collapsed');
+
+// Remove it
+await ctx.storage.local.remove('sidebar_collapsed');
+```
+
+### sessionState
+
+Data is cleared when the browser tab is closed. Use for transient state within a single session.
+
+```typescript
+import { extensions } from '@apache-superset/core';
+
+const ctx = extensions.getContext();
+
+// Save wizard progress (lost when tab closes)
+await ctx.storage.session.set('wizard_step', 3);
+await ctx.storage.session.set('unsaved_form', { name: 'Draft' });
+
+// Retrieve on page reload within same tab
+const step = await ctx.storage.session.get('wizard_step');
+```
+
+Tier 1 has no `shared` accessor. Browser storage is per-device, so a "shared" value would only be visible to other users of that same browser, not to other users of the extension generally β see `.shared` on [Tier 2](#tier-2-ephemeral-state) or [Tier 3](#tier-3-persistent-state) for storage that's actually shared across users.
+
+### When to Use Tier 1
+
+- UI state (sidebar collapsed, panel sizes)
+- Recently used items
+- Draft form values
+- Any data acceptable to lose if user clears browser
+
+### Limitations
+
+- Per-browser, per-device (not shared across devices)
+- Subject to browser storage quotas (~5-10 MB)
+- Not accessible from backend code
+
+## Tier 2: Ephemeral State
+
+Server-side cache storage with automatic TTL expiration. Use for temporary data that needs to be shared between frontend and backend, or persist across page reloads.
+
+TTL is required for every `set` call. The maximum allowed TTL is controlled by `MAX_TTL` in the server configuration.
+
+### Frontend Usage
+
+```typescript
+import { extensions } from '@apache-superset/core';
+
+const ctx = extensions.getContext();
+
+// Store with a 5-minute TTL
+await ctx.storage.ephemeral.set('job_progress', { pct: 42, status: 'running' }, { ttl: 300 });
+
+// Retrieve
+const progress = await ctx.storage.ephemeral.get('job_progress');
+
+// Remove
+await ctx.storage.ephemeral.remove('job_progress');
+```
+
+### Backend Usage
+
+```python
+from superset_core.extensions.context import get_context
+from superset_core.extensions.storage.ephemeral import EphemeralSetOptions
+
+ctx = get_context()
+
+# Store job progress with a 5-minute TTL
+ctx.storage.ephemeral.set(
+ 'job_progress', {'pct': 42, 'status': 'running'}, EphemeralSetOptions(ttl=300)
+)
+
+# Retrieve
+progress = ctx.storage.ephemeral.get('job_progress')
+
+# Remove
+ctx.storage.ephemeral.remove('job_progress')
+```
+
+### Shared State
+
+For data that needs to be visible to all users:
+
+```typescript
+import { extensions } from '@apache-superset/core';
+
+const ctx = extensions.getContext();
+
+await ctx.storage.ephemeral.shared.set('shared_result', { data: [1, 2, 3] }, { ttl: 3600 });
+const result = await ctx.storage.ephemeral.shared.get('shared_result');
+```
+
+```python
+from superset_core.extensions.context import get_context
+from superset_core.extensions.storage.ephemeral import EphemeralSetOptions
+
+ctx = get_context()
+
+ctx.storage.ephemeral.shared.set(
+ 'shared_result', {'data': [1, 2, 3]}, EphemeralSetOptions(ttl=3600)
+)
+result = ctx.storage.ephemeral.shared.get('shared_result')
+```
+
+### When to Use Tier 2
+
+- Background job progress indicators
+- Cross-request intermediate state
+- Query result previews
+- Temporary computation results
+- Any data that can be recomputed if lost
+
+### Limitations
+
+- Not guaranteed to survive server restarts
+- Subject to cache eviction under memory pressure
+- TTL-based expiration (data disappears after timeout)
+- TTL is required on every `set` call
+
+## Tier 3: Persistent State
+
+Database-backed storage that survives server restarts, cache evictions, and browser clears. Use for any data that must not be lost.
+
+### Frontend Usage
+
+```typescript
+import { extensions } from '@apache-superset/core';
+
+const ctx = extensions.getContext();
+
+// Store a saved SQL snippet
+await ctx.storage.persistent.set('snippet:top_customers', { sql: 'SELECT ...' });
+
+// Retrieve
+const snippet = await ctx.storage.persistent.get('snippet:top_customers');
+
+// Remove
+await ctx.storage.persistent.remove('snippet:top_customers');
+```
+
+### Backend Usage
+
+```python
+from superset_core.extensions.context import get_context
+
+ctx = get_context()
+
+# Store a saved SQL snippet
+ctx.storage.persistent.set('snippet:top_customers', {'sql': 'SELECT ...'})
+
+# Retrieve
+snippet = ctx.storage.persistent.get('snippet:top_customers')
+
+# Remove
+ctx.storage.persistent.remove('snippet:top_customers')
+```
+
+### Shared State
+
+For data that should be visible to all users of the extension:
+
+```typescript
+import { extensions } from '@apache-superset/core';
+
+const ctx = extensions.getContext();
+
+await ctx.storage.persistent.shared.set('global_config', { version: 2 });
+const config = await ctx.storage.persistent.shared.get('global_config');
+```
+
+```python
+from superset_core.extensions.context import get_context
+
+ctx = get_context()
+
+ctx.storage.persistent.shared.set('global_config', {'version': 2})
+config = ctx.storage.persistent.shared.get('global_config')
+```
+
+### Enumerating and Managing Storage from the Backend
+
+The `ctx.storage.persistent` accessor is for single-key reads and writes. Backend extension code that needs to enumerate its own rows in bulk β for example, a cleanup routine that removes storage linked to resources that no longer exist β can use `superset_core.extensions.storage.dao.ExtensionStorageDAO` instead:
+
+```python
+from superset_core.extensions.storage.dao import ExtensionStorageDAO
+
+# All entries this extension has linked to a given resource type
+entries = ExtensionStorageDAO.filter_by(resource_type="my-resource-type")
+
+# Bulk-delete rows for resources that no longer exist
+orphaned = [e for e in entries if e.resource_uuid not in my_extension_live_resource_ids()]
+ExtensionStorageDAO.delete(orphaned)
+```
+
+`ExtensionStorageDAO` is automatically scoped to the calling extension's own rows β `extension_id` is never a parameter you pass, so there is no way to reach another extension's storage through this API. It supports the standard DAO operations (`find_all`, `find_one_or_none`, `filter_by`, `query`, `create`, `update`, `delete`), and, unlike `ctx.storage.persistent`, gives direct access to each row's scope columns (`resource_type`, `resource_uuid`, `category`, `description`) for entries created with those set.
+
+### When to Use Tier 3
+
+- Extension configuration that must survive restarts
+- User-specific saved artifacts that need to roam across devices and browsers (e.g. saved SQL snippets, annotations)
+- Any data where loss is unacceptable
+
+### Limitations
+
+- Higher latency than Tiers 1β2 (database round-trip per operation)
+- Subject to the 16 MB value size limit
+- Subject to the per-extension quota configured via `EXTENSIONS_PERSISTENT_STORAGE.QUOTA_PER_EXTENSION` (see [Quotas](#quotas))
+- Requires a database migration when first deployed
+
+## Key Patterns
+
+All storage keys are automatically namespaced:
+
+| Scope | Key Pattern |
+| ---------------------- | --------------------------------------------------- |
+| User-scoped | `superset-ext:{extension_id}:user:{user_id}:{key}` |
+| Shared (Tiers 2 and 3) | `superset-ext:{extension_id}:shared:{key}` |
Review Comment:
Fixed
##########
superset-frontend/packages/superset-core/src/common/index.ts:
##########
@@ -227,8 +227,6 @@ export interface Extension {
id: string;
/** Human-readable name of the extension */
name: string;
- /** URL or path to the extension's remote entry point */
- remoteEntry: string;
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(extensions): add a Storage API for extensions [superset]
Posted by "michael-s-molina (via GitHub)" <gi...@apache.org>.
michael-s-molina commented on code in PR #39171:
URL: https://github.com/apache/superset/pull/39171#discussion_r3561374091
##########
superset-frontend/packages/superset-core/src/extensions/index.ts:
##########
@@ -24,9 +24,80 @@
* including querying extension metadata and monitoring extension lifecycle events.
* Extensions can use this API to discover other extensions and react to changes
* in the extension ecosystem.
+ *
+ * Extensions can access their own context via `getContext()`, which provides:
+ * - Extension metadata (id, name, version, etc.)
+ * - Extension-scoped storage (localStorage, sessionStorage, ephemeral cache)
+ *
+ * @example
+ * ```typescript
+ * import { extensions } from '@apache-superset/core';
+ *
+ * // Get the current extension's context
+ * const ctx = extensions.getContext();
+ *
+ * // Access extension metadata
+ * console.log(`Running ${ctx.extension.name} v${ctx.extension.version}`);
+ *
+ * // Access extension-scoped storage
+ * await ctx.storage.local.set('preference', { theme: 'dark' });
+ * await ctx.storage.ephemeral.set('cache', data, { ttl: 300 });
+ * ```
*/
import { Extension } from '../common';
+import { ExtensionStorage } from '../storage';
+
+/**
+ * Context object providing extension-specific resources.
+ *
+ * This context is only available during extension execution.
+ * Calling `getContext()` outside of an extension will throw an error.
+ */
+export interface ExtensionContext {
+ /**
+ * Metadata about the current extension.
+ */
+ extension: Extension;
+
+ /**
+ * Extension-scoped storage across all tiers.
+ * All keys are automatically namespaced to prevent collisions.
+ */
+ storage: ExtensionStorage;
+}
+
+/**
+ * Get the current extension's context.
+ *
+ * This function returns the context for the currently executing extension,
+ * providing access to extension metadata and scoped resources like storage.
+ *
+ * @returns The current extension's context.
+ * @throws Error if called outside of an extension context.
+ *
+ * @example
+ * ```typescript
+ * import { extensions } from '@apache-superset/core';
+ *
+ * const ctx = extensions.getContext();
+ *
+ * // Access extension metadata
+ * console.log(`Extension: ${ctx.extension.id}`);
+ * console.log(`Version: ${ctx.extension.version}`);
+ *
+ * // Access extension-scoped storage
+ * await ctx.storage.local.set('userPref', { sidebar: 'collapsed' });
+ * const pref = await ctx.storage.local.get('userPref');
+ *
+ * // Use ephemeral storage with TTL
+ * await ctx.storage.ephemeral.set('tempData', data, { ttl: 3600 });
+ *
+ * // Access shared (cross-user) storage
+ * await ctx.storage.ephemeral.shared.set('globalCounter', count);
Review Comment:
Fixed β the example now passes the required `{ ttl: 3600 }` options argument to `.shared.set`.
##########
superset/extensions/storage/persistent_dao.py:
##########
@@ -0,0 +1,527 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from __future__ import annotations
+
+import hashlib
+import hmac
+import logging
+from dataclasses import dataclass
+from typing import Any
+
+from flask import current_app
+from flask_babel import gettext as _
+from sqlalchemy import and_, desc, func, LargeBinary
+
+from superset import db
+from superset.daos.base import BaseDAO
+from superset.exceptions import SupersetGenericErrorException
+from superset.extensions.storage.codecs import DEFAULT_CODEC, get_codec
+from superset.extensions.storage.filters import ExtensionStorageFilter
+from superset.extensions.storage.persistent_model import ExtensionStorage
+from superset.utils.encrypt import (
+ DEFAULT_ENCRYPTION_ENGINE_NAME,
+ EncryptedType,
+ resolve_encryption_engine,
+)
+
+logger = logging.getLogger(__name__)
+
+
+class ExtensionStorageQuotaExceeded(SupersetGenericErrorException):
+ """Raised when a write would exceed an extension's persistent storage quota."""
+
+ status = 413
+
+ def __init__(self, extension_id: str, quota: int) -> None:
+ super().__init__(
+ message=_(
+ "Extension '%(extension_id)s' has exceeded its persistent "
+ "storage quota of %(quota)d bytes.",
+ extension_id=extension_id,
+ quota=quota,
+ ),
+ )
+
+
+class ExtensionStorageValueTooLarge(SupersetGenericErrorException):
+ """Raised when a `set` call's value exceeds MAX_VALUE_SIZE."""
+
+ status = 400
+
+ def __init__(self, max_size: int) -> None:
+ super().__init__(
+ message=_(
+ "Value exceeds the maximum allowed size of %(max_size)d bytes.",
+ max_size=max_size,
+ ),
+ )
+
+
+class ExtensionStorageListPayloadTooLarge(SupersetGenericErrorException):
+ """Raised when a `list` page's combined value size exceeds
+ MAX_LIST_PAYLOAD_SIZE."""
+
+ status = 400
+
+ def __init__(self, requested_size: int, max_size: int) -> None:
+ super().__init__(
+ message=_(
+ "Requested page would return %(requested_size)d bytes, "
+ "exceeding the maximum allowed list payload size of "
+ "%(max_size)d bytes. Reduce page_size and try again.",
+ requested_size=requested_size,
+ max_size=max_size,
+ ),
+ )
+
+
+def _derive_key(secret: str, user_fk: int) -> str:
+ """Derive a per-user encryption key from the master secret and user ID.
+
+ Uses HMAC-SHA256 so the derived key is cryptographically bound to both.
+ Ciphertext encrypted for one user cannot be decrypted as another.
+ """
+ return hmac.new(
+ secret.encode(),
+ str(user_fk).encode(),
+ hashlib.sha256,
+ ).hexdigest()
+
+
+def _enc_type(user_fk: int | None) -> EncryptedType:
+ """Return an EncryptedType for the given scope.
+
+ User-scoped rows use a key derived from SECRET_KEY and the user ID, so
+ ciphertext written for one user cannot be decrypted as another.
+ Shared rows (user_fk=None) use SECRET_KEY directly.
+ """
+ secret = current_app.config["SECRET_KEY"]
+ key = _derive_key(secret, user_fk) if user_fk is not None else secret
+ engine_name = current_app.config.get(
+ "SQLALCHEMY_ENCRYPTED_FIELD_ENGINE", DEFAULT_ENCRYPTION_ENGINE_NAME
+ )
+ return EncryptedType(
+ LargeBinary,
+ key=key,
+ engine=resolve_encryption_engine(engine_name),
+ )
+
+
+def _get_quota() -> int | None:
+ """Return the configured per-extension persistent storage quota in bytes.
+
+ Returns None when no quota is configured (unlimited).
+ """
+ config = current_app.config["EXTENSIONS_PERSISTENT_STORAGE"]
+ return config.get("QUOTA_PER_EXTENSION")
+
+
+def _get_max_list_payload_size() -> int | None:
+ """Return the configured max combined value size for a `list()` page.
+
+ Returns None when no limit is configured (unlimited).
+ """
+ config = current_app.config["EXTENSIONS_PERSISTENT_STORAGE"]
+ return config.get("MAX_LIST_PAYLOAD_SIZE")
+
+
+def _validate_value_size(encoded: bytes) -> None:
+ """Validate encoded value bytes against MAX_VALUE_SIZE. Raises if over.
+
+ Enforced independently of `QUOTA_PER_EXTENSION`: the quota bounds an
+ extension's total storage, while this bounds a single value. A
+ `list_entries()` page's combined size is bounded separately by
+ MAX_LIST_PAYLOAD_SIZE (see `ExtensionStorageDAO.list_entries`).
+ """
+ config = current_app.config["EXTENSIONS_PERSISTENT_STORAGE"]
+ max_size = config.get("MAX_VALUE_SIZE")
+ if max_size is None:
+ return
+ if len(encoded) > max_size:
+ raise ExtensionStorageValueTooLarge(max_size)
+
+
+def _check_quota(extension_id: str, new_size: int, existing_size: int = 0) -> None:
+ """Raise if writing `new_size` bytes would push the extension over quota.
+
+ `existing_size` is the size of the row being replaced (0 for inserts),
+ subtracted from current usage so overwriting a key doesn't double-count
+ it against the quota it already occupies.
+
+ The usage query sums the `value_size` column via the covering
+ `ix_ext_storage_extension_id` index (on `extension_id, value_size`)
+ instead of `LENGTH(value)`, so it can run as an index-only scan rather
+ than reading every one of the extension's stored (and potentially
+ TOASTed) blobs. It is still O(rows) for the extension, so write latency
+ grows with row count β acceptable at the row counts a single extension
+ is expected to accumulate, but not a fully constant-time check.
+ """
+ quota = _get_quota()
+ if quota is None:
+ return
+ current_usage = (
+ db.session.query(func.coalesce(func.sum(ExtensionStorage.value_size), 0))
+ .filter(ExtensionStorage.extension_id == extension_id)
+ .scalar()
+ )
+ if current_usage - existing_size + new_size > quota:
+ raise ExtensionStorageQuotaExceeded(extension_id, quota)
+
+
+def _decrypt_if_needed(
+ entry: ExtensionStorage, extension_id: str, key: str
+) -> bytes | None:
+ """Return `entry.value`, decrypted if `entry.is_encrypted`.
+
+ Returns None (and logs) if decryption fails, e.g. after key rotation.
+ """
+ if not entry.is_encrypted:
+ return entry.value
+ try:
+ return _enc_type(entry.user_fk).process_result_value(
+ entry.value, db.engine.dialect
+ )
+ except Exception: # noqa: BLE001
+ logger.error(
+ "Failed to decrypt extension storage value for "
+ "extension_id=%s key=%s β possible key rotation issue",
+ extension_id,
+ key,
+ )
+ return None
+
+
+def _scope_filter(
+ extension_id: str,
+ key: str,
+ user_fk: int | None = None,
+ resource_type: str | None = None,
+ resource_uuid: str | None = None,
+) -> list[object]:
+ """Build the SQLAlchemy filter list for a scoped lookup."""
+ filters: list[object] = [
+ ExtensionStorage.extension_id == extension_id,
+ ExtensionStorage.key == key,
+ ]
+ if user_fk is None:
+ filters.append(ExtensionStorage.user_fk.is_(None))
+ else:
+ filters.append(ExtensionStorage.user_fk == user_fk)
+ if resource_type is None:
+ filters.append(ExtensionStorage.resource_type.is_(None))
+ else:
+ filters.append(ExtensionStorage.resource_type == resource_type)
+ if resource_uuid is None:
+ filters.append(ExtensionStorage.resource_uuid.is_(None))
+ else:
+ filters.append(ExtensionStorage.resource_uuid == resource_uuid)
+ return filters
+
+
+def _list_scope_filter(
+ extension_id: str,
+ user_fk: int | None = None,
+ resource_type: str | None = None,
+ resource_uuid: str | None = None,
+) -> list[object]:
+ """Build the SQLAlchemy filter list for a `list()` call.
+
+ Same scope semantics as `_scope_filter` (an explicit None means "match
+ the global/unset scope"), minus the `key` filter.
+ """
+ filters: list[object] = [ExtensionStorage.extension_id == extension_id]
+ if user_fk is None:
+ filters.append(ExtensionStorage.user_fk.is_(None))
+ else:
+ filters.append(ExtensionStorage.user_fk == user_fk)
+ if resource_type is None:
+ filters.append(ExtensionStorage.resource_type.is_(None))
+ else:
+ filters.append(ExtensionStorage.resource_type == resource_type)
+ if resource_uuid is None:
+ filters.append(ExtensionStorage.resource_uuid.is_(None))
+ else:
+ filters.append(ExtensionStorage.resource_uuid == resource_uuid)
+ return filters
+
+
+@dataclass
+class ExtensionStorageListEntry:
+ """A single row returned by `ExtensionStorageDAO.list_entries`.
+
+ `value` is the raw (decrypted) payload, mirroring `get_value()` β it is
+ not decoded with `codec` here, since REST callers must first check
+ `codec` against `SAFE_CODECS` before deciding whether decoding it is
+ safe to expose over the API, while ambient backend callers (no such
+ restriction) decode unconditionally, same as `get_decoded_value()`.
+ """
+
+ key: str
+ value: bytes | None
+ codec: str
+ is_encrypted: bool
+
+
+class ExtensionStorageDAO(BaseDAO[ExtensionStorage]):
+ """Persistent key-value store for extensions (Tier 3).
+
+ Every query and mutation is scoped to the extension currently executing
+ (resolved from ambient extension context by `ExtensionStorageFilter`),
+ so an extension can never read or modify another extension's storage
+ through this DAO.
+
+ Two access shapes are supported:
+
+ * `get`/`set`/`delete_by_key`/`list_entries` β single-key or
+ paginated-listing operations with explicit scope discriminators
+ (`user_fk`, `resource_type`, `resource_uuid`), used by the ambient
+ `persistent_state`/`ephemeral_state` accessors and the REST API (both
+ of which establish the ambient extension context themselves before
+ calling in).
+ * `find_all`/`find_one_or_none`/`filter_by`/`query`/`create`/`update`/
+ `delete` (inherited from `BaseDAO`) β general-purpose DAO access for
+ backend extension code that needs to enumerate or bulk-manage its own
+ rows, exposed to extensions via `superset_core.extensions.storage.dao`.
+
+ Scope is one of:
+
+ * Global scope β user_fk=None, resource_type=None
+ * User scope β user_fk=<id>, resource_type=None
+ * Resource scope β resource_type + resource_uuid set
+ """
+
+ base_filter = ExtensionStorageFilter
+
+ # ββ Read βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
+
+ @staticmethod
+ def get(
+ extension_id: str,
+ key: str,
+ user_fk: int | None = None,
+ resource_type: str | None = None,
+ resource_uuid: str | None = None,
+ ) -> ExtensionStorage | None:
+ """Return the raw storage entry. The value field may be encrypted."""
+ entry = (
+ db.session.query(ExtensionStorage)
+ .filter(
+ and_(
+ *_scope_filter(
+ extension_id, key, user_fk, resource_type, resource_uuid
+ )
+ )
+ )
+ .first()
+ )
+ return entry
+
+ @staticmethod
+ def get_value(
+ extension_id: str,
+ key: str,
+ user_fk: int | None = None,
+ resource_type: str | None = None,
+ resource_uuid: str | None = None,
+ ) -> bytes | None:
+ """Return the raw (decrypted) value bytes, or None if not found."""
+ entry = ExtensionStorageDAO.get(
+ extension_id, key, user_fk, resource_type, resource_uuid
+ )
+ if entry is None:
+ return None
+ return _decrypt_if_needed(entry, extension_id, key)
+
+ @staticmethod
+ def get_decoded_value(
+ extension_id: str,
+ key: str,
+ user_fk: int | None = None,
+ resource_type: str | None = None,
+ resource_uuid: str | None = None,
+ ) -> Any:
+ """Return the value decoded with the codec it was written with.
+
+ :returns: The decoded value, or None if not found.
+ """
+ entry = ExtensionStorageDAO.get(
+ extension_id, key, user_fk, resource_type, resource_uuid
+ )
+ if entry is None:
+ return None
+ raw = _decrypt_if_needed(entry, extension_id, key)
+ if raw is None:
+ return None
+ return get_codec(entry.codec).decode(raw)
+
+ @staticmethod
+ def list_entries(
+ extension_id: str,
+ user_fk: int | None = None,
+ resource_type: str | None = None,
+ resource_uuid: str | None = None,
+ page: int = 0,
+ page_size: int = 10,
+ ) -> tuple[list[ExtensionStorageListEntry], int]:
+ """List entries in the given scope, most recently changed first.
+
+ Named `list_entries` rather than `list` (unlike the single-key
+ `get`/`set`/`delete_by_key` shape) because `BaseDAO` already
+ defines a generic `list()` with a different, incompatible
+ signature (search/sort/column-selection over arbitrary models);
+ overriding it here would break that contract for callers relying
+ on polymorphic DAO access.
+
+ Callers may request any `page_size` β there is no row-count
+ ceiling β but the combined `value_size` of the requested page is
+ checked against MAX_LIST_PAYLOAD_SIZE *before* the (potentially
+ large) `value` column is fetched for any row, so an oversized
+ request fails fast rather than reading data it will then reject.
+
+ :returns: (entries, total_count). `entries[i].value` is the raw
+ (decrypted) payload β see `ExtensionStorageListEntry`.
+ :raises ExtensionStorageListPayloadTooLarge: if the requested page's
+ combined value size exceeds MAX_LIST_PAYLOAD_SIZE.
+ """
+ filters = _list_scope_filter(
+ extension_id, user_fk, resource_type, resource_uuid
+ )
+ base_query = db.session.query(ExtensionStorage).filter(and_(*filters))
+ total_count = base_query.count()
+
+ page = max(page, 0)
+ page_size = max(page_size, 1)
+ page_query = (
+ base_query.order_by(desc(ExtensionStorage.changed_on))
+ .offset(page * page_size)
+ .limit(page_size)
+ )
+
+ if (max_payload_size := _get_max_list_payload_size()) is not None:
+ # Sum value_size for exactly this page's rows, without touching
+ # the `value` column, by reusing the same ordered/offset/limited
+ # query as a subquery.
+ subquery = page_query.with_entities(ExtensionStorage.value_size).subquery()
+ requested_size = db.session.query(
+ func.coalesce(func.sum(subquery.c.value_size), 0)
+ ).scalar()
+ if requested_size > max_payload_size:
+ raise ExtensionStorageListPayloadTooLarge(
+ requested_size, max_payload_size
+ )
+
+ entries = [
+ ExtensionStorageListEntry(
+ key=row.key,
+ value=_decrypt_if_needed(row, extension_id, row.key),
+ codec=row.codec,
+ is_encrypted=row.is_encrypted,
+ )
+ for row in page_query.all()
+ ]
+ return entries, total_count
+
+ # ββ Write (upsert) ββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
+
+ @staticmethod
+ def set(
+ extension_id: str,
+ key: str,
+ value: bytes,
+ codec: str = DEFAULT_CODEC,
+ user_fk: int | None = None,
+ resource_type: str | None = None,
+ resource_uuid: str | None = None,
+ encrypt: bool = False,
+ ) -> ExtensionStorage:
+ """Upsert a storage entry. Encrypts value when encrypt=True.
+
+ :raises ExtensionStorageValueTooLarge: if `value` exceeds
+ MAX_VALUE_SIZE.
+ :raises ExtensionStorageQuotaExceeded: if writing `value` would push
+ the extension over its total storage quota.
+ """
+ _validate_value_size(value)
+ stored_value = (
+ _enc_type(user_fk).process_bind_param(value, db.engine.dialect)
+ if encrypt
+ else value
+ )
+
+ entry = (
+ db.session.query(ExtensionStorage)
+ .filter(
+ and_(
+ *_scope_filter(
+ extension_id, key, user_fk, resource_type, resource_uuid
+ )
+ )
+ )
+ .first()
+ )
+ new_size = len(stored_value)
+ existing_size = entry.value_size if entry is not None else 0
+ _check_quota(extension_id, new_size, existing_size)
+ if entry is not None:
+ entry.value = stored_value
+ entry.value_size = new_size
+ entry.codec = codec
+ entry.is_encrypted = encrypt
+ else:
+ entry = ExtensionStorage(
+ extension_id=extension_id,
+ key=key,
+ value=stored_value,
+ value_size=new_size,
+ codec=codec,
+ user_fk=user_fk,
+ resource_type=resource_type,
+ resource_uuid=resource_uuid,
+ is_encrypted=encrypt,
+ )
+ db.session.add(entry)
+ db.session.flush()
+ return entry
Review Comment:
Fixed β added `_validate_key_length`/`ExtensionStorageKeyTooLong` at the DAO boundary (`ExtensionStorageDAO.set`), wired into the REST API's error handling. Also removed the now-redundant client-side length check in the frontend SDK so key-length validation lives in a single place.
##########
superset-frontend/src/core/storage/localState.ts:
##########
@@ -0,0 +1,63 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import type { JsonValue, StorageAccessor } from '@apache-superset/core/storage';
+import getBootstrapData from 'src/utils/getBootstrapData';
+
+const KEY_PREFIX = 'superset-ext';
+
+function getCurrentUserId(): number {
+ const bootstrapData = getBootstrapData();
+ const userId = bootstrapData?.user?.userId;
+ if (userId === undefined) {
+ throw new Error('Storage APIs require an authenticated user.');
+ }
+ return userId;
+}
+
+function buildKey(...parts: (string | number)[]): string {
+ return [KEY_PREFIX, ...parts].join(':');
+}
+
+/**
+ * Create browser storage (localStorage/sessionStorage) bound to an extension ID.
+ */
+export function createBrowserStorage(
+ storage: Storage,
+ extensionId: string,
+): StorageAccessor {
+ return {
+ async get<T = JsonValue>(key: string): Promise<T | null> {
+ const userId = getCurrentUserId();
+ const storageKey = buildKey(extensionId, 'user', userId, key);
+ const value = storage.getItem(storageKey);
+ return value ? (JSON.parse(value) as T) : null;
+ },
+ async set<T = JsonValue>(key: string, value: T): Promise<void> {
+ const userId = getCurrentUserId();
+ const storageKey = buildKey(extensionId, 'user', userId, key);
+ storage.setItem(storageKey, JSON.stringify(value));
Review Comment:
Fixed β `set()` now throws if the value isn't JSON-serializable, instead of silently writing the string `"undefined"` and breaking a later `get()`.
##########
superset-frontend/src/extensions/ExtensionContext.ts:
##########
@@ -0,0 +1,65 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+import type {
+ common,
+ extensions as extensionsApi,
+} from '@apache-superset/core';
+import {
+ createBrowserStorage,
+ createEphemeralState,
+ createPersistentState,
+} from 'src/core/storage';
+
+type Extension = common.Extension;
+type ExtensionContextType = extensionsApi.ExtensionContext;
+
+/**
+ * Extension context with lazy-initialized services bound to the extension ID.
+ */
+class ExtensionContext implements ExtensionContextType {
+ readonly extension: Extension;
+
+ private _storage?: ExtensionContextType['storage'];
+
+ constructor(extension: Extension) {
+ this.extension = extension;
+ }
+
+ get storage(): ExtensionContextType['storage'] {
+ if (!this._storage) {
+ const { id } = this.extension;
+ this._storage = {
+ local: createBrowserStorage(localStorage, id),
+ session: createBrowserStorage(sessionStorage, id),
+ ephemeral: createEphemeralState(id),
+ persistent: createPersistentState(id),
+ };
Review Comment:
Fixed β `local`/`session` are now exposed as lazy per-tier getters, so a blocked/unavailable Web Storage API no longer prevents access to `ephemeral`/`persistent`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(extensions): add a Storage API for extensions [superset]
Posted by "bito-code-review[bot] (via GitHub)" <gi...@apache.org>.
bito-code-review[bot] commented on PR #39171:
URL: https://github.com/apache/superset/pull/39171#issuecomment-4972905462
AI Code Review is in progress (usually takes 3 to 15 minutes unless it's a very large PR).
<details>
<summary>
Bito Usage Guide
</summary>
**Commands**
Type the following command in the pull request comment and save the comment.
- `/review` - Manually triggers a full AI review.
- `/pause` - Pauses automatic reviews on this pull request.
- `/resume` - Resumes automatic reviews.
- `/resolve` - Marks all Bito-posted review comments as resolved.
- `/abort` - Cancels all in-progress reviews.
Refer to the <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/available-commands" target="_blank" rel="noopener noreferrer">documentation</a> for additional commands.
**Configuration**
This repository uses `Superset` You can customize the agent settings <a href="https://alpha.bito.ai/home/ai-agents/code-review-agent" target="_blank" rel="noopener noreferrer">here</a> or contact your Bito workspace admin at evan@preset.io.
**Documentation & Help**
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/getting-started/install-run-using-bito-cloud/create-or-customize-an-agent-instance" target="_blank" rel="noopener noreferrer">Customize agent settings</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/implementing-custom-code-review-rules" target="_blank" rel="noopener noreferrer">Review rules</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent" target="_blank" rel="noopener noreferrer">General documentation</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/faqs" target="_blank" rel="noopener noreferrer">FAQ</a>
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(extensions): add Tier 1 and Tier 2 storage APIs [superset]
Posted by "michael-s-molina (via GitHub)" <gi...@apache.org>.
michael-s-molina commented on code in PR #39171:
URL: https://github.com/apache/superset/pull/39171#discussion_r3058615862
##########
superset-frontend/packages/superset-core/src/storage/ephemeralState.ts:
##########
@@ -0,0 +1,160 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import type { JsonValue, StorageAccessor } from './types';
+
+/**
+ * @fileoverview Ephemeral State API for Superset extensions (Tier 2 Storage).
+ *
+ * Provides short-lived KV storage backed by the configured server-side cache
+ * backend (Redis, Memcached, or filesystem). Automatically expires based on TTL.
+ * Not guaranteed to survive server restarts.
+ *
+ * By default, all operations are user-scoped (private to the current user).
+ * Use `shared` to access state that is visible to all users.
+ *
+ * Cache keys are namespaced automatically:
+ * - User-scoped (default): superset-ext:{extension_id}:user:{user_id}:{key}
+ * - Shared (global): superset-ext:{extension_id}:{key}
+ *
+ * @example
+ * ```typescript
+ * import { ephemeralState } from '@apache-superset/core/storage';
+ *
+ * // User-scoped state (default - private to current user)
+ * const progress = await ephemeralState.get('job_progress');
+ * await ephemeralState.set('job_progress', { pct: 42 }, { ttl: 3600 });
+ * await ephemeralState.remove('job_progress');
+ *
+ * // Shared state (explicit opt-in - visible to all users)
+ * const result = await ephemeralState.shared.get('shared_result');
+ * await ephemeralState.shared.set('shared_result', { data: [1, 2, 3] });
+ * await ephemeralState.shared.remove('shared_result');
+ * ```
+ */
+
+/**
+ * Default TTL in seconds (1 hour).
+ */
+export const DEFAULT_TTL = 3600;
Review Comment:
Good point. Addressed in [d407804](https://github.com/apache/superset/pull/39171/commits/d4078042c335c2f22206cff50988ae50f6899406)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(extensions): add Tier 1 and Tier 2 storage APIs [superset]
Posted by "michael-s-molina (via GitHub)" <gi...@apache.org>.
michael-s-molina commented on code in PR #39171:
URL: https://github.com/apache/superset/pull/39171#discussion_r3058629694
##########
superset/extensions/storage/__init__.py:
##########
@@ -15,10 +15,19 @@
# specific language governing permissions and limitations
# under the License.
-from .get_database_info import get_database_info
-from .list_databases import list_databases
+"""
+Host implementations for extension storage APIs.
+
+This module provides concrete implementations that are injected into
+superset_core.extensions.storage at startup.
+"""
+
+from superset.extensions.storage.ephemeral_state import (
+ EphemeralStateImpl,
+ SharedEphemeralStateAccessor,
+)
__all__ = [
- "list_databases",
- "get_database_info",
+ "EphemeralStateImpl",
+ "SharedEphemeralStateAccessor",
]
Review Comment:
Good catch! Removed in [65d7ecf](https://github.com/apache/superset/pull/39171/commits/65d7ecf5e9db288b9b45b1131be818f27eda43dc)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(extensions): add a Storage API for extensions [superset]
Posted by "michael-s-molina (via GitHub)" <gi...@apache.org>.
michael-s-molina commented on PR #39171:
URL: https://github.com/apache/superset/pull/39171#issuecomment-4833534643
@matebenyovszky There are other problems I'm still addressing such as handling isolated contexts per extension. I have local changes that I still intend to push. Also, I'll open a SIP once the prototype is ready.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(extensions): add Tier 1 and Tier 2 storage APIs [superset]
Posted by "codeant-ai-for-open-source[bot] (via GitHub)" <gi...@apache.org>.
codeant-ai-for-open-source[bot] commented on code in PR #39171:
URL: https://github.com/apache/superset/pull/39171#discussion_r3047374331
##########
superset-frontend/src/core/storage/ephemeralState.ts:
##########
@@ -0,0 +1,199 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/**
+ * Host implementation for Tier 2 Ephemeral State (Server Cache).
+ */
+
+import {
+ storage as storageApi,
+ type storage as StorageTypes,
+} from '@apache-superset/core';
+import { SupersetClient } from '@superset-ui/core';
+import { DEFAULT_TTL, getCurrentExtensionId } from './utils';
+
+/**
+ * Build the API URL for ephemeral state operations.
+ */
+function buildEphemeralStateUrl(
+ extensionId: string,
+ key: string,
+ isShared: boolean,
+): string {
+ const basePath = '/api/v1/extensions/storage/ephemeral';
+ return isShared
+ ? `${basePath}/shared/${extensionId}/${key}`
+ : `${basePath}/${extensionId}/${key}`;
Review Comment:
**Suggestion:** The URL is built with raw `extensionId` and `key`, so reserved URL characters in keys (for example `/`, `?`, `#`, `%`) will break routing or change the request path/query, causing get/set/remove to hit the wrong endpoint or fail with 404. Encode path segments before interpolation so all valid string keys work reliably. [logic error]
<details>
<summary><b>Severity Level:</b> Major β οΈ</summary>
```mdx
- β Ephemeral state API fails for keys with '/' or '?'
- β οΈ Extensions using hierarchical or URL-like keys lose state
- β οΈ Shared ephemeral state endpoints similarly break for such keys
- β οΈ Debugging is harder due to 404s despite valid inputs
```
</details>
```suggestion
const encodedExtensionId = encodeURIComponent(extensionId);
const encodedKey = encodeURIComponent(key);
return isShared
? `${basePath}/shared/${encodedExtensionId}/${encodedKey}`
: `${basePath}/${encodedExtensionId}/${encodedKey}`;
```
<details>
<summary><b>Steps of Reproduction β
</b></summary>
```mdx
1. Implement a Superset extension that uses the public storage API declared in
`superset-frontend/packages/superset-core/src/storage/ephemeralState.ts:95-149`, for
example calling `await ephemeralState.set('job/progress', { pct: 42 })` where the key
contains a `/` character.
2. Load the extension in Superset so that the host implementation from
`superset-frontend/src/core/storage/index.ts:32-49` is wired in; the extension's call to
`ephemeralState.set` is served by the host implementation in
`superset-frontend/src/core/storage/ephemeralState.ts:89-115`, which builds the REST URL
via `buildEphemeralStateUrl` at
`superset-frontend/src/core/storage/ephemeralState.ts:34-43` without encoding the `key`
argument.
3. Observe that the browser issues a request to an endpoint like
`/api/v1/extensions/storage/ephemeral/<extension_id>/job/progress` (the raw `key` is
interpolated as a path segment), rather than treating `'job/progress'` as a single path
parameter value.
4. On the server, the Flask route `@expose("/ephemeral/<extension_id>/<key>",
methods=("PUT",))` in `superset/extensions/storage/api.py:76-80` only matches two path
segments after `/ephemeral`; because the request path contains an extra `/progress`
segment and the `<key>` converter is the default `string` (which excludes `/`), no route
matches, the API returns HTTP 404, and the ephemeral state operation fails for any key
containing `/`, `?`, `#`, or similar reserved URL characters.
```
</details>
<details>
<summary><b>Prompt for AI Agent π€ </b></summary>
```mdx
This is a comment left during a code review.
**Path:** superset-frontend/src/core/storage/ephemeralState.ts
**Line:** 40:42
**Comment:**
*Logic Error: The URL is built with raw `extensionId` and `key`, so reserved URL characters in keys (for example `/`, `?`, `#`, `%`) will break routing or change the request path/query, causing get/set/remove to hit the wrong endpoint or fail with 404. Encode path segments before interpolation so all valid string keys work reliably.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F39171&comment_hash=63339c544149a2bfc3d7d9346e2bd43a043306c6f073492eb9ef128b66399c2a&reaction=like'>π</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F39171&comment_hash=63339c544149a2bfc3d7d9346e2bd43a043306c6f073492eb9ef128b66399c2a&reaction=dislike'>π</a>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(extensions): add a Storage API for extensions [superset]
Posted by "matebenyovszky (via GitHub)" <gi...@apache.org>.
matebenyovszky commented on PR #39171:
URL: https://github.com/apache/superset/pull/39171#issuecomment-4439951420
**Update**: Added backend Python tests to the PR (https://github.com/michael-s-molina/superset/pull/775).
Second commit adds 25 unit tests for `api.py` (the REST API that was at 0% coverage / 131 missing lines). Tests cover all helper functions and endpoint logic: `_build_cache_key`, `_parse_ttl`, `_get_extension_or_404`, `_build_storage_key`, ephemeral and persistent CRUD flows, shared vs user scope, TTL handling, and edge cases.
**Total: ~870 lines of tests** across 4 new test files (3 frontend + 1 backend).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(extensions): add a Storage API for extensions [superset]
Posted by "codeant-ai-for-open-source[bot] (via GitHub)" <gi...@apache.org>.
codeant-ai-for-open-source[bot] commented on code in PR #39171:
URL: https://github.com/apache/superset/pull/39171#discussion_r3622866531
##########
superset/extensions/storage/api.py:
##########
@@ -0,0 +1,738 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""
+REST API for extension storage.
+
+Provides HTTP endpoints for frontend extensions to access server-side
+ephemeral storage without direct backend code.
+
+All operations are user-scoped by default. Use `?shared=true` query param
+to access shared state visible to all users.
+"""
+
+from __future__ import annotations
+
+import base64
+from typing import Any
+
+from flask import g, request
+from flask.wrappers import Response
+from flask_appbuilder.api import BaseApi, expose, protect, safe
+
+from superset.extensions.storage.codecs import DEFAULT_CODEC, get_codec, SAFE_CODECS
+from superset.extensions.storage.ephemeral_dao import (
+ ExtensionEphemeralDAO,
+ ExtensionEphemeralTTLInvalid,
+ ExtensionEphemeralValueTooLarge,
+)
+from superset.extensions.storage.persistent_dao import (
+ ExtensionStorageDAO,
+ ExtensionStorageKeyTooLong,
+ ExtensionStorageListPayloadTooLarge,
+ ExtensionStorageQuotaExceeded,
+ ExtensionStorageValueTooLarge,
+)
+from superset.extensions.storage.utils import get_extension_or_404, parse_ttl
+from superset.key_value.exceptions import KeyValueCodecEncodeException
+from superset.utils.decorators import transaction
+
+
+def _decoded_result_for_wire(decoded: Any) -> tuple[Any, bool]:
+ """Convert a codec-decoded value into its JSON wire representation.
+
+ JSON has no byte type, so a raw `bytes` value is base64-encoded to a
+ string for the response and flagged as such; every other value is
+ already JSON-representable as-is. Checked on the decoded value's
+ actual type, not the codec's name, so this keeps working for a codec
+ this module didn't define.
+
+ :returns: (wire_value, is_binary)
+ """
+ if isinstance(decoded, bytes):
+ return base64.b64encode(decoded).decode("ascii"), True
+ return decoded, False
+
+
+def _wire_value_for_request(value: Any, is_binary: bool) -> Any:
+ """Convert a request body's JSON `value` into a codec's input type.
+
+ `is_binary` is an explicit flag from the caller: JSON has no byte
+ type, so there is no way to tell, from the JSON `value` alone,
+ whether it is a base64 string that must be decoded to bytes before
+ being handed to the codec's `encode`, or a literal value to pass
+ through as-is. Only the caller knows which it sent.
+
+ :raises ValueError: if `is_binary` is set and `value` is not a valid
+ base64 string.
+ """
+ if is_binary:
+ return base64.b64decode(value, validate=True)
+ return value
+
+
+class ExtensionStorageRestApi(BaseApi):
+ """REST API for extension ephemeral (Tier 2) and persistent (Tier 3) storage."""
+
+ allow_browser_login = True
+ route_base = "/api/v1/extensions"
+
+ def response(self, status_code: int, **kwargs: Any) -> Response:
+ """Helper method to create JSON responses."""
+ from flask import jsonify
+
+ return jsonify(kwargs), status_code
+
+ def response_404(self, message: str = "Not found") -> Response:
+ """Helper method to create 404 responses."""
+ from flask import jsonify
+
+ return jsonify({"message": message}), 404
+
+ def response_400(self, message: str) -> Response:
+ """Helper method to create 400 responses."""
+ from flask import jsonify
+
+ return jsonify({"message": message}), 400
+
+ @protect()
+ @safe
+ @expose("/<publisher>/<name>/storage/ephemeral/<key>", methods=("GET",))
+ def get_ephemeral(
+ self, publisher: str, name: str, key: str, **kwargs: Any
+ ) -> Response:
+ """Get a value from ephemeral state.
+ ---
+ get:
+ summary: Get a value from ephemeral state
+ parameters:
+ - in: path
+ name: publisher
+ schema:
+ type: string
+ required: true
+ description: Extension publisher
+ - in: path
+ name: name
+ schema:
+ type: string
+ required: true
+ description: Extension name
+ - in: path
+ name: key
+ schema:
+ type: string
+ required: true
+ description: Storage key
+ - in: query
+ name: shared
+ schema:
+ type: boolean
+ required: false
+ description: If true, read from shared state visible to all users
+ responses:
+ 200:
+ description: Value retrieved successfully
+ content:
+ application/json:
+ schema:
+ type: object
+ properties:
+ result:
+ description: The stored value. When 'isBinary' is
+ true, this is a base64 string that must be
+ decoded to get the actual value.
+ codec:
+ type: string
+ description: Name of the codec 'result' was encoded
+ with, e.g. "json" (default) or "binary"
+ isBinary:
+ type: boolean
+ description: Whether the stored value is binary
+ data
+ 400:
+ description: Value was stored with a codec unavailable over the API
+ 404:
+ description: Extension not found
+ """
+ extension_id = f"{publisher}.{name}"
+ extension = get_extension_or_404(extension_id)
+ if not extension:
+ return self.response_404("Extension not found")
+
+ shared = request.args.get("shared", "false").lower() == "true"
+ raw = ExtensionEphemeralDAO.get_raw(extension_id, key, shared=shared)
+ if raw is None:
+ return self.response(200, result=None)
+ value, codec = raw
+ if codec not in SAFE_CODECS:
+ return self.response_400(
+ f"Value was stored with codec '{codec}', which cannot be "
+ "read over the REST API."
+ )
+
+ result, is_binary = _decoded_result_for_wire(get_codec(codec).decode(value))
+ return self.response(200, result=result, codec=codec, isBinary=is_binary)
+
+ @protect()
+ @safe
+ @expose("/<publisher>/<name>/storage/ephemeral/<key>", methods=("PUT",))
+ def set_ephemeral(
+ self, publisher: str, name: str, key: str, **kwargs: Any
+ ) -> Response:
+ """Set a value in ephemeral state.
+ ---
+ put:
+ summary: Set a value in ephemeral state
+ parameters:
+ - in: path
+ name: publisher
+ schema:
+ type: string
+ required: true
+ description: Extension publisher
+ - in: path
+ name: name
+ schema:
+ type: string
+ required: true
+ description: Extension name
+ - in: path
+ name: key
+ schema:
+ type: string
+ required: true
+ description: Storage key
+ - in: query
+ name: shared
+ schema:
+ type: boolean
+ required: false
+ description: If true, store as shared state visible to all users
+ requestBody:
+ required: true
+ content:
+ application/json:
+ schema:
+ type: object
+ required:
+ - value
+ - ttl
+ properties:
+ value:
+ description: The value to store (must not exceed
+ MAX_VALUE_SIZE bytes once encoded with 'codec').
+ When 'isBinary' is true, this must be a base64
+ string, decoded before being handed to 'codec'.
+ codec:
+ type: string
+ description: Name of the codec used to encode 'value',
+ e.g. "json" (default). Must be one of the codecs
+ allowed over the REST API.
+ isBinary:
+ type: boolean
+ description: Whether 'value' is binary data
+ ttl:
+ type: integer
+ description: Time-to-live in seconds (must be a positive
+ integer not exceeding MAX_TTL)
+ responses:
+ 200:
+ description: Value stored successfully
+ 400:
+ description: Invalid request body, or codec not allowed over the API
+ 404:
+ description: Extension not found
+ """
+ extension_id = f"{publisher}.{name}"
+ extension = get_extension_or_404(extension_id)
+ if not extension:
+ return self.response_404("Extension not found")
+
+ body = request.get_json(silent=True) or {}
+ if "value" not in body:
+ return self.response_400("Request body must contain 'value' field")
+
+ codec = body.get("codec", DEFAULT_CODEC)
+ if not isinstance(codec, str) or codec not in SAFE_CODECS:
+ return self.response_400(
+ f"Codec '{codec}' is not allowed over the REST API."
+ )
+
+ is_binary = bool(body.get("isBinary", False))
+ try:
+ value = _wire_value_for_request(body["value"], is_binary)
+ except (ValueError, TypeError):
+ return self.response_400(
+ "Value must be a valid base64 string when 'isBinary' is true."
+ )
+ ttl, error = parse_ttl(body)
+ if error:
+ return self.response_400(error)
+
+ shared = request.args.get("shared", "false").lower() == "true"
+ try:
+ ExtensionEphemeralDAO.set(
+ extension_id, key, value, ttl, codec=codec, shared=shared
+ )
+ except (ExtensionEphemeralTTLInvalid, ExtensionEphemeralValueTooLarge) as ex:
+ return self.response_400(ex.message)
+
+ return self.response(200, message="Value stored successfully")
+
+ @protect()
+ @safe
+ @expose("/<publisher>/<name>/storage/ephemeral/<key>", methods=("DELETE",))
+ def delete_ephemeral(
+ self, publisher: str, name: str, key: str, **kwargs: Any
+ ) -> Response:
+ """Delete a value from ephemeral state.
+ ---
+ delete:
+ summary: Delete a value from ephemeral state
+ parameters:
+ - in: path
+ name: publisher
+ schema:
+ type: string
+ required: true
+ description: Extension publisher
+ - in: path
+ name: name
+ schema:
+ type: string
+ required: true
+ description: Extension name
+ - in: path
+ name: key
+ schema:
+ type: string
+ required: true
+ description: Storage key
+ - in: query
+ name: shared
+ schema:
+ type: boolean
+ required: false
+ description: If true, delete from shared state
+ responses:
+ 200:
+ description: Value deleted successfully
+ 404:
+ description: Extension not found
+ """
+ extension_id = f"{publisher}.{name}"
+ extension = get_extension_or_404(extension_id)
+ if not extension:
+ return self.response_404("Extension not found")
+
+ shared = request.args.get("shared", "false").lower() == "true"
+ ExtensionEphemeralDAO.delete(extension_id, key, shared=shared)
+
+ return self.response(200, message="Value deleted successfully")
+
+ @protect()
+ @safe
+ @expose("/<publisher>/<name>/storage/persistent", methods=("GET",))
+ def list_persistent(self, publisher: str, name: str, **kwargs: Any) -> Response:
+ """List entries in persistent state.
+ ---
+ get:
+ summary: List entries in persistent state
+ parameters:
+ - in: path
+ name: publisher
+ schema:
+ type: string
+ required: true
+ description: Extension publisher
+ - in: path
+ name: name
+ schema:
+ type: string
+ required: true
+ description: Extension name
+ - in: query
+ name: shared
+ schema:
+ type: boolean
+ required: false
+ description: If true, list shared state visible to all users
+ - in: query
+ name: resource_type
+ schema:
+ type: string
+ required: false
+ description: Filter by resource type
+ - in: query
+ name: resource_uuid
+ schema:
+ type: string
+ required: false
+ description: Filter by resource UUID (requires resource_type)
+ - in: query
+ name: page
+ schema:
+ type: integer
+ required: false
+ description: Zero-indexed page number. Defaults to 0.
+ - in: query
+ name: page_size
+ schema:
+ type: integer
+ required: false
+ description: Number of entries per page. Defaults to 10. There
+ is no fixed ceiling, but a page whose combined value size
+ exceeds MAX_LIST_PAYLOAD_SIZE is rejected β reduce page_size
+ and retry if that happens.
+ responses:
+ 200:
+ description: Entries retrieved successfully
+ content:
+ application/json:
+ schema:
+ type: object
+ properties:
+ result:
+ type: array
+ items:
+ type: object
+ properties:
+ key:
+ type: string
+ value:
+ description: The stored value, or null if its
+ codec cannot be read over the REST API.
+ When 'isBinary' is true, this is a base64
+ string that must be decoded to get the
+ actual value.
+ codec:
+ type: string
+ isBinary:
+ type: boolean
+ description: Whether the stored value is
+ binary data
+ count:
+ type: integer
+ description: Total number of entries matching the
+ given scope/filters, across all pages
+ 400:
+ description: The requested page's combined value size exceeds
+ MAX_LIST_PAYLOAD_SIZE
+ 404:
+ description: Extension not found
+ """
+ extension_id = f"{publisher}.{name}"
+ extension = get_extension_or_404(extension_id)
+ if not extension:
+ return self.response_404("Extension not found")
+
+ shared = request.args.get("shared", "false").lower() == "true"
+ user_fk = None if shared else g.user.id
+ resource_type = request.args.get("resource_type")
+ resource_uuid = request.args.get("resource_uuid")
+ try:
+ page = int(request.args.get("page", 0))
+ page_size = int(request.args.get("page_size", 10))
+ except (TypeError, ValueError):
+ return self.response_400("'page' and 'page_size' must be integers")
+
+ try:
+ entries, count = ExtensionStorageDAO.list_entries(
+ extension_id,
+ user_fk=user_fk,
+ resource_type=resource_type,
+ resource_uuid=resource_uuid,
+ page=page,
+ page_size=page_size,
+ )
+ except ExtensionStorageListPayloadTooLarge as ex:
+ return self.response(ex.status, message=ex.message)
+
+ result = []
+ for entry in entries:
+ value: Any = None
+ is_binary = False
+ if entry.codec in SAFE_CODECS and entry.value is not None:
+ decoded = get_codec(entry.codec).decode(entry.value)
+ value, is_binary = _decoded_result_for_wire(decoded)
+ result.append(
+ {
+ "key": entry.key,
+ "value": value,
+ "codec": entry.codec,
+ "isBinary": is_binary,
+ }
+ )
+
+ return self.response(200, result=result, count=count)
+
+ @protect()
+ @safe
+ @expose("/<publisher>/<name>/storage/persistent/<key>", methods=("GET",))
+ def get_persistent(
+ self, publisher: str, name: str, key: str, **kwargs: Any
+ ) -> Response:
+ """Get a value from persistent state.
+ ---
+ get:
+ summary: Get a value from persistent state
+ parameters:
+ - in: path
+ name: publisher
+ schema:
+ type: string
+ required: true
+ description: Extension publisher
+ - in: path
+ name: name
+ schema:
+ type: string
+ required: true
+ description: Extension name
+ - in: path
+ name: key
+ schema:
+ type: string
+ required: true
+ description: Storage key
+ - in: query
+ name: shared
+ schema:
+ type: boolean
+ required: false
+ description: If true, read from shared state visible to all users
+ responses:
+ 200:
+ description: Value retrieved successfully
+ content:
+ application/json:
+ schema:
+ type: object
+ properties:
+ result:
+ description: The stored value. When 'isBinary' is
+ true, this is a base64 string that must be
+ decoded to get the actual value.
+ codec:
+ type: string
+ description: Name of the codec 'result' was encoded
+ with, e.g. "json" (default) or "binary"
+ isBinary:
+ type: boolean
+ description: Whether the stored value is binary
+ data
+ 400:
+ description: Value was stored with a codec unavailable over the API
+ 404:
+ description: Extension not found
+ """
+ extension_id = f"{publisher}.{name}"
+ extension = get_extension_or_404(extension_id)
+ if not extension:
+ return self.response_404("Extension not found")
+
+ shared = request.args.get("shared", "false").lower() == "true"
+ user_fk = None if shared else g.user.id
+ entry = ExtensionStorageDAO.get(extension_id, key, user_fk=user_fk)
+ if entry is None:
+ return self.response(200, result=None)
+ if entry.codec not in SAFE_CODECS:
+ return self.response_400(
+ f"Value was stored with codec '{entry.codec}', which cannot be "
+ "read over the REST API."
+ )
+
+ decoded = ExtensionStorageDAO.get_decoded_value(
+ extension_id, key, user_fk=user_fk
+ )
Review Comment:
**Suggestion:** This handler reads the same row twice (`get` then `get_decoded_value`) without any consistency guard, so a concurrent update/delete can make the codec safety check apply to one version while decoding another version. Decode the already-fetched entry (or perform a single atomic DAO read/decode) to avoid race-driven inconsistencies and intermittent 500s. [race condition]
<details>
<summary><b>Severity Level:</b> Major β οΈ</summary>
```mdx
- β Persistent GET may intermittently fail under concurrent writes.
- β οΈ Persistent storage reads can return metadata/value mismatches.
- β οΈ Extensions relying on persistent storage see flaky behaviour.
```
</details>
<details>
<summary><b>Steps of Reproduction β
</b></summary>
```mdx
1. In a running Superset instance where `ExtensionStorageRestApi` is registered, issue a
PUT request to `/api/v1/extensions/{publisher}/{name}/storage/persistent/{key}` (handled
by `set_persistent` at `superset/extensions/storage/api.py:569-685`) to create a stored
value.
2. Concurrently, start a GET request to
`/api/v1/extensions/{publisher}/{name}/storage/persistent/{key}` (handled by
`get_persistent` at `superset/extensions/storage/api.py:488-565`), which first calls
`ExtensionStorageDAO.get(...)` at line 551 to fetch `entry`.
3. Between the first DAO call at line 551 and the second DAO call
`ExtensionStorageDAO.get_decoded_value(...)` at lines 560-562, issue another concurrent
request that updates or deletes the same key using `set_persistent` (lines 569-685) or
`delete_persistent` (lines 690-737).
4. Because `get_persistent` reads the row twice without using the previously fetched
`entry` for decoding, the safety check against `SAFE_CODECS` (lines 554-557) can apply to
one row version while `get_decoded_value()` decodes another, which can lead to mismatched
`codec` metadata or DAO-level errors surfacing as intermittent 500s under concurrent
writes.
```
</details>
[](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=523e120691604c909d31a867af26baf4&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=523e120691604c909d31a867af26baf4&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset)
*(Use Cmd/Ctrl + Click for best experience)*
<details>
<summary><b>Prompt for AI Agent π€ </b></summary>
```mdx
This is a comment left during a code review.
**Path:** superset/extensions/storage/api.py
**Line:** 551:562
**Comment:**
*Race Condition: This handler reads the same row twice (`get` then `get_decoded_value`) without any consistency guard, so a concurrent update/delete can make the codec safety check apply to one version while decoding another version. Decode the already-fetched entry (or perform a single atomic DAO read/decode) to avoid race-driven inconsistencies and intermittent 500s.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F39171&comment_hash=74ed9ae6d5496c95b44882d67b3bf9fe5f26b32f881e69f52d18aab5ac7730fc&reaction=like'>π</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F39171&comment_hash=74ed9ae6d5496c95b44882d67b3bf9fe5f26b32f881e69f52d18aab5ac7730fc&reaction=dislike'>π</a>
##########
superset/extensions/storage/api.py:
##########
@@ -0,0 +1,738 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""
+REST API for extension storage.
+
+Provides HTTP endpoints for frontend extensions to access server-side
+ephemeral storage without direct backend code.
+
+All operations are user-scoped by default. Use `?shared=true` query param
+to access shared state visible to all users.
+"""
+
+from __future__ import annotations
+
+import base64
+from typing import Any
+
+from flask import g, request
+from flask.wrappers import Response
+from flask_appbuilder.api import BaseApi, expose, protect, safe
+
+from superset.extensions.storage.codecs import DEFAULT_CODEC, get_codec, SAFE_CODECS
+from superset.extensions.storage.ephemeral_dao import (
+ ExtensionEphemeralDAO,
+ ExtensionEphemeralTTLInvalid,
+ ExtensionEphemeralValueTooLarge,
+)
+from superset.extensions.storage.persistent_dao import (
+ ExtensionStorageDAO,
+ ExtensionStorageKeyTooLong,
+ ExtensionStorageListPayloadTooLarge,
+ ExtensionStorageQuotaExceeded,
+ ExtensionStorageValueTooLarge,
+)
+from superset.extensions.storage.utils import get_extension_or_404, parse_ttl
+from superset.key_value.exceptions import KeyValueCodecEncodeException
+from superset.utils.decorators import transaction
+
+
+def _decoded_result_for_wire(decoded: Any) -> tuple[Any, bool]:
+ """Convert a codec-decoded value into its JSON wire representation.
+
+ JSON has no byte type, so a raw `bytes` value is base64-encoded to a
+ string for the response and flagged as such; every other value is
+ already JSON-representable as-is. Checked on the decoded value's
+ actual type, not the codec's name, so this keeps working for a codec
+ this module didn't define.
+
+ :returns: (wire_value, is_binary)
+ """
+ if isinstance(decoded, bytes):
+ return base64.b64encode(decoded).decode("ascii"), True
+ return decoded, False
+
+
+def _wire_value_for_request(value: Any, is_binary: bool) -> Any:
+ """Convert a request body's JSON `value` into a codec's input type.
+
+ `is_binary` is an explicit flag from the caller: JSON has no byte
+ type, so there is no way to tell, from the JSON `value` alone,
+ whether it is a base64 string that must be decoded to bytes before
+ being handed to the codec's `encode`, or a literal value to pass
+ through as-is. Only the caller knows which it sent.
+
+ :raises ValueError: if `is_binary` is set and `value` is not a valid
+ base64 string.
+ """
+ if is_binary:
+ return base64.b64decode(value, validate=True)
+ return value
+
+
+class ExtensionStorageRestApi(BaseApi):
+ """REST API for extension ephemeral (Tier 2) and persistent (Tier 3) storage."""
+
+ allow_browser_login = True
+ route_base = "/api/v1/extensions"
+
+ def response(self, status_code: int, **kwargs: Any) -> Response:
+ """Helper method to create JSON responses."""
+ from flask import jsonify
+
+ return jsonify(kwargs), status_code
+
+ def response_404(self, message: str = "Not found") -> Response:
+ """Helper method to create 404 responses."""
+ from flask import jsonify
+
+ return jsonify({"message": message}), 404
+
+ def response_400(self, message: str) -> Response:
+ """Helper method to create 400 responses."""
+ from flask import jsonify
+
+ return jsonify({"message": message}), 400
+
+ @protect()
+ @safe
+ @expose("/<publisher>/<name>/storage/ephemeral/<key>", methods=("GET",))
+ def get_ephemeral(
+ self, publisher: str, name: str, key: str, **kwargs: Any
+ ) -> Response:
+ """Get a value from ephemeral state.
+ ---
+ get:
+ summary: Get a value from ephemeral state
+ parameters:
+ - in: path
+ name: publisher
+ schema:
+ type: string
+ required: true
+ description: Extension publisher
+ - in: path
+ name: name
+ schema:
+ type: string
+ required: true
+ description: Extension name
+ - in: path
+ name: key
+ schema:
+ type: string
+ required: true
+ description: Storage key
+ - in: query
+ name: shared
+ schema:
+ type: boolean
+ required: false
+ description: If true, read from shared state visible to all users
+ responses:
+ 200:
+ description: Value retrieved successfully
+ content:
+ application/json:
+ schema:
+ type: object
+ properties:
+ result:
+ description: The stored value. When 'isBinary' is
+ true, this is a base64 string that must be
+ decoded to get the actual value.
+ codec:
+ type: string
+ description: Name of the codec 'result' was encoded
+ with, e.g. "json" (default) or "binary"
+ isBinary:
+ type: boolean
+ description: Whether the stored value is binary
+ data
+ 400:
+ description: Value was stored with a codec unavailable over the API
+ 404:
+ description: Extension not found
+ """
+ extension_id = f"{publisher}.{name}"
+ extension = get_extension_or_404(extension_id)
+ if not extension:
+ return self.response_404("Extension not found")
+
+ shared = request.args.get("shared", "false").lower() == "true"
+ raw = ExtensionEphemeralDAO.get_raw(extension_id, key, shared=shared)
+ if raw is None:
+ return self.response(200, result=None)
+ value, codec = raw
+ if codec not in SAFE_CODECS:
+ return self.response_400(
+ f"Value was stored with codec '{codec}', which cannot be "
+ "read over the REST API."
+ )
+
+ result, is_binary = _decoded_result_for_wire(get_codec(codec).decode(value))
+ return self.response(200, result=result, codec=codec, isBinary=is_binary)
+
+ @protect()
+ @safe
+ @expose("/<publisher>/<name>/storage/ephemeral/<key>", methods=("PUT",))
+ def set_ephemeral(
+ self, publisher: str, name: str, key: str, **kwargs: Any
+ ) -> Response:
+ """Set a value in ephemeral state.
+ ---
+ put:
+ summary: Set a value in ephemeral state
+ parameters:
+ - in: path
+ name: publisher
+ schema:
+ type: string
+ required: true
+ description: Extension publisher
+ - in: path
+ name: name
+ schema:
+ type: string
+ required: true
+ description: Extension name
+ - in: path
+ name: key
+ schema:
+ type: string
+ required: true
+ description: Storage key
+ - in: query
+ name: shared
+ schema:
+ type: boolean
+ required: false
+ description: If true, store as shared state visible to all users
+ requestBody:
+ required: true
+ content:
+ application/json:
+ schema:
+ type: object
+ required:
+ - value
+ - ttl
+ properties:
+ value:
+ description: The value to store (must not exceed
+ MAX_VALUE_SIZE bytes once encoded with 'codec').
+ When 'isBinary' is true, this must be a base64
+ string, decoded before being handed to 'codec'.
+ codec:
+ type: string
+ description: Name of the codec used to encode 'value',
+ e.g. "json" (default). Must be one of the codecs
+ allowed over the REST API.
+ isBinary:
+ type: boolean
+ description: Whether 'value' is binary data
+ ttl:
+ type: integer
+ description: Time-to-live in seconds (must be a positive
+ integer not exceeding MAX_TTL)
+ responses:
+ 200:
+ description: Value stored successfully
+ 400:
+ description: Invalid request body, or codec not allowed over the API
+ 404:
+ description: Extension not found
+ """
+ extension_id = f"{publisher}.{name}"
+ extension = get_extension_or_404(extension_id)
+ if not extension:
+ return self.response_404("Extension not found")
+
+ body = request.get_json(silent=True) or {}
+ if "value" not in body:
+ return self.response_400("Request body must contain 'value' field")
+
+ codec = body.get("codec", DEFAULT_CODEC)
+ if not isinstance(codec, str) or codec not in SAFE_CODECS:
+ return self.response_400(
+ f"Codec '{codec}' is not allowed over the REST API."
+ )
+
+ is_binary = bool(body.get("isBinary", False))
+ try:
+ value = _wire_value_for_request(body["value"], is_binary)
+ except (ValueError, TypeError):
+ return self.response_400(
+ "Value must be a valid base64 string when 'isBinary' is true."
+ )
+ ttl, error = parse_ttl(body)
+ if error:
+ return self.response_400(error)
+
+ shared = request.args.get("shared", "false").lower() == "true"
+ try:
+ ExtensionEphemeralDAO.set(
+ extension_id, key, value, ttl, codec=codec, shared=shared
+ )
+ except (ExtensionEphemeralTTLInvalid, ExtensionEphemeralValueTooLarge) as ex:
+ return self.response_400(ex.message)
Review Comment:
**Suggestion:** The ephemeral write path does not handle codec-encoding failures, so invalid `value`/`codec` combinations can bubble up as unhandled exceptions and return 500. Catch codec encode/type errors here and return a 400 like the persistent path does. [possible bug]
<details>
<summary><b>Severity Level:</b> Major β οΈ</summary>
```mdx
- β Ephemeral writes can crash on invalid codec/value input.
- β οΈ Extensions see 500s instead of clear validation errors.
- β οΈ Error handling inconsistent between ephemeral and persistent APIs.
```
</details>
<details>
<summary><b>Steps of Reproduction β
</b></summary>
```mdx
1. Call `PUT /api/v1/extensions/{publisher}/{name}/storage/ephemeral/{key}` which is
handled by `set_ephemeral` in `superset/extensions/storage/api.py:193-295`.
2. Provide a JSON body with a valid `codec` from `SAFE_CODECS` but a `value`/`codec`
combination that causes the underlying codec or cache layer in
`ExtensionEphemeralDAO.set(...)` (invoked at lines 289-291) to raise, for example by
sending a non-bytes value with a codec expecting bytes.
3. The try/except block at lines 288-293 only catches `ExtensionEphemeralTTLInvalid` and
`ExtensionEphemeralValueTooLarge`, and does not handle codec encode/type errors, allowing
unexpected exceptions from `ExtensionEphemeralDAO.set(...)` to propagate.
4. Observe that the request returns a generic 500 server error instead of a 400-style
validation response like the persistent path (which explicitly catches
`KeyValueCodecEncodeException`, `TypeError`, and `ValueError` at lines 664-668), making
bad client input crash the ephemeral write endpoint.
```
</details>
[](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=75edd711b2ae4855b2e70734a127b508&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=75edd711b2ae4855b2e70734a127b508&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset)
*(Use Cmd/Ctrl + Click for best experience)*
<details>
<summary><b>Prompt for AI Agent π€ </b></summary>
```mdx
This is a comment left during a code review.
**Path:** superset/extensions/storage/api.py
**Line:** 288:293
**Comment:**
*Possible Bug: The ephemeral write path does not handle codec-encoding failures, so invalid `value`/`codec` combinations can bubble up as unhandled exceptions and return 500. Catch codec encode/type errors here and return a 400 like the persistent path does.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F39171&comment_hash=f0c5489b499e673cd45792cfab697253a9f7315144c6e43b1f8c679e230bdaeb&reaction=like'>π</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F39171&comment_hash=f0c5489b499e673cd45792cfab697253a9f7315144c6e43b1f8c679e230bdaeb&reaction=dislike'>π</a>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(extensions): add a Storage API for extensions [superset]
Posted by "codeant-ai-for-open-source[bot] (via GitHub)" <gi...@apache.org>.
codeant-ai-for-open-source[bot] commented on code in PR #39171:
URL: https://github.com/apache/superset/pull/39171#discussion_r3622494042
##########
superset-core/src/superset_core/extensions/context.py:
##########
@@ -0,0 +1,136 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""
+Extension Context API for superset-core extensions.
+
+Provides access to the current extension's context, including metadata
+and scoped resources like storage. Extensions call `get_context()` to
+access their context during execution.
+
+The context is set by the host (Superset) during extension loading and
+is only available within extension code.
+
+Usage:
+ from superset_core.extensions.context import get_context
+
+ def setup():
+ ctx = get_context()
+
+ # Access extension metadata
+ print(f"Running {ctx.extension.displayName} v{ctx.extension.version}")
+
+ # Access extension-scoped storage
+ ctx.storage.ephemeral.set("lastRun", time.time())
+ data = ctx.storage.ephemeral.get("cachedData")
+"""
+
+from __future__ import annotations
+
+from typing import Any, Protocol, TYPE_CHECKING
+
+if TYPE_CHECKING:
+ from superset_core.extensions.types import BaseExtension
+
+
+class StorageAccessor(Protocol):
+ """Protocol for storage access with user-scoped and shared modes."""
+
+ def get(self, key: str) -> Any:
+ """Get a value from storage."""
+ ...
+
+ def set(self, key: str, value: Any, ttl: int = 3600) -> None:
+ """Set a value in storage with optional TTL."""
+ ...
+
+ def remove(self, key: str) -> None:
+ """Remove a value from storage."""
+ ...
+
+ @property
+ def shared(self) -> "StorageAccessor":
+ """Shared (cross-user) storage accessor."""
+ ...
+
+
+class ExtensionStorage(Protocol):
+ """Extension-scoped storage accessor for all available tiers."""
+
+ @property
+ def ephemeral(self) -> StorageAccessor:
+ """Server-side cache (Redis/Memcached) with TTL."""
+ ...
+
+ @property
+ def persistent(self) -> StorageAccessor:
+ """Database-backed persistent storage."""
+ ...
+
+
+class ExtensionContext(Protocol):
+ """
+ Context object providing extension-specific resources.
+
+ This context is only available during extension execution.
+ Calling `get_context()` outside of an extension will raise an error.
+ """
+
+ @property
+ def extension(self) -> "BaseExtension":
+ """Metadata about the current extension."""
+ ...
+
+ @property
+ def storage(self) -> ExtensionStorage:
+ """Extension-scoped storage across all available tiers."""
+ ...
+
+
+def get_context() -> ExtensionContext:
+ """
+ Get the current extension's context.
+
+ This function returns the context for the currently executing extension,
+ providing access to extension metadata and scoped resources like storage.
+
+ Host implementations will replace this function during initialization
+ with a concrete implementation providing actual functionality.
+
+ :returns: The current extension's context.
+ :raises RuntimeError: If called outside of an extension context.
+
+ Example:
+ from superset_core.extensions.context import get_context
+
+ ctx = get_context()
+
+ # Access extension metadata
+ print(f"Extension: {ctx.extension.id}")
+ print(f"Version: {ctx.extension.version}")
+
+ # Access extension-scoped storage
+ ctx.storage.ephemeral.set("tempData", data, ttl=3600)
+ value = ctx.storage.ephemeral.get("tempData")
+
+ # Access shared (cross-user) storage
+ ctx.storage.ephemeral.shared.set("globalCounter", count)
+ """
+ raise NotImplementedError(
+ "get_context() must be called within an extension context. "
+ "This function is replaced by the host during extension loading."
+ )
Review Comment:
**Suggestion:** The function documentation says a `RuntimeError` is raised, but the implementation raises `NotImplementedError`; this mismatch can break callers/tests that rely on the documented exception type. Update the docs or raise the documented exception consistently. [docstring mismatch]
<details>
<summary><b>Severity Level:</b> Major β οΈ</summary>
```mdx
- β οΈ Docstring misleads callers about raised exception type.
- β οΈ Tests expecting RuntimeError from get_context may fail.
- β οΈ Extension code may catch wrong exception class.
```
</details>
<details>
<summary><b>Steps of Reproduction β
</b></summary>
```mdx
1. In `superset-core/src/superset_core/extensions/context.py`, locate the `get_context`
function defined at lines 104-132; its docstring at line 115 states `:raises RuntimeError:
If called outside of an extension context.`.
2. Observe the implementation at lines 133-136, where `get_context()` ends by raising
`NotImplementedError` with a message explaining that the host replaces this function
during extension loading.
3. Start a Python interpreter or unit test that imports `get_context` from
`superset_core.extensions.context` without any Superset host initialization that would
patch in a concrete implementation.
4. Call `get_context()` directly; the default implementation executes lines 133-136 and
raises `NotImplementedError`, contradicting the documented `RuntimeError` in line 115, so
any caller or test written to catch `RuntimeError` will mis-handle the error.
```
</details>
[](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=58376ce1938a48898dcf0bfc5064b340&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=58376ce1938a48898dcf0bfc5064b340&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset)
*(Use Cmd/Ctrl + Click for best experience)*
<details>
<summary><b>Prompt for AI Agent π€ </b></summary>
```mdx
This is a comment left during a code review.
**Path:** superset-core/src/superset_core/extensions/context.py
**Line:** 115:136
**Comment:**
*Docstring Mismatch: The function documentation says a `RuntimeError` is raised, but the implementation raises `NotImplementedError`; this mismatch can break callers/tests that rely on the documented exception type. Update the docs or raise the documented exception consistently.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F39171&comment_hash=aef640b51f0e0c8cfc799bcb027ae90dde98b67f5683235c543f487abc8b7f93&reaction=like'>π</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F39171&comment_hash=aef640b51f0e0c8cfc799bcb027ae90dde98b67f5683235c543f487abc8b7f93&reaction=dislike'>π</a>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(extensions): add a Storage API for extensions [superset]
Posted by "michael-s-molina (via GitHub)" <gi...@apache.org>.
michael-s-molina commented on code in PR #39171:
URL: https://github.com/apache/superset/pull/39171#discussion_r3622751118
##########
superset-core/src/superset_core/extensions/context.py:
##########
@@ -0,0 +1,136 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""
+Extension Context API for superset-core extensions.
+
+Provides access to the current extension's context, including metadata
+and scoped resources like storage. Extensions call `get_context()` to
+access their context during execution.
+
+The context is set by the host (Superset) during extension loading and
+is only available within extension code.
+
+Usage:
+ from superset_core.extensions.context import get_context
+
+ def setup():
+ ctx = get_context()
+
+ # Access extension metadata
+ print(f"Running {ctx.extension.displayName} v{ctx.extension.version}")
+
+ # Access extension-scoped storage
+ ctx.storage.ephemeral.set("lastRun", time.time())
+ data = ctx.storage.ephemeral.get("cachedData")
+"""
+
+from __future__ import annotations
+
+from typing import Any, Protocol, TYPE_CHECKING
+
+if TYPE_CHECKING:
+ from superset_core.extensions.types import BaseExtension
+
+
+class StorageAccessor(Protocol):
+ """Protocol for storage access with user-scoped and shared modes."""
+
+ def get(self, key: str) -> Any:
+ """Get a value from storage."""
+ ...
+
+ def set(self, key: str, value: Any, ttl: int = 3600) -> None:
+ """Set a value in storage with optional TTL."""
+ ...
+
+ def remove(self, key: str) -> None:
+ """Remove a value from storage."""
+ ...
+
+ @property
+ def shared(self) -> "StorageAccessor":
+ """Shared (cross-user) storage accessor."""
+ ...
+
+
+class ExtensionStorage(Protocol):
+ """Extension-scoped storage accessor for all available tiers."""
+
+ @property
+ def ephemeral(self) -> StorageAccessor:
+ """Server-side cache (Redis/Memcached) with TTL."""
+ ...
+
+ @property
+ def persistent(self) -> StorageAccessor:
+ """Database-backed persistent storage."""
+ ...
+
+
+class ExtensionContext(Protocol):
+ """
+ Context object providing extension-specific resources.
+
+ This context is only available during extension execution.
+ Calling `get_context()` outside of an extension will raise an error.
+ """
+
+ @property
+ def extension(self) -> "BaseExtension":
+ """Metadata about the current extension."""
+ ...
+
+ @property
+ def storage(self) -> ExtensionStorage:
+ """Extension-scoped storage across all available tiers."""
+ ...
+
+
+def get_context() -> ExtensionContext:
+ """
+ Get the current extension's context.
+
+ This function returns the context for the currently executing extension,
+ providing access to extension metadata and scoped resources like storage.
+
+ Host implementations will replace this function during initialization
+ with a concrete implementation providing actual functionality.
+
+ :returns: The current extension's context.
+ :raises RuntimeError: If called outside of an extension context.
+
+ Example:
+ from superset_core.extensions.context import get_context
+
+ ctx = get_context()
+
+ # Access extension metadata
+ print(f"Extension: {ctx.extension.id}")
+ print(f"Version: {ctx.extension.version}")
+
+ # Access extension-scoped storage
+ ctx.storage.ephemeral.set("tempData", data, ttl=3600)
+ value = ctx.storage.ephemeral.get("tempData")
+
+ # Access shared (cross-user) storage
+ ctx.storage.ephemeral.shared.set("globalCounter", count)
+ """
+ raise NotImplementedError(
+ "get_context() must be called within an extension context. "
+ "This function is replaced by the host during extension loading."
+ )
Review Comment:
Skipping this one β this is a stub that always raises `NotImplementedError` as a placeholder ("Host implementations will replace this function during initialization"). The `:raises RuntimeError:` in the docstring describes the contract of the real host implementation that replaces this stub, not the stub's own failure mode. Leaving as-is.
##########
superset/extensions/storage/persistent_dao.py:
##########
@@ -0,0 +1,594 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from __future__ import annotations
+
+import hashlib
+import hmac
+import logging
+from dataclasses import dataclass
+from typing import Any
+
+import backoff
+from flask import current_app
+from flask_babel import gettext as _
+from sqlalchemy import and_, desc, func, LargeBinary
+
+from superset import db
+from superset.daos.base import BaseDAO
+from superset.distributed_lock import DistributedLock
+from superset.exceptions import (
+ AcquireDistributedLockFailedException,
+ SupersetGenericErrorException,
+)
+from superset.extensions.storage.codecs import DEFAULT_CODEC, get_codec
+from superset.extensions.storage.filters import ExtensionStorageFilter
+from superset.extensions.storage.persistent_model import ExtensionStorage
+from superset.utils.encrypt import (
+ DEFAULT_ENCRYPTION_ENGINE_NAME,
+ EncryptedType,
+ resolve_encryption_engine,
+)
+
+logger = logging.getLogger(__name__)
+
+
+class ExtensionStorageQuotaExceeded(SupersetGenericErrorException):
+ """Raised when a write would exceed an extension's persistent storage quota."""
+
+ status = 413
+
+ def __init__(self, extension_id: str, quota: int) -> None:
+ super().__init__(
+ message=_(
+ "Extension '%(extension_id)s' has exceeded its persistent "
+ "storage quota of %(quota)d bytes.",
+ extension_id=extension_id,
+ quota=quota,
+ ),
+ )
+
+
+class ExtensionStorageValueTooLarge(SupersetGenericErrorException):
+ """Raised when a `set` call's value exceeds MAX_VALUE_SIZE."""
+
+ status = 400
+
+ def __init__(self, max_size: int) -> None:
+ super().__init__(
+ message=_(
+ "Value exceeds the maximum allowed size of %(max_size)d bytes.",
+ max_size=max_size,
+ ),
+ )
+
+
+class ExtensionStorageListPayloadTooLarge(SupersetGenericErrorException):
+ """Raised when a `list` page's combined value size exceeds
+ MAX_LIST_PAYLOAD_SIZE."""
+
+ status = 400
+
+ def __init__(self, requested_size: int, max_size: int) -> None:
+ super().__init__(
+ message=_(
+ "Requested page would return %(requested_size)d bytes, "
+ "exceeding the maximum allowed list payload size of "
+ "%(max_size)d bytes. Reduce page_size and try again.",
+ requested_size=requested_size,
+ max_size=max_size,
+ ),
+ )
+
+
+class ExtensionStorageKeyTooLong(SupersetGenericErrorException):
+ """Raised when a `set` call's key exceeds MAX_KEY_LENGTH."""
+
+ status = 400
+
+ def __init__(self, max_length: int) -> None:
+ super().__init__(
+ message=_(
+ "Storage key exceeds the maximum allowed length of "
+ "%(max_length)d characters.",
+ max_length=max_length,
+ ),
+ )
+
+
+#: Matches the `key` column's `String(255)` definition in `ExtensionStorage`
+#: (`persistent_model.py`). Enforced here, at the DAO boundary, so every
+#: caller (REST API, ambient accessors, direct DAO use) gets a consistent
+#: error instead of a database integrity/truncation failure β the frontend
+#: SDK enforces the same limit client-side purely so it can fail fast
+#: without a round trip, not as the source of truth.
+MAX_KEY_LENGTH = 255
+
+#: TTL for the lock guarding `ExtensionStorageDAO.set`'s select-then-write.
+#: Short because the guarded section is a handful of local DB operations, no
+#: network calls β long enough to cover normal execution, short enough that
+#: a crashed holder doesn't block the same key for long.
+SET_LOCK_TTL_SECONDS = 5
+
+
+def _validate_key_length(key: str) -> None:
+ """Validate key length against MAX_KEY_LENGTH. Raises if over."""
+ if len(key) > MAX_KEY_LENGTH:
+ raise ExtensionStorageKeyTooLong(MAX_KEY_LENGTH)
+
+
+def _derive_key(secret: str, user_fk: int) -> str:
+ """Derive a per-user encryption key from the master secret and user ID.
+
+ Uses HMAC-SHA256 so the derived key is cryptographically bound to both.
+ Ciphertext encrypted for one user cannot be decrypted as another.
+ """
+ return hmac.new(
+ secret.encode(),
+ str(user_fk).encode(),
+ hashlib.sha256,
+ ).hexdigest()
+
+
+def _enc_type(user_fk: int | None) -> EncryptedType:
+ """Return an EncryptedType for the given scope.
+
+ User-scoped rows use a key derived from SECRET_KEY and the user ID, so
+ ciphertext written for one user cannot be decrypted as another.
+ Shared rows (user_fk=None) use SECRET_KEY directly.
+ """
+ secret = current_app.config["SECRET_KEY"]
+ key = _derive_key(secret, user_fk) if user_fk is not None else secret
+ engine_name = current_app.config.get(
+ "SQLALCHEMY_ENCRYPTED_FIELD_ENGINE", DEFAULT_ENCRYPTION_ENGINE_NAME
+ )
+ return EncryptedType(
+ LargeBinary,
+ key=key,
+ engine=resolve_encryption_engine(engine_name),
+ )
+
+
+def _get_quota() -> int | None:
+ """Return the configured per-extension persistent storage quota in bytes.
+
+ Returns None when no quota is configured (unlimited).
+ """
+ config = current_app.config["EXTENSIONS_PERSISTENT_STORAGE"]
+ return config.get("QUOTA_PER_EXTENSION")
+
+
+def _get_max_list_payload_size() -> int | None:
+ """Return the configured max combined value size for a `list()` page.
+
+ Returns None when no limit is configured (unlimited).
+ """
+ config = current_app.config["EXTENSIONS_PERSISTENT_STORAGE"]
+ return config.get("MAX_LIST_PAYLOAD_SIZE")
+
+
+def _validate_value_size(encoded: bytes) -> None:
+ """Validate encoded value bytes against MAX_VALUE_SIZE. Raises if over.
+
+ Enforced independently of `QUOTA_PER_EXTENSION`: the quota bounds an
+ extension's total storage, while this bounds a single value. A
+ `list_entries()` page's combined size is bounded separately by
+ MAX_LIST_PAYLOAD_SIZE (see `ExtensionStorageDAO.list_entries`).
+ """
+ config = current_app.config["EXTENSIONS_PERSISTENT_STORAGE"]
+ max_size = config.get("MAX_VALUE_SIZE")
+ if max_size is None:
+ return
+ if len(encoded) > max_size:
+ raise ExtensionStorageValueTooLarge(max_size)
+
+
+def _check_quota(extension_id: str, new_size: int, existing_size: int = 0) -> None:
+ """Raise if writing `new_size` bytes would push the extension over quota.
+
+ `existing_size` is the size of the row being replaced (0 for inserts),
+ subtracted from current usage so overwriting a key doesn't double-count
+ it against the quota it already occupies.
+
+ The usage query sums the `value_size` column via the covering
+ `ix_ext_storage_extension_id` index (on `extension_id, value_size`)
+ instead of `LENGTH(value)`, so it can run as an index-only scan rather
+ than reading every one of the extension's stored (and potentially
+ TOASTed) blobs. It is still O(rows) for the extension, so write latency
+ grows with row count β acceptable at the row counts a single extension
+ is expected to accumulate, but not a fully constant-time check.
+ """
+ quota = _get_quota()
+ if quota is None:
+ return
+ current_usage = (
+ db.session.query(func.coalesce(func.sum(ExtensionStorage.value_size), 0))
+ .filter(ExtensionStorage.extension_id == extension_id)
+ .scalar()
+ )
+ if current_usage - existing_size + new_size > quota:
+ raise ExtensionStorageQuotaExceeded(extension_id, quota)
+
+
+def _decrypt_if_needed(
+ entry: ExtensionStorage, extension_id: str, key: str
+) -> bytes | None:
+ """Return `entry.value`, decrypted if `entry.is_encrypted`.
+
+ Returns None (and logs) if decryption fails, e.g. after key rotation.
+ """
+ if not entry.is_encrypted:
+ return entry.value
+ try:
+ return _enc_type(entry.user_fk).process_result_value(
+ entry.value, db.engine.dialect
+ )
+ except Exception: # noqa: BLE001
+ logger.error(
+ "Failed to decrypt extension storage value for "
+ "extension_id=%s key=%s β possible key rotation issue",
+ extension_id,
+ key,
+ )
+ return None
+
+
+def _scope_filter(
+ extension_id: str,
+ key: str,
+ user_fk: int | None = None,
+ resource_type: str | None = None,
+ resource_uuid: str | None = None,
+) -> list[object]:
+ """Build the SQLAlchemy filter list for a scoped lookup."""
+ filters: list[object] = [
+ ExtensionStorage.extension_id == extension_id,
+ ExtensionStorage.key == key,
+ ]
+ if user_fk is None:
+ filters.append(ExtensionStorage.user_fk.is_(None))
+ else:
+ filters.append(ExtensionStorage.user_fk == user_fk)
+ if resource_type is None:
+ filters.append(ExtensionStorage.resource_type.is_(None))
+ else:
+ filters.append(ExtensionStorage.resource_type == resource_type)
+ if resource_uuid is None:
+ filters.append(ExtensionStorage.resource_uuid.is_(None))
+ else:
+ filters.append(ExtensionStorage.resource_uuid == resource_uuid)
+ return filters
+
+
+def _list_scope_filter(
+ extension_id: str,
+ user_fk: int | None = None,
+ resource_type: str | None = None,
+ resource_uuid: str | None = None,
+) -> list[object]:
+ """Build the SQLAlchemy filter list for a `list()` call.
+
+ Same scope semantics as `_scope_filter` (an explicit None means "match
+ the global/unset scope"), minus the `key` filter.
+ """
+ filters: list[object] = [ExtensionStorage.extension_id == extension_id]
+ if user_fk is None:
+ filters.append(ExtensionStorage.user_fk.is_(None))
+ else:
+ filters.append(ExtensionStorage.user_fk == user_fk)
+ if resource_type is None:
+ filters.append(ExtensionStorage.resource_type.is_(None))
+ else:
+ filters.append(ExtensionStorage.resource_type == resource_type)
+ if resource_uuid is None:
+ filters.append(ExtensionStorage.resource_uuid.is_(None))
+ else:
+ filters.append(ExtensionStorage.resource_uuid == resource_uuid)
+ return filters
+
+
+@dataclass
+class ExtensionStorageListEntry:
+ """A single row returned by `ExtensionStorageDAO.list_entries`.
+
+ `value` is the raw (decrypted) payload, mirroring `get_value()` β it is
+ not decoded with `codec` here, since REST callers must first check
+ `codec` against `SAFE_CODECS` before deciding whether decoding it is
+ safe to expose over the API, while ambient backend callers (no such
+ restriction) decode unconditionally, same as `get_decoded_value()`.
+ """
+
+ key: str
+ value: bytes | None
+ codec: str
+ is_encrypted: bool
+
+
+class ExtensionStorageDAO(BaseDAO[ExtensionStorage]):
+ """Persistent key-value store for extensions (Tier 3).
+
+ Every query and mutation is scoped to the extension currently executing
+ (resolved from ambient extension context by `ExtensionStorageFilter`),
+ so an extension can never read or modify another extension's storage
+ through this DAO.
+
+ Two access shapes are supported:
+
+ * `get`/`set`/`delete_by_key`/`list_entries` β single-key or
+ paginated-listing operations with explicit scope discriminators
+ (`user_fk`, `resource_type`, `resource_uuid`), used by the ambient
+ `persistent_state`/`ephemeral_state` accessors and the REST API (both
+ of which establish the ambient extension context themselves before
+ calling in).
+ * `find_all`/`find_one_or_none`/`filter_by`/`query`/`create`/`update`/
+ `delete` (inherited from `BaseDAO`) β general-purpose DAO access for
+ backend extension code that needs to enumerate or bulk-manage its own
+ rows, exposed to extensions via `superset_core.extensions.storage.dao`.
+
+ Scope is one of:
+
+ * Global scope β user_fk=None, resource_type=None
+ * User scope β user_fk=<id>, resource_type=None
+ * Resource scope β resource_type + resource_uuid set
+ """
+
+ base_filter = ExtensionStorageFilter
+
+ # ββ Read βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
+
+ @staticmethod
+ def get(
+ extension_id: str,
+ key: str,
+ user_fk: int | None = None,
+ resource_type: str | None = None,
+ resource_uuid: str | None = None,
+ ) -> ExtensionStorage | None:
+ """Return the raw storage entry. The value field may be encrypted."""
+ entry = (
+ db.session.query(ExtensionStorage)
+ .filter(
+ and_(
+ *_scope_filter(
+ extension_id, key, user_fk, resource_type, resource_uuid
+ )
+ )
+ )
+ .first()
+ )
+ return entry
+
+ @staticmethod
+ def get_value(
+ extension_id: str,
+ key: str,
+ user_fk: int | None = None,
+ resource_type: str | None = None,
+ resource_uuid: str | None = None,
+ ) -> bytes | None:
+ """Return the raw (decrypted) value bytes, or None if not found."""
+ entry = ExtensionStorageDAO.get(
+ extension_id, key, user_fk, resource_type, resource_uuid
+ )
+ if entry is None:
+ return None
+ return _decrypt_if_needed(entry, extension_id, key)
+
+ @staticmethod
+ def get_decoded_value(
+ extension_id: str,
+ key: str,
+ user_fk: int | None = None,
+ resource_type: str | None = None,
+ resource_uuid: str | None = None,
+ ) -> Any:
+ """Return the value decoded with the codec it was written with.
+
+ :returns: The decoded value, or None if not found.
+ """
+ entry = ExtensionStorageDAO.get(
+ extension_id, key, user_fk, resource_type, resource_uuid
+ )
+ if entry is None:
+ return None
+ raw = _decrypt_if_needed(entry, extension_id, key)
+ if raw is None:
+ return None
+ return get_codec(entry.codec).decode(raw)
+
+ @staticmethod
+ def list_entries(
+ extension_id: str,
+ user_fk: int | None = None,
+ resource_type: str | None = None,
+ resource_uuid: str | None = None,
+ page: int = 0,
+ page_size: int = 10,
+ ) -> tuple[list[ExtensionStorageListEntry], int]:
+ """List entries in the given scope, most recently changed first.
+
+ Named `list_entries` rather than `list` (unlike the single-key
+ `get`/`set`/`delete_by_key` shape) because `BaseDAO` already
+ defines a generic `list()` with a different, incompatible
+ signature (search/sort/column-selection over arbitrary models);
+ overriding it here would break that contract for callers relying
+ on polymorphic DAO access.
+
+ Callers may request any `page_size` β there is no row-count
+ ceiling β but the combined `value_size` of the requested page is
+ checked against MAX_LIST_PAYLOAD_SIZE *before* the (potentially
+ large) `value` column is fetched for any row, so an oversized
+ request fails fast rather than reading data it will then reject.
+
+ :returns: (entries, total_count). `entries[i].value` is the raw
+ (decrypted) payload β see `ExtensionStorageListEntry`.
+ :raises ExtensionStorageListPayloadTooLarge: if the requested page's
+ combined value size exceeds MAX_LIST_PAYLOAD_SIZE.
+ """
+ filters = _list_scope_filter(
+ extension_id, user_fk, resource_type, resource_uuid
+ )
+ base_query = db.session.query(ExtensionStorage).filter(and_(*filters))
+ total_count = base_query.count()
+
+ page = max(page, 0)
+ page_size = max(page_size, 1)
+ page_query = (
+ base_query.order_by(desc(ExtensionStorage.changed_on))
+ .offset(page * page_size)
+ .limit(page_size)
+ )
+
+ if (max_payload_size := _get_max_list_payload_size()) is not None:
+ # Sum value_size for exactly this page's rows, without touching
+ # the `value` column, by reusing the same ordered/offset/limited
+ # query as a subquery.
+ subquery = page_query.with_entities(ExtensionStorage.value_size).subquery()
+ requested_size = db.session.query(
+ func.coalesce(func.sum(subquery.c.value_size), 0)
+ ).scalar()
+ if requested_size > max_payload_size:
+ raise ExtensionStorageListPayloadTooLarge(
+ requested_size, max_payload_size
+ )
+
+ entries = [
+ ExtensionStorageListEntry(
+ key=row.key,
+ value=_decrypt_if_needed(row, extension_id, row.key),
+ codec=row.codec,
+ is_encrypted=row.is_encrypted,
+ )
+ for row in page_query.all()
+ ]
+ return entries, total_count
+
+ # ββ Write (upsert) ββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
+
+ @staticmethod
+ @backoff.on_exception(
+ backoff.expo,
+ AcquireDistributedLockFailedException,
+ factor=0.1,
+ base=2,
+ max_tries=8,
+ )
+ def set(
+ extension_id: str,
+ key: str,
+ value: bytes,
+ codec: str = DEFAULT_CODEC,
+ user_fk: int | None = None,
+ resource_type: str | None = None,
+ resource_uuid: str | None = None,
+ encrypt: bool = False,
+ ) -> ExtensionStorage:
+ """Upsert a storage entry. Encrypts value when encrypt=True.
+
+ The select-then-insert/update below is guarded by a distributed lock
+ scoped to this exact key: the row's unique constraint can't be relied
+ on to catch concurrent writes here, since `user_fk`, `resource_type`,
+ and `resource_uuid` are nullable and standard SQL never treats
+ NULL = NULL, so two concurrent inserts for the same *global* or
+ *resource*-scoped key (where one or more of those columns is NULL)
+ would not violate it and would land as duplicate rows.
+
+ :raises ExtensionStorageKeyTooLong: if `key` exceeds MAX_KEY_LENGTH.
+ :raises ExtensionStorageValueTooLarge: if `value` exceeds
+ MAX_VALUE_SIZE.
+ :raises ExtensionStorageQuotaExceeded: if writing `value` would push
+ the extension over its total storage quota.
+ """
+ _validate_key_length(key)
+ _validate_value_size(value)
+ stored_value = (
+ _enc_type(user_fk).process_bind_param(value, db.engine.dialect)
+ if encrypt
+ else value
+ )
+
+ with DistributedLock(
+ namespace="extension_storage_set",
+ ttl_seconds=SET_LOCK_TTL_SECONDS,
+ extension_id=extension_id,
+ key=key,
+ user_fk=user_fk,
+ resource_type=resource_type,
+ resource_uuid=resource_uuid,
+ ):
+ entry = (
+ db.session.query(ExtensionStorage)
+ .filter(
+ and_(
+ *_scope_filter(
+ extension_id, key, user_fk, resource_type, resource_uuid
+ )
+ )
+ )
+ .first()
+ )
+ new_size = len(stored_value)
+ existing_size = entry.value_size if entry is not None else 0
+ _check_quota(extension_id, new_size, existing_size)
+ if entry is not None:
Review Comment:
Fixed β widened the `DistributedLock` in `ExtensionStorageDAO.set()` from per-key to extension-wide scope, so the quota check-then-write is now atomic across all of an extension's keys, not just within one key.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(extensions): add a Storage API for extensions [superset]
Posted by "michael-s-molina (via GitHub)" <gi...@apache.org>.
michael-s-molina commented on code in PR #39171:
URL: https://github.com/apache/superset/pull/39171#discussion_r3560999616
##########
superset/extensions/context.py:
##########
@@ -16,75 +16,124 @@
# under the License.
"""
-Extension Context Management - provides ambient context during extension loading.
+Extension Context Management - provides ambient context for extensions.
-This module provides a thread-local context system that allows decorators to
-automatically detect whether they are being called in host or extension code
-during extension loading.
+This module provides a context system using Python's contextvars that allows
+extensions to access their context (metadata and scoped resources) via get_context().
+
+The context is set during extension loading and when extension callbacks are invoked.
+Uses ContextVar for thread-safe and async-safe context management with automatic
+save/restore for nested contexts.
"""
from __future__ import annotations
-import contextlib
-from threading import local
-from typing import Any, Generator
+from contextlib import contextmanager
+from contextvars import ContextVar
+from typing import Any, Iterator
from superset_core.extensions.types import Manifest
-# Thread-local storage for extension context
-_extension_context: local = local()
-
-class ExtensionContext:
- """Manages ambient extension context during loading."""
+class ExtensionStorage:
+ """Extension storage with all available tiers."""
- def __init__(self, manifest: Manifest):
- self.manifest = manifest
+ @property
+ def ephemeral(self) -> Any:
+ from superset.extensions.storage.ephemeral import EphemeralState
- def __enter__(self) -> "ExtensionContext":
- if getattr(_extension_context, "current", None) is not None:
- current_extension = _extension_context.current.manifest.id
- raise RuntimeError(
- f"Cannot initialize extension {self.manifest.id} while extension "
- f"{current_extension} is already being initialized. "
- f"Nested extension initialization is not supported."
- )
+ return EphemeralState
- _extension_context.current = self
- return self
+ @property
+ def persistent(self) -> Any:
+ from superset.extensions.storage.persistent import PersistentState
- def __exit__(self, exc_type: Any, exc_val: Any, exc_tb: Any) -> None:
- # Clear the current context
- _extension_context.current = None
+ return PersistentState
-class ExtensionContextWrapper:
- """Wrapper for extension context with extensible properties."""
+class ConcreteExtensionContext:
+ """Concrete implementation of ExtensionContext for the host."""
def __init__(self, manifest: Manifest):
self._manifest = manifest
+ self._storage = ExtensionStorage()
+
+ @property
+ def extension(self) -> Manifest:
+ """Extension metadata (new API)."""
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(extensions): add a Storage API for extensions [superset]
Posted by "codeant-ai-for-open-source[bot] (via GitHub)" <gi...@apache.org>.
codeant-ai-for-open-source[bot] commented on code in PR #39171:
URL: https://github.com/apache/superset/pull/39171#discussion_r3561099393
##########
superset-frontend/src/core/storage/ephemeralState.test.ts:
##########
@@ -0,0 +1,144 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import { SupersetClient } from '@superset-ui/core';
+import { createEphemeralState } from './ephemeralState';
+
+jest.mock('@superset-ui/core', () => ({
+ SupersetClient: {
+ get: jest.fn(),
+ put: jest.fn(),
+ delete: jest.fn(),
+ },
+}));
+
+const mockGet = SupersetClient.get as jest.Mock;
+const mockPut = SupersetClient.put as jest.Mock;
+const mockDelete = SupersetClient.delete as jest.Mock;
+
+beforeEach(() => {
+ jest.clearAllMocks();
+ mockGet.mockResolvedValue({ json: { result: null } });
+ mockPut.mockResolvedValue({});
+ mockDelete.mockResolvedValue({});
+});
+
+test('get calls correct URL with publisher/name pattern', async () => {
+ const store = createEphemeralState('myorg.myext');
+ await store.get('job_progress');
+ expect(mockGet).toHaveBeenCalledWith({
+ endpoint: '/api/v1/extensions/myorg/myext/storage/ephemeral/job_progress',
+ });
+});
+
+test('get returns result from response', async () => {
+ mockGet.mockResolvedValue({ json: { result: { pct: 42 } } });
+ const store = createEphemeralState('myorg.myext');
+ const result = await store.get('job_progress');
+ expect(result).toEqual({ pct: 42 });
+});
+
+test('get returns null when result is absent', async () => {
+ mockGet.mockResolvedValue({ json: {} });
+ const store = createEphemeralState('myorg.myext');
+ expect(await store.get('key')).toBeNull();
+});
+
+test('set calls correct URL and includes value and ttl in body', async () => {
+ const store = createEphemeralState('myorg.myext');
+ await store.set('job_progress', { pct: 42 }, { ttl: 300 });
+ expect(mockPut).toHaveBeenCalledWith({
+ endpoint: '/api/v1/extensions/myorg/myext/storage/ephemeral/job_progress',
+ body: JSON.stringify({ value: { pct: 42 }, ttl: 300, codec: 'json' }),
+ headers: { 'Content-Type': 'application/json' },
+ });
+});
+
+test('set passes codec from options', async () => {
+ const store = createEphemeralState('myorg.myext');
+ await store.set('job_progress', 'sk-...', { ttl: 300, codec: 'pickle' });
+ expect(mockPut).toHaveBeenCalledWith({
+ endpoint: '/api/v1/extensions/myorg/myext/storage/ephemeral/job_progress',
+ body: JSON.stringify({ value: 'sk-...', ttl: 300, codec: 'pickle' }),
+ headers: { 'Content-Type': 'application/json' },
+ });
Review Comment:
**Suggestion:** This test encodes `pickle` as a valid REST codec, but the server-side API only allows safe codecs (`json`/`base64`) and rejects `pickle` with HTTP 400. Keeping this expectation in tests documents an invalid client contract and can mask real integration failures; use an allowed codec in this test. [api mismatch]
<details>
<summary><b>Severity Level:</b> Major β οΈ</summary>
```mdx
- β Ephemeral REST writes using pickle codec are rejected 400.
- β οΈ Tests advertise unsupported codec, misleading extension implementers.
```
</details>
<details>
<summary><b>Steps of Reproduction β
</b></summary>
```mdx
1. The frontend test `set passes codec from options` in
`superset-frontend/src/core/storage/ephemeralState.test.ts:73-80` calls
`store.set('job_progress', 'sk-...', { ttl: 300, codec: 'pickle' })` and asserts that the
client sends `body: JSON.stringify({ value: 'sk-...', ttl: 300, codec: 'pickle' })` to
`/api/v1/extensions/myorg/myext/storage/ephemeral/job_progress`.
2. In production, `createEphemeralState` in
`superset-frontend/src/core/storage/ephemeralState.ts:72-92` forwards this `codec:
'pickle'` unchanged: `resolveSetPayload` at `binaryCodec.ts:58-63` returns `{ value:
'sk-...', codec: 'pickle' }`, and `SupersetClient.put` issues the same JSON body the test
expects.
3. On the backend, the `set_ephemeral` REST handler in
`superset/extensions/storage/api.py:29-35, 99-107` reads `codec = body.get("codec",
DEFAULT_CODEC)` and checks `if codec not in SAFE_CODECS:` returning `response_400(f"Codec
'{codec}' is not allowed over the REST API.")` (api.py:103-107). `SAFE_CODECS` is defined
in `superset/extensions/storage/codecs.py:42-46` as `{"json", "base64"}`, explicitly
excluding `"pickle"` for safety.
4. As a result, any real client following this test's contract and sending `codec:
'pickle'` to the ephemeral storage endpoint will receive HTTP 400 rather than succeed,
while the testβusing a mocked `SupersetClient`βcontinues to pass, documenting an invalid
codec choice and risking future extension code adopting `pickle` over REST and failing at
runtime.
```
</details>
[](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=fcbc8209c2e3403a90841ea7e4865c1e&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=fcbc8209c2e3403a90841ea7e4865c1e&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset)
*(Use Cmd/Ctrl + Click for best experience)*
<details>
<summary><b>Prompt for AI Agent π€ </b></summary>
```mdx
This is a comment left during a code review.
**Path:** superset-frontend/src/core/storage/ephemeralState.test.ts
**Line:** 75:80
**Comment:**
*Api Mismatch: This test encodes `pickle` as a valid REST codec, but the server-side API only allows safe codecs (`json`/`base64`) and rejects `pickle` with HTTP 400. Keeping this expectation in tests documents an invalid client contract and can mask real integration failures; use an allowed codec in this test.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F39171&comment_hash=04574434b6a9b660a731da8a94d055b1802ae388c5f6d14abd93a53f3ec6059e&reaction=like'>π</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F39171&comment_hash=04574434b6a9b660a731da8a94d055b1802ae388c5f6d14abd93a53f3ec6059e&reaction=dislike'>π</a>
##########
superset-frontend/src/core/storage/binaryCodec.ts:
##########
@@ -0,0 +1,70 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/** Convert bytes to a base64 string, for transport over the JSON REST API. */
+export function bytesToBase64(bytes: Uint8Array): string {
+ let binary = '';
+ bytes.forEach(byte => {
+ binary += String.fromCharCode(byte);
+ });
+ return btoa(binary);
+}
+
+function isBinaryValue(value: unknown): value is Uint8Array | ArrayBuffer {
+ return value instanceof Uint8Array || value instanceof ArrayBuffer;
+}
+
+/**
+ * Infer which codec to use for `value` when the caller didn't specify one.
+ *
+ * "json" names the serialization format applied to encode the value for
+ * storage/transport -- it is not a claim about the value's shape. Per
+ * RFC 8259, a JSON text may be any JSON value (object, array, string,
+ * number, boolean, or null), so scalars like `1` or `"foo"` are encoded
+ * with the "json" codec too, alongside objects and arrays.
+ *
+ * Binary values (Uint8Array/ArrayBuffer) are the one case JSON cannot
+ * represent, so they're base64-encoded instead.
+ */
+function inferDefaultCodec(value: unknown): 'json' | 'base64' {
+ return isBinaryValue(value) ? 'base64' : 'json';
+}
+
+/**
+ * Resolve the codec and request-body value for a `set()` call.
+ *
+ * If the caller explicitly set `options.codec`, it is always respected
+ * as-is and `value` is passed through untouched -- the caller is
+ * responsible for encoding it correctly (e.g. base64-encoding it
+ * themselves before calling `set()`). Auto base64-encoding only happens
+ * when no codec was specified and `value` is binary.
+ */
+export function resolveSetPayload<T>(
+ value: T,
+ explicitCodec: string | undefined,
+): { value: unknown; codec: string } {
+ if (explicitCodec !== undefined) {
+ return { value, codec: explicitCodec };
Review Comment:
**Suggestion:** The explicit-codec fast path returns binary values unchanged, so callers using `codec: 'base64'` with a `Uint8Array`/`ArrayBuffer` will send JSON-serialized typed-array objects instead of a base64 string. The backend base64 codec expects a string and this will fail decoding at runtime. Keep honoring explicit codec, but still perform base64 encoding when the explicit codec is `base64` and the value is binary. [api mismatch]
<details>
<summary><b>Severity Level:</b> Major β οΈ</summary>
```mdx
- β Ephemeral REST storage fails for base64-encoded binary values.
- β οΈ Persistent REST storage similarly mis-encodes binary payloads.
```
</details>
<details>
<summary><b>Steps of Reproduction β
</b></summary>
```mdx
1. An extension frontend obtains its context via `getContext()` wired through
`superset-frontend/src/core/extensions/index.ts:37-40`, then calls
`ctx.storage.ephemeral.set('icon', bytes, { ttl: 300, codec: 'base64' })`, where `bytes`
is a `Uint8Array` (storage wiring in
`superset-frontend/src/extensions/ExtensionContext.ts:44-52` uses
`createEphemeralState(id)`).
2. `createEphemeralState` in `superset-frontend/src/core/storage/ephemeralState.ts:72-92`
calls `resolveSetPayload(value, options.codec)`; with `options.codec === 'base64'`, the
explicit-codec branch in `binaryCodec.ts:58-63` returns `{ value: bytes, codec: 'base64'
}` without encoding, so `SupersetClient.put` sends a JSON body where `value` is a typed
array that `JSON.stringify` serializes as an array of integers.
3. The backend endpoint `/api/v1/extensions/<publisher>/<name>/storage/ephemeral/<key>`
implemented in `superset/extensions/storage/api.py:29-35, 99-109` reads `codec =
body.get("codec", DEFAULT_CODEC)` and `value = body["value"]`, then calls
`ExtensionEphemeralDAO.set(extension_id, key, value, ttl, codec=codec, shared=shared)`
(api.py:116-118) with `codec == "base64"` and `value` as a JSON array.
4. `ExtensionEphemeralDAO.set` in `superset/extensions/storage/ephemeral_dao.py:129-148`
computes `encoded = get_codec(codec).encode(value)`; for `codec == "base64"`, `get_codec`
returns `Base64KeyValueCodec` whose `encode(self, value: str) -> bytes` in
`superset/key_value/types.py:92-103` expects a base64 string. Passing a list of integers
causes `base64.b64decode` to raise `TypeError`, wrapped as `KeyValueCodecEncodeException`,
which bubbles up and fails the REST request, so extensions using `codec: "base64"` with
binary values cannot successfully store data.
```
</details>
[](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=4f5ad00bc0cb499ba656cc3ae993e82a&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=4f5ad00bc0cb499ba656cc3ae993e82a&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset)
*(Use Cmd/Ctrl + Click for best experience)*
<details>
<summary><b>Prompt for AI Agent π€ </b></summary>
```mdx
This is a comment left during a code review.
**Path:** superset-frontend/src/core/storage/binaryCodec.ts
**Line:** 62:63
**Comment:**
*Api Mismatch: The explicit-codec fast path returns binary values unchanged, so callers using `codec: 'base64'` with a `Uint8Array`/`ArrayBuffer` will send JSON-serialized typed-array objects instead of a base64 string. The backend base64 codec expects a string and this will fail decoding at runtime. Keep honoring explicit codec, but still perform base64 encoding when the explicit codec is `base64` and the value is binary.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F39171&comment_hash=95e2a2cc8374634398894316fd7db12e080b0871754f86ba14ca61cc9d6d3bdd&reaction=like'>π</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F39171&comment_hash=95e2a2cc8374634398894316fd7db12e080b0871754f86ba14ca61cc9d6d3bdd&reaction=dislike'>π</a>
##########
superset-frontend/src/core/storage/persistentState.test.ts:
##########
@@ -0,0 +1,249 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import { SupersetClient } from '@superset-ui/core';
+import { createPersistentState } from './persistentState';
+
+jest.mock('@superset-ui/core', () => ({
+ SupersetClient: {
+ get: jest.fn(),
+ put: jest.fn(),
+ delete: jest.fn(),
+ },
+}));
+
+const mockGet = SupersetClient.get as jest.Mock;
+const mockPut = SupersetClient.put as jest.Mock;
+const mockDelete = SupersetClient.delete as jest.Mock;
+
+beforeEach(() => {
+ jest.clearAllMocks();
+ mockGet.mockResolvedValue({ json: { result: null } });
+ mockPut.mockResolvedValue({});
+ mockDelete.mockResolvedValue({});
+});
+
+test('get calls correct URL with publisher/name pattern', async () => {
+ const store = createPersistentState('myorg.myext');
+ await store.get('prefs');
+ expect(mockGet).toHaveBeenCalledWith({
+ endpoint: '/api/v1/extensions/myorg/myext/storage/persistent/prefs',
+ });
+});
+
+test('get returns result from response', async () => {
+ mockGet.mockResolvedValue({ json: { result: { theme: 'dark' } } });
+ const store = createPersistentState('myorg.myext');
+ expect(await store.get('prefs')).toEqual({ theme: 'dark' });
+});
+
+test('get returns null when result is absent', async () => {
+ mockGet.mockResolvedValue({ json: {} });
+ const store = createPersistentState('myorg.myext');
+ expect(await store.get('key')).toBeNull();
+});
+
+test('set calls correct URL with value in body', async () => {
+ const store = createPersistentState('myorg.myext');
+ await store.set('prefs', { theme: 'dark' });
+ expect(mockPut).toHaveBeenCalledWith({
+ endpoint: '/api/v1/extensions/myorg/myext/storage/persistent/prefs',
+ body: JSON.stringify({
+ value: { theme: 'dark' },
+ encrypt: false,
+ codec: 'json',
+ }),
+ headers: { 'Content-Type': 'application/json' },
+ });
+});
+
+test('set passes codec from options', async () => {
+ const store = createPersistentState('myorg.myext');
+ await store.set('prefs', 'sk-...', { codec: 'pickle' });
+ expect(mockPut).toHaveBeenCalledWith({
+ endpoint: '/api/v1/extensions/myorg/myext/storage/persistent/prefs',
+ body: JSON.stringify({
+ value: 'sk-...',
+ encrypt: false,
+ codec: 'pickle',
+ }),
+ headers: { 'Content-Type': 'application/json' },
+ });
Review Comment:
**Suggestion:** This test similarly treats `pickle` as an acceptable frontend REST codec even though the backend rejects it for API safety, so it validates behavior that cannot succeed in real requests. Update the test to use an allowed codec and payload shape that matches the server contract. [api mismatch]
<details>
<summary><b>Severity Level:</b> Major β οΈ</summary>
```mdx
- β Persistent REST writes using pickle codec fail with 400.
- β οΈ Tests validate impossible behavior against real storage API.
```
</details>
<details>
<summary><b>Steps of Reproduction β
</b></summary>
```mdx
1. The test `set passes codec from options` in
`superset-frontend/src/core/storage/persistentState.test.ts:76-87` calls
`store.set('prefs', 'sk-...', { codec: 'pickle' })` and asserts that the client sends
`body: JSON.stringify({ value: 'sk-...', encrypt: false, codec: 'pickle' })` to
`/api/v1/extensions/myorg/myext/storage/persistent/prefs`.
2. In production, `createPersistentState` in
`superset-frontend/src/core/storage/persistentState.ts:116-136` uses
`resolveSetPayload(value, options?.codec)`; with `codec: 'pickle'`, the explicit branch in
`binaryCodec.ts:58-63` returns `{ value: 'sk-...', codec: 'pickle' }`, and
`SupersetClient.put` issues exactly the JSON body asserted in the test.
3. The backend persistent storage REST handler (co-located with ephemeral in
`superset/extensions/storage/api.py:36-42, 220-226`) similarly computes `codec =
body.get("codec", DEFAULT_CODEC)` and enforces `if codec not in SAFE_CODECS:` then returns
`response_400(f"Codec '{codec}' is not allowed over the REST API.")`, with `SAFE_CODECS`
from `superset/extensions/storage/codecs.py:42-46` excluding `"pickle"`.
4. Therefore any extension code that follows this test's pattern and uses `codec:
'pickle'` with the persistent REST API will be rejected with HTTP 400, while the
testβmocking `SupersetClient`βcontinues to pass and encode a client contract (`pickle`
over REST) that is explicitly forbidden by the backend, risking integration failures and
confusion about which codecs are supported.
```
</details>
[](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=b77b7f7fa205406f933c3c97a33e6a53&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=b77b7f7fa205406f933c3c97a33e6a53&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset)
*(Use Cmd/Ctrl + Click for best experience)*
<details>
<summary><b>Prompt for AI Agent π€ </b></summary>
```mdx
This is a comment left during a code review.
**Path:** superset-frontend/src/core/storage/persistentState.test.ts
**Line:** 78:87
**Comment:**
*Api Mismatch: This test similarly treats `pickle` as an acceptable frontend REST codec even though the backend rejects it for API safety, so it validates behavior that cannot succeed in real requests. Update the test to use an allowed codec and payload shape that matches the server contract.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F39171&comment_hash=f1215aa71a32715f908d657579ed8bd9e5cf6023565fce6b45cdbfff2fa528e3&reaction=like'>π</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F39171&comment_hash=f1215aa71a32715f908d657579ed8bd9e5cf6023565fce6b45cdbfff2fa528e3&reaction=dislike'>π</a>
##########
superset-frontend/src/core/storage/persistentState.ts:
##########
@@ -0,0 +1,147 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import type {
+ JsonValue,
+ PersistentListOptions,
+ PersistentListResult,
+ PersistentSetOptions,
+ PersistentStorageAccessor,
+ PersistentStorageTier,
+} from '@apache-superset/core/storage';
+import { SupersetClient } from '@superset-ui/core';
+import { resolveSetPayload } from './binaryCodec';
+
+/**
+ * Create persistent state (database-backed) bound to an extension ID.
+ */
+export function createPersistentState(
+ extensionId: string,
+): PersistentStorageTier {
+ const MAX_KEY_LENGTH = 255;
+ const [publisher, name] = extensionId.split('.');
+
+ const buildUrl = (key: string, shared?: boolean): string => {
+ if (key.length > MAX_KEY_LENGTH) {
+ throw new Error(
+ `Persistent storage key must be ${MAX_KEY_LENGTH} characters or less.`,
+ );
+ }
+ const encodedPublisher = encodeURIComponent(publisher);
+ const encodedName = encodeURIComponent(name);
+ const encodedKey = encodeURIComponent(key);
+ const url = `/api/v1/extensions/${encodedPublisher}/${encodedName}/storage/persistent/${encodedKey}`;
+ return shared ? `${url}?shared=true` : url;
+ };
+
+ const buildListUrl = (
+ options: PersistentListOptions,
+ isShared?: boolean,
+ ): string => {
+ const encodedPublisher = encodeURIComponent(publisher);
+ const encodedName = encodeURIComponent(name);
+ const url = `/api/v1/extensions/${encodedPublisher}/${encodedName}/storage/persistent`;
+ const params = new URLSearchParams();
+ if (isShared) params.set('shared', 'true');
+ if (options.resourceType) params.set('resource_type', options.resourceType);
+ if (options.resourceUuid) params.set('resource_uuid', options.resourceUuid);
+ params.set('page', String(options.page));
+ params.set('page_size', String(options.pageSize));
+ const query = params.toString();
+ return query ? `${url}?${query}` : url;
+ };
+
+ const list = async <T = JsonValue>(
+ options: PersistentListOptions,
+ isShared: boolean,
+ ): Promise<PersistentListResult<T>> => {
+ const response = await SupersetClient.get({
+ endpoint: buildListUrl(options, isShared),
+ });
+ return {
+ entries: response.json?.result ?? [],
+ count: response.json?.count ?? 0,
+ };
+ };
+
+ const shared: PersistentStorageAccessor = {
+ async get<T = JsonValue>(key: string): Promise<T | null> {
+ const response = await SupersetClient.get({
+ endpoint: buildUrl(key, true),
+ });
+ return (response.json?.result ?? null) as T | null;
+ },
+ async set<T = JsonValue>(
+ key: string,
+ value: T,
+ options?: PersistentSetOptions,
+ ): Promise<void> {
+ const payload = resolveSetPayload(value, options?.codec);
+ await SupersetClient.put({
+ endpoint: buildUrl(key, true),
+ body: JSON.stringify({
+ value: payload.value,
+ encrypt: options?.encrypt ?? false,
+ codec: payload.codec,
+ }),
+ headers: { 'Content-Type': 'application/json' },
+ });
+ },
+ async list<T = JsonValue>(
+ options: PersistentListOptions,
+ ): Promise<PersistentListResult<T>> {
+ return list<T>(options, true);
+ },
+ async remove(key: string): Promise<void> {
+ await SupersetClient.delete({ endpoint: buildUrl(key, true) });
+ },
+ };
+
+ return {
+ async get<T = JsonValue>(key: string): Promise<T | null> {
+ const response = await SupersetClient.get({ endpoint: buildUrl(key) });
+ return (response.json?.result ?? null) as T | null;
+ },
+ async set<T = JsonValue>(
+ key: string,
+ value: T,
+ options?: PersistentSetOptions,
+ ): Promise<void> {
+ const payload = resolveSetPayload(value, options?.codec);
+ await SupersetClient.put({
+ endpoint: buildUrl(key),
+ body: JSON.stringify({
+ value: payload.value,
+ encrypt: options?.encrypt ?? false,
+ codec: payload.codec,
+ }),
+ headers: { 'Content-Type': 'application/json' },
+ });
Review Comment:
**Suggestion:** The REST backend only accepts safe codecs (`json`/`base64`) for extension API writes, but this code forwards any caller-provided codec directly. Calls like `codec: 'pickle'` will deterministically fail with 400 responses at runtime, so validate/whitelist allowed codecs in the frontend API contract before sending requests. [api mismatch]
<details>
<summary><b>Severity Level:</b> Major β οΈ</summary>
```mdx
- β Persistent extension writes with unsafe codecs always 400.
- β οΈ Extensions see failed saves for explicit pickle codec.
- β οΈ Tier 3 storage contract differs between frontend/backend.
- β οΈ Extension authors must debug opaque REST 400 errors.
```
</details>
<details>
<summary><b>Steps of Reproduction β
</b></summary>
```mdx
1. An extension is loaded by the host, which constructs a per-extension context via
`createExtensionContext()` at
`superset-frontend/src/extensions/ExtensionContext.ts:61-64`; the loader wires this into
`extensions.getContext` in `superset-frontend/src/extensions/ExtensionsLoader.ts:14-20`,
so extension code can call `getContext()` to obtain `ctx`.
2. The extension code calls the Tier 3 API `ctx.storage.persistent.set('prefs', 'sk-...',
{ codec: 'pickle' })`; this is valid per the TypeScript interface `PersistentSetOptions`
in `superset-frontend/packages/superset-core/src/storage/index.ts:75-79`, which documents
`codec?: string` without constraining values.
3. That call reaches the persistent implementation `createPersistentState()` in
`superset-frontend/src/core/storage/persistentState.ts:121-135`, which computes `const
payload = resolveSetPayload(value, options?.codec);` and then issues `SupersetClient.put`
with a JSON body containing `codec: payload.codec`. The unit test
`superset-frontend/src/core/storage/persistentState.test.ts:17-27` verifies that when
`codec: 'pickle'` is passed, the client sends `codec: 'pickle'` unchanged to
`/api/v1/extensions/myorg/myext/storage/persistent/prefs`.
4. On the backend, the REST API handler `ExtensionStorageRestApi.set_persistent()`
enforces the codec whitelist `SAFE_CODECS = {'json', 'base64'}` defined in
`superset/extensions/storage/codecs.py:42-47`; the test
`superset/tests/unit_tests/extensions/storage/test_api.py:5-18` shows that a PUT to
`/api/v1/extensions/acme/dashboard/storage/persistent/prefs` with `json={"value":
"sk-...", "codec": "pickle"}` returns HTTP 400, with the DAO `mock_dao.set` not called. As
a result, any extension that sets `codec: 'pickle'` (or other unsupported identifiers) via
the persistent storage API deterministically gets a 400 and its data is never persisted.
```
</details>
[](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=07ced2c3e5d845c8867aeb8f97f6b32c&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=07ced2c3e5d845c8867aeb8f97f6b32c&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset)
*(Use Cmd/Ctrl + Click for best experience)*
<details>
<summary><b>Prompt for AI Agent π€ </b></summary>
```mdx
This is a comment left during a code review.
**Path:** superset-frontend/src/core/storage/persistentState.ts
**Line:** 126:135
**Comment:**
*Api Mismatch: The REST backend only accepts safe codecs (`json`/`base64`) for extension API writes, but this code forwards any caller-provided codec directly. Calls like `codec: 'pickle'` will deterministically fail with 400 responses at runtime, so validate/whitelist allowed codecs in the frontend API contract before sending requests.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F39171&comment_hash=17daafa0f8911cb082544b4d18d1ada2d47db2d0d4367d094b6aac4ffefdc986&reaction=like'>π</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F39171&comment_hash=17daafa0f8911cb082544b4d18d1ada2d47db2d0d4367d094b6aac4ffefdc986&reaction=dislike'>π</a>
##########
superset-frontend/src/extensions/ExtensionContext.ts:
##########
@@ -0,0 +1,65 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+import type {
+ common,
+ extensions as extensionsApi,
+} from '@apache-superset/core';
+import {
+ createBrowserStorage,
+ createEphemeralState,
+ createPersistentState,
+} from 'src/core/storage';
+
+type Extension = common.Extension;
+type ExtensionContextType = extensionsApi.ExtensionContext;
+
+/**
+ * Extension context with lazy-initialized services bound to the extension ID.
+ */
+class ExtensionContext implements ExtensionContextType {
+ readonly extension: Extension;
+
+ private _storage?: ExtensionContextType['storage'];
+
+ constructor(extension: Extension) {
+ this.extension = extension;
+ }
+
+ get storage(): ExtensionContextType['storage'] {
+ if (!this._storage) {
+ const { id } = this.extension;
+ this._storage = {
+ local: createBrowserStorage(localStorage, id),
+ session: createBrowserStorage(sessionStorage, id),
+ ephemeral: createEphemeralState(id),
+ persistent: createPersistentState(id),
+ };
Review Comment:
**Suggestion:** All storage tiers are initialized together, and this eagerly touches `localStorage`/`sessionStorage`; in environments where Web Storage is unavailable (or blocked), accessing `ctx.storage` throws before `ephemeral`/`persistent` can be used. Initialize browser tiers defensively or lazily per-tier so server-backed storage remains usable. [logic error]
<details>
<summary><b>Severity Level:</b> Major β οΈ</summary>
```mdx
- β ctx.storage access fails when Web Storage unavailable.
- β οΈ Ephemeral server cache inaccessible despite healthy backend.
- β οΈ Persistent Tier 3 storage unusable under blocked storage.
- β οΈ Extensions depending on ctx.storage cannot degrade gracefully.
```
</details>
<details>
<summary><b>Steps of Reproduction β
</b></summary>
```mdx
1. When an extension is loaded, the host constructs its context via
`createExtensionContext(extension)` at
`superset-frontend/src/extensions/ExtensionContext.ts:61-64`; `ExtensionsLoader` then
injects this into the module-federated scope so `extensions.getContext()` returns that
context (`superset-frontend/src/extensions/ExtensionsLoader.ts:14-20`).
2. Inside extension code, the developer calls `const ctx = getContext();` and then
accesses `ctx.storage.ephemeral` or `ctx.storage.persistent` to use server-backed storage;
this triggers the `storage` getter in `ExtensionContext` at
`superset-frontend/src/extensions/ExtensionContext.ts:44-55`.
3. On the first access, `this._storage` is undefined, so the getter eagerly initializes
all tiers at once: `this._storage = { local: createBrowserStorage(localStorage, id),
session: createBrowserStorage(sessionStorage, id), ephemeral: createEphemeralState(id),
persistent: createPersistentState(id) };` (lines 47-52). This directly reads the global
`localStorage` and `sessionStorage` objects before constructing the ephemeral/persistent
tiers.
4. In environments where Web Storage APIs are unavailable or blocked (for example, a
browser or embedded frame where `window.localStorage` and/or `window.sessionStorage`
access throws or is undefined), the call site `createBrowserStorage(localStorage, id)`
fails before `_storage` is populated. Because the getter never completes,
`ctx.storage.ephemeral` and `ctx.storage.persistent` cannot be reached, even though their
implementations (`superset-frontend/src/core/storage/ephemeralState.ts:32-98` and
`superset-frontend/src/core/storage/persistentState.ts:34-147`) and corresponding backend
tiers (`superset-core/src/superset_core/extensions/storage/ephemeral.py` and
`persistent.py`) would otherwise be usable.
```
</details>
[](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=d31d813346f447e18fbcf64a4a9bb752&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=d31d813346f447e18fbcf64a4a9bb752&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset)
*(Use Cmd/Ctrl + Click for best experience)*
<details>
<summary><b>Prompt for AI Agent π€ </b></summary>
```mdx
This is a comment left during a code review.
**Path:** superset-frontend/src/extensions/ExtensionContext.ts
**Line:** 47:52
**Comment:**
*Logic Error: All storage tiers are initialized together, and this eagerly touches `localStorage`/`sessionStorage`; in environments where Web Storage is unavailable (or blocked), accessing `ctx.storage` throws before `ephemeral`/`persistent` can be used. Initialize browser tiers defensively or lazily per-tier so server-backed storage remains usable.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F39171&comment_hash=442765637cb3527cd52a3596165b47ca5d70384e03814f414d9141bda53f374b&reaction=like'>π</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F39171&comment_hash=442765637cb3527cd52a3596165b47ca5d70384e03814f414d9141bda53f374b&reaction=dislike'>π</a>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(extensions): add a Storage API for extensions [superset]
Posted by "codeant-ai-for-open-source[bot] (via GitHub)" <gi...@apache.org>.
codeant-ai-for-open-source[bot] commented on code in PR #39171:
URL: https://github.com/apache/superset/pull/39171#discussion_r3561087002
##########
superset-frontend/packages/superset-core/src/extensions/index.ts:
##########
@@ -24,9 +24,80 @@
* including querying extension metadata and monitoring extension lifecycle events.
* Extensions can use this API to discover other extensions and react to changes
* in the extension ecosystem.
+ *
+ * Extensions can access their own context via `getContext()`, which provides:
+ * - Extension metadata (id, name, version, etc.)
+ * - Extension-scoped storage (localStorage, sessionStorage, ephemeral cache)
+ *
+ * @example
+ * ```typescript
+ * import { extensions } from '@apache-superset/core';
+ *
+ * // Get the current extension's context
+ * const ctx = extensions.getContext();
+ *
+ * // Access extension metadata
+ * console.log(`Running ${ctx.extension.name} v${ctx.extension.version}`);
+ *
+ * // Access extension-scoped storage
+ * await ctx.storage.local.set('preference', { theme: 'dark' });
+ * await ctx.storage.ephemeral.set('cache', data, { ttl: 300 });
+ * ```
*/
import { Extension } from '../common';
+import { ExtensionStorage } from '../storage';
+
+/**
+ * Context object providing extension-specific resources.
+ *
+ * This context is only available during extension execution.
+ * Calling `getContext()` outside of an extension will throw an error.
+ */
+export interface ExtensionContext {
+ /**
+ * Metadata about the current extension.
+ */
+ extension: Extension;
+
+ /**
+ * Extension-scoped storage across all tiers.
+ * All keys are automatically namespaced to prevent collisions.
+ */
+ storage: ExtensionStorage;
+}
+
+/**
+ * Get the current extension's context.
+ *
+ * This function returns the context for the currently executing extension,
+ * providing access to extension metadata and scoped resources like storage.
+ *
+ * @returns The current extension's context.
+ * @throws Error if called outside of an extension context.
+ *
+ * @example
+ * ```typescript
+ * import { extensions } from '@apache-superset/core';
+ *
+ * const ctx = extensions.getContext();
+ *
+ * // Access extension metadata
+ * console.log(`Extension: ${ctx.extension.id}`);
+ * console.log(`Version: ${ctx.extension.version}`);
+ *
+ * // Access extension-scoped storage
+ * await ctx.storage.local.set('userPref', { sidebar: 'collapsed' });
+ * const pref = await ctx.storage.local.get('userPref');
+ *
+ * // Use ephemeral storage with TTL
+ * await ctx.storage.ephemeral.set('tempData', data, { ttl: 3600 });
+ *
+ * // Access shared (cross-user) storage
+ * await ctx.storage.ephemeral.shared.set('globalCounter', count);
Review Comment:
**Suggestion:** The JSDoc example calls `ephemeral.shared.set` without the required TTL options argument, which contradicts the declared API contract and will fail for consumers who copy this example. Update the example to pass an options object with `ttl` so it matches the actual method signature. [docstring mismatch]
<details>
<summary><b>Severity Level:</b> Major β οΈ</summary>
```mdx
β Extensions using shared ephemeral storage crash on writes.
β οΈ TypeScript consumers hit compile errors copying JSDoc example.
β οΈ Documentation misleads about TTL requirements for shared cache.
```
</details>
<details>
<summary><b>Steps of Reproduction β
</b></summary>
```mdx
1. Open `superset-frontend/packages/superset-core/src/extensions/index.ts` and locate the
`getContext()` JSDoc example at lines 79β98 (verified via Read), which shows `await
ctx.storage.ephemeral.shared.set('globalCounter', count);` on line 97 with only two
arguments.
2. Open `superset-frontend/packages/superset-core/src/storage/index.ts` and inspect
`EphemeralStorageAccessor` at lines 132β138 and `EphemeralStorageTier` at lines 145β147
(verified via Read); note that `set` is declared as `set(key, value, options:
EphemeralSetOptions): Promise<void>` and that `.shared` is typed as
`EphemeralStorageAccessor`, meaning the third `options` argument with TTL is required for
shared calls.
3. Open the concrete implementation `superset-frontend/src/core/storage/ephemeralState.ts`
and inspect the `shared` accessor definition at lines 44β55 (verified via Read);
`shared.set` is implemented as `async set(key, value, options: EphemeralSetOptions)` and
immediately uses `options.codec` and `options.ttl`, so calling it without the `options`
object will result in `options` being `undefined` and a runtime `TypeError` when
`options.codec` is accessed in JavaScript.
4. As an extension author, copy the JSDoc example from `extensions/index.ts` into a
TypeScript extension module that imports `extensions` from `@apache-superset/core`, call
`ctx.storage.ephemeral.shared.set('globalCounter', count)` exactly as shown, and observe
either (a) a TypeScript compile-time error due to the missing `options` argument based on
`EphemeralStorageTier`/`EphemeralStorageAccessor` types, or (b) in a plain JavaScript
extension, a runtime crash when `createEphemeralState.shared.set` dereferences
`options.codec` and `options.ttl`, demonstrating that the documented call shape is
incorrect.
```
</details>
[](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=b2e1e063ba77430f926457fbf1364121&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=b2e1e063ba77430f926457fbf1364121&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset)
*(Use Cmd/Ctrl + Click for best experience)*
<details>
<summary><b>Prompt for AI Agent π€ </b></summary>
```mdx
This is a comment left during a code review.
**Path:** superset-frontend/packages/superset-core/src/extensions/index.ts
**Line:** 96:97
**Comment:**
*Docstring Mismatch: The JSDoc example calls `ephemeral.shared.set` without the required TTL options argument, which contradicts the declared API contract and will fail for consumers who copy this example. Update the example to pass an options object with `ttl` so it matches the actual method signature.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F39171&comment_hash=5b63ad9de8d122b241ec9aa4fb2648cf6bad5d04e2ca5d4ac5360249d45c575f&reaction=like'>π</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F39171&comment_hash=5b63ad9de8d122b241ec9aa4fb2648cf6bad5d04e2ca5d4ac5360249d45c575f&reaction=dislike'>π</a>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(extensions): add a Storage API for extensions [superset]
Posted by "matebenyovszky (via GitHub)" <gi...@apache.org>.
matebenyovszky commented on PR #39171:
URL: https://github.com/apache/superset/pull/39171#issuecomment-4833488953
Hey @michael-s-molina β great to see the progress! Almost everything is green now. π
The **only remaining CI failure is `pre-commit`** β specifically `prettier` auto-formatting on `ephemeralState.ts`. The fix is trivial β two `set` method signatures need line-breaking because they exceed the line width:
```diff
--- a/superset-frontend/src/core/storage/ephemeralState.ts
+++ b/superset-frontend/src/core/storage/ephemeralState.ts
@@ line 50 and 67
- set: async (key: string, value: JsonValue, options: EphemeralSetOptions) => {
+ set: async (
+ key: string,
+ value: JsonValue,
+ options: EphemeralSetOptions,
+ ) => {
```
**Quick fix** β from `superset-frontend/`:
```bash
npx prettier --write src/core/storage/ephemeralState.ts
git add -u && git commit --trailer "Co-authored-by: Cursor <cu...@cursor.com>" -m "Fix prettier formatting" && git push
```
Once that's committed, all CI should be green and this can be marked **Ready for Review**! π
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(extensions): add Tier 1 and Tier 2 storage APIs [superset]
Posted by "codeant-ai-for-open-source[bot] (via GitHub)" <gi...@apache.org>.
codeant-ai-for-open-source[bot] commented on code in PR #39171:
URL: https://github.com/apache/superset/pull/39171#discussion_r3047385675
##########
superset/extensions/storage/api.py:
##########
@@ -0,0 +1,372 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""
+REST API for extension storage.
+
+Provides HTTP endpoints for frontend extensions to access server-side
+ephemeral storage without direct backend code.
+
+By default, all operations are user-scoped (private to the current user).
+Use the /shared/ endpoints to access state visible to all users.
+"""
+
+from __future__ import annotations
+
+from typing import Any
+
+from flask import g, request
+from flask.wrappers import Response
+from flask_appbuilder.api import BaseApi, expose, protect, safe
+
+from superset.extensions import cache_manager
+from superset.extensions.types import LoadedExtension
+from superset.extensions.utils import get_extensions
+
+# Key separator
+SEPARATOR = ":"
+
+# Key prefix for extension ephemeral state
+KEY_PREFIX = "superset-ext"
+
+# Default TTL: 1 hour
+DEFAULT_TTL = 3600
+
+
+def _build_cache_key(*parts: Any) -> str:
+ """Build a namespaced cache key from parts."""
+ return SEPARATOR.join(str(part) for part in parts)
+
+
+def _get_extension_or_404(extension_id: str) -> LoadedExtension | None:
+ """Get extension by ID or return None if not found."""
+ extensions = get_extensions()
+ return extensions.get(extension_id)
+
+
+class ExtensionStorageRestApi(BaseApi):
+ """REST API for extension ephemeral state storage."""
+
+ allow_browser_login = True
+ route_base = "/api/v1/extensions/storage"
+
+ def response(self, status_code: int, **kwargs: Any) -> Response:
+ """Helper method to create JSON responses."""
+ from flask import jsonify
+
+ return jsonify(kwargs), status_code
+
+ def response_404(self, message: str = "Not found") -> Response:
+ """Helper method to create 404 responses."""
+ from flask import jsonify
+
+ return jsonify({"message": message}), 404
+
+ def response_400(self, message: str) -> Response:
+ """Helper method to create 400 responses."""
+ from flask import jsonify
+
+ return jsonify({"message": message}), 400
+
+ # =========================================================================
+ # User-Scoped Ephemeral State Endpoints (Default)
+ # =========================================================================
+
+ @protect()
+ @safe
+ @expose("/ephemeral/<extension_id>/<key>", methods=("GET",))
+ def get_ephemeral(self, extension_id: str, key: str, **kwargs: Any) -> Response:
+ """Get a value from user-scoped ephemeral state.
+ ---
+ get:
+ summary: Get a value from user-scoped ephemeral state (default)
+ parameters:
+ - in: path
+ name: extension_id
+ schema:
+ type: string
+ required: true
+ description: Extension ID (publisher.name)
+ - in: path
+ name: key
+ schema:
+ type: string
+ required: true
+ description: Storage key
+ responses:
+ 200:
+ description: Value retrieved successfully
+ content:
+ application/json:
+ schema:
+ type: object
+ properties:
+ result:
+ description: The stored value
+ 404:
+ description: Extension not found
+ """
+ extension = _get_extension_or_404(extension_id)
+ if not extension:
+ return self.response_404("Extension not found")
+
+ user_id = g.user.id
+ cache_key = _build_cache_key(KEY_PREFIX, extension_id, "user", user_id, key)
+ value = cache_manager.extension_ephemeral_state_cache.get(cache_key)
+
+ return self.response(200, result=value)
+
+ @protect()
+ @safe
+ @expose("/ephemeral/<extension_id>/<key>", methods=("PUT",))
+ def set_ephemeral(self, extension_id: str, key: str, **kwargs: Any) -> Response:
+ """Set a value in user-scoped ephemeral state.
+ ---
+ put:
+ summary: Set a value in user-scoped ephemeral state (default)
+ parameters:
+ - in: path
+ name: extension_id
+ schema:
+ type: string
+ required: true
+ description: Extension ID (publisher.name)
+ - in: path
+ name: key
+ schema:
+ type: string
+ required: true
+ description: Storage key
+ requestBody:
+ required: true
+ content:
+ application/json:
+ schema:
+ type: object
+ properties:
+ value:
+ description: The value to store
+ ttl:
+ type: integer
+ description: Time-to-live in seconds (default 3600)
+ responses:
+ 200:
+ description: Value stored successfully
+ 400:
+ description: Invalid request body
+ 404:
+ description: Extension not found
+ """
+ extension = _get_extension_or_404(extension_id)
+ if not extension:
+ return self.response_404("Extension not found")
+
+ body = request.get_json(silent=True) or {}
+ if "value" not in body:
+ return self.response_400("Request body must contain 'value' field")
+
+ value = body["value"]
+ ttl = body.get("ttl", DEFAULT_TTL)
Review Comment:
**Suggestion:** `ttl` is accepted without type/range validation in the user-scoped PUT endpoint, so non-integer or invalid values can cause cache backend errors and return 500. Parse and validate `ttl` before using it. [type error]
<details>
<summary><b>Severity Level:</b> Major β οΈ</summary>
```mdx
- β Invalid ttl can cause 500s on user-scoped PUT.
- β οΈ Any extension using custom TTL is affected.
- β οΈ Reduces robustness of new extension storage API.
```
</details>
```suggestion
try:
ttl = int(body.get("ttl", DEFAULT_TTL))
except (TypeError, ValueError):
return self.response_400("Field 'ttl' must be a positive integer")
if ttl <= 0:
return self.response_400("Field 'ttl' must be a positive integer")
```
<details>
<summary><b>Steps of Reproduction β
</b></summary>
```mdx
1. Configure `EXTENSIONS_STORAGE['EPHEMERAL']` so `CacheManager.init_app` sets up
`extension_ephemeral_state_cache` using a real backend (see
`superset/utils/cache_manager.py:58-76`, which recommends `RedisCache`, `MemcachedCache`
or similar), and ensure an extension is loaded so `_get_extension_or_404` succeeds.
2. From an extension frontend or via direct API call, invoke the user-scoped PUT endpoint
`PUT /api/v1/extensions/storage/ephemeral/<extension_id>/bad_ttl_key` handled by
`ExtensionStorageRestApi.set_ephemeral` in `superset/extensions/storage/api.py:135-189`
with JSON body `{"value": "test", "ttl": "not-an-int"}` (or another non-integer / negative
value).
3. Inside `set_ephemeral`, lines `178-183` parse the body, setting `value = body["value"]`
(line `182`) and `ttl = body.get("ttl", DEFAULT_TTL)` (line `183`), so `ttl` becomes the
string `"not-an-int"` (or other invalid type/value) without any validation.
4. Lines `185-187` then build the cache key and call
`cache_manager.extension_ephemeral_state_cache.set(cache_key, value, timeout=ttl)`,
passing the invalid `ttl` straight to the underlying `SupersetCache` (subclass of
`flask_caching.Cache` defined in `superset/utils/cache_manager.py:13-47`); typical cache
backends expect an integer timeout and will raise an error when given a bad type/value,
resulting in a 500 error from this endpoint instead of a clean 400-level response.
```
</details>
<details>
<summary><b>Prompt for AI Agent π€ </b></summary>
```mdx
This is a comment left during a code review.
**Path:** superset/extensions/storage/api.py
**Line:** 183:183
**Comment:**
*Type Error: `ttl` is accepted without type/range validation in the user-scoped PUT endpoint, so non-integer or invalid values can cause cache backend errors and return 500. Parse and validate `ttl` before using it.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F39171&comment_hash=0e2c40af16b31011f26d7b1fafaec2c2492b2d09b340955d11199993efb6d119&reaction=like'>π</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F39171&comment_hash=0e2c40af16b31011f26d7b1fafaec2c2492b2d09b340955d11199993efb6d119&reaction=dislike'>π</a>
##########
superset/extensions/storage/api.py:
##########
@@ -0,0 +1,372 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""
+REST API for extension storage.
+
+Provides HTTP endpoints for frontend extensions to access server-side
+ephemeral storage without direct backend code.
+
+By default, all operations are user-scoped (private to the current user).
+Use the /shared/ endpoints to access state visible to all users.
+"""
+
+from __future__ import annotations
+
+from typing import Any
+
+from flask import g, request
+from flask.wrappers import Response
+from flask_appbuilder.api import BaseApi, expose, protect, safe
+
+from superset.extensions import cache_manager
+from superset.extensions.types import LoadedExtension
+from superset.extensions.utils import get_extensions
+
+# Key separator
+SEPARATOR = ":"
+
+# Key prefix for extension ephemeral state
+KEY_PREFIX = "superset-ext"
+
+# Default TTL: 1 hour
+DEFAULT_TTL = 3600
+
+
+def _build_cache_key(*parts: Any) -> str:
+ """Build a namespaced cache key from parts."""
+ return SEPARATOR.join(str(part) for part in parts)
+
+
+def _get_extension_or_404(extension_id: str) -> LoadedExtension | None:
+ """Get extension by ID or return None if not found."""
+ extensions = get_extensions()
+ return extensions.get(extension_id)
+
+
+class ExtensionStorageRestApi(BaseApi):
+ """REST API for extension ephemeral state storage."""
+
+ allow_browser_login = True
+ route_base = "/api/v1/extensions/storage"
+
+ def response(self, status_code: int, **kwargs: Any) -> Response:
+ """Helper method to create JSON responses."""
+ from flask import jsonify
+
+ return jsonify(kwargs), status_code
+
+ def response_404(self, message: str = "Not found") -> Response:
+ """Helper method to create 404 responses."""
+ from flask import jsonify
+
+ return jsonify({"message": message}), 404
+
+ def response_400(self, message: str) -> Response:
+ """Helper method to create 400 responses."""
+ from flask import jsonify
+
+ return jsonify({"message": message}), 400
+
+ # =========================================================================
+ # User-Scoped Ephemeral State Endpoints (Default)
+ # =========================================================================
+
+ @protect()
+ @safe
+ @expose("/ephemeral/<extension_id>/<key>", methods=("GET",))
+ def get_ephemeral(self, extension_id: str, key: str, **kwargs: Any) -> Response:
+ """Get a value from user-scoped ephemeral state.
+ ---
+ get:
+ summary: Get a value from user-scoped ephemeral state (default)
+ parameters:
+ - in: path
+ name: extension_id
+ schema:
+ type: string
+ required: true
+ description: Extension ID (publisher.name)
+ - in: path
+ name: key
+ schema:
+ type: string
+ required: true
+ description: Storage key
+ responses:
+ 200:
+ description: Value retrieved successfully
+ content:
+ application/json:
+ schema:
+ type: object
+ properties:
+ result:
+ description: The stored value
+ 404:
+ description: Extension not found
+ """
+ extension = _get_extension_or_404(extension_id)
+ if not extension:
+ return self.response_404("Extension not found")
+
+ user_id = g.user.id
+ cache_key = _build_cache_key(KEY_PREFIX, extension_id, "user", user_id, key)
+ value = cache_manager.extension_ephemeral_state_cache.get(cache_key)
+
+ return self.response(200, result=value)
+
+ @protect()
+ @safe
+ @expose("/ephemeral/<extension_id>/<key>", methods=("PUT",))
+ def set_ephemeral(self, extension_id: str, key: str, **kwargs: Any) -> Response:
+ """Set a value in user-scoped ephemeral state.
+ ---
+ put:
+ summary: Set a value in user-scoped ephemeral state (default)
+ parameters:
+ - in: path
+ name: extension_id
+ schema:
+ type: string
+ required: true
+ description: Extension ID (publisher.name)
+ - in: path
+ name: key
+ schema:
+ type: string
+ required: true
+ description: Storage key
+ requestBody:
+ required: true
+ content:
+ application/json:
+ schema:
+ type: object
+ properties:
+ value:
+ description: The value to store
+ ttl:
+ type: integer
+ description: Time-to-live in seconds (default 3600)
+ responses:
+ 200:
+ description: Value stored successfully
+ 400:
+ description: Invalid request body
+ 404:
+ description: Extension not found
+ """
+ extension = _get_extension_or_404(extension_id)
+ if not extension:
+ return self.response_404("Extension not found")
+
+ body = request.get_json(silent=True) or {}
+ if "value" not in body:
+ return self.response_400("Request body must contain 'value' field")
+
+ value = body["value"]
+ ttl = body.get("ttl", DEFAULT_TTL)
+
+ user_id = g.user.id
+ cache_key = _build_cache_key(KEY_PREFIX, extension_id, "user", user_id, key)
+ cache_manager.extension_ephemeral_state_cache.set(cache_key, value, timeout=ttl)
+
+ return self.response(200, message="Value stored successfully")
+
+ @protect()
+ @safe
+ @expose("/ephemeral/<extension_id>/<key>", methods=("DELETE",))
+ def delete_ephemeral(self, extension_id: str, key: str, **kwargs: Any) -> Response:
+ """Delete a value from user-scoped ephemeral state.
+ ---
+ delete:
+ summary: Delete a value from user-scoped ephemeral state (default)
+ parameters:
+ - in: path
+ name: extension_id
+ schema:
+ type: string
+ required: true
+ description: Extension ID (publisher.name)
+ - in: path
+ name: key
+ schema:
+ type: string
+ required: true
+ description: Storage key
+ responses:
+ 200:
+ description: Value deleted successfully
+ 404:
+ description: Extension not found
+ """
+ extension = _get_extension_or_404(extension_id)
+ if not extension:
+ return self.response_404("Extension not found")
+
+ user_id = g.user.id
+ cache_key = _build_cache_key(KEY_PREFIX, extension_id, "user", user_id, key)
+ cache_manager.extension_ephemeral_state_cache.delete(cache_key)
+
+ return self.response(200, message="Value deleted successfully")
+
+ # =========================================================================
+ # Shared (Global) Ephemeral State Endpoints
+ # =========================================================================
+
+ @protect()
+ @safe
+ @expose("/ephemeral/shared/<extension_id>/<key>", methods=("GET",))
+ def get_ephemeral_shared(
+ self, extension_id: str, key: str, **kwargs: Any
+ ) -> Response:
+ """Get a value from shared ephemeral state.
+ ---
+ get:
+ summary: Get a value from shared (global) ephemeral state
+ parameters:
+ - in: path
+ name: extension_id
+ schema:
+ type: string
+ required: true
+ description: Extension ID (publisher.name)
+ - in: path
+ name: key
+ schema:
+ type: string
+ required: true
+ description: Storage key
+ responses:
+ 200:
+ description: Value retrieved successfully
+ content:
+ application/json:
+ schema:
+ type: object
+ properties:
+ result:
+ description: The stored value
+ 404:
+ description: Extension not found
+ """
+ extension = _get_extension_or_404(extension_id)
+ if not extension:
+ return self.response_404("Extension not found")
+
+ cache_key = _build_cache_key(KEY_PREFIX, extension_id, key)
+ value = cache_manager.extension_ephemeral_state_cache.get(cache_key)
+
+ return self.response(200, result=value)
+
+ @protect()
+ @safe
+ @expose("/ephemeral/shared/<extension_id>/<key>", methods=("PUT",))
+ def set_ephemeral_shared(
+ self, extension_id: str, key: str, **kwargs: Any
+ ) -> Response:
+ """Set a value in shared ephemeral state.
+ ---
+ put:
+ summary: Set a value in shared (global) ephemeral state
+ parameters:
+ - in: path
+ name: extension_id
+ schema:
+ type: string
+ required: true
+ description: Extension ID (publisher.name)
+ - in: path
+ name: key
+ schema:
+ type: string
+ required: true
+ description: Storage key
+ requestBody:
+ required: true
+ content:
+ application/json:
+ schema:
+ type: object
+ properties:
+ value:
+ description: The value to store
+ ttl:
+ type: integer
+ description: Time-to-live in seconds (default 3600)
+ responses:
+ 200:
+ description: Value stored successfully
+ 400:
+ description: Invalid request body
+ 404:
+ description: Extension not found
+ """
+ extension = _get_extension_or_404(extension_id)
+ if not extension:
+ return self.response_404("Extension not found")
+
+ body = request.get_json(silent=True) or {}
+ if "value" not in body:
+ return self.response_400("Request body must contain 'value' field")
+
+ value = body["value"]
+ ttl = body.get("ttl", DEFAULT_TTL)
+
+ cache_key = _build_cache_key(KEY_PREFIX, extension_id, key)
Review Comment:
**Suggestion:** The shared PUT endpoint can write to private user-scoped cache entries when the key starts with `user:` and matches the internal private key pattern. Reject reserved `user:`-prefixed keys to prevent cross-scope overwrites. [security]
<details>
<summary><b>Severity Level:</b> Critical π¨</summary>
```mdx
- β Shared PUT can overwrite other users' ephemeral state.
- β Breaks integrity of user-scoped extension storage.
- β οΈ Impacts all extensions relying on private ephemeral keys.
```
</details>
```suggestion
if key.startswith("user:"):
return self.response_400("Invalid key prefix for shared storage")
```
<details>
<summary><b>Steps of Reproduction β
</b></summary>
```mdx
1. Ensure an extension is registered so `extension_id` is valid (extensions are loaded in
`superset/extensions/utils.py:252-38` and referenced by ID via `_get_extension_or_404` in
`superset/extensions/storage/api.py:55-58`).
2. As user A, store a user-scoped value by calling `PUT
/api/v1/extensions/storage/ephemeral/<extension_id>/job_progress` handled by
`set_ephemeral` in `superset/extensions/storage/api.py:135-189`; at lines `185-187` the
code builds key `superset-ext:<extension_id>:user:<userA_id>:job_progress` and writes the
initial value (e.g. `{"pct": 42}`) to `extension_ephemeral_state_cache`.
3. As user B, call the shared PUT endpoint `PUT
/api/v1/extensions/storage/ephemeral/shared/<extension_id>/user:<userA_id>:job_progress`
implemented by `set_ephemeral_shared` in `superset/extensions/storage/api.py:20-75`; after
validating the extension at lines `61-63`, lines `69-73` (full file lines `328-332`)
execute `value = body["value"]`, `ttl = body.get("ttl", DEFAULT_TTL)`, then `cache_key =
_build_cache_key(KEY_PREFIX, extension_id, key)` and
`cache_manager.extension_ephemeral_state_cache.set(cache_key, value, timeout=ttl)`,
computing the same concrete cache key
`superset-ext:<extension_id>:user:<userA_id>:job_progress` previously used for user A's
private state.
4. Subsequent calls by user A to `GET
/api/v1/extensions/storage/ephemeral/<extension_id>/job_progress` via `get_ephemeral` in
`superset/extensions/storage/api.py:91-131` will read the overwritten value written by
user B, demonstrating that a shared write with key `user:<id>:<key>` can overwrite private
user-scoped state.
```
</details>
<details>
<summary><b>Prompt for AI Agent π€ </b></summary>
```mdx
This is a comment left during a code review.
**Path:** superset/extensions/storage/api.py
**Line:** 331:331
**Comment:**
*Security: The shared PUT endpoint can write to private user-scoped cache entries when the key starts with `user:` and matches the internal private key pattern. Reject reserved `user:`-prefixed keys to prevent cross-scope overwrites.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F39171&comment_hash=6082ef194693f62b61e8d1cf9e48623488068862975c1e0226acc30d64640dd7&reaction=like'>π</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F39171&comment_hash=6082ef194693f62b61e8d1cf9e48623488068862975c1e0226acc30d64640dd7&reaction=dislike'>π</a>
##########
superset/extensions/storage/api.py:
##########
@@ -0,0 +1,372 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""
+REST API for extension storage.
+
+Provides HTTP endpoints for frontend extensions to access server-side
+ephemeral storage without direct backend code.
+
+By default, all operations are user-scoped (private to the current user).
+Use the /shared/ endpoints to access state visible to all users.
+"""
+
+from __future__ import annotations
+
+from typing import Any
+
+from flask import g, request
+from flask.wrappers import Response
+from flask_appbuilder.api import BaseApi, expose, protect, safe
+
+from superset.extensions import cache_manager
+from superset.extensions.types import LoadedExtension
+from superset.extensions.utils import get_extensions
+
+# Key separator
+SEPARATOR = ":"
+
+# Key prefix for extension ephemeral state
+KEY_PREFIX = "superset-ext"
+
+# Default TTL: 1 hour
+DEFAULT_TTL = 3600
+
+
+def _build_cache_key(*parts: Any) -> str:
+ """Build a namespaced cache key from parts."""
+ return SEPARATOR.join(str(part) for part in parts)
+
+
+def _get_extension_or_404(extension_id: str) -> LoadedExtension | None:
+ """Get extension by ID or return None if not found."""
+ extensions = get_extensions()
+ return extensions.get(extension_id)
+
+
+class ExtensionStorageRestApi(BaseApi):
+ """REST API for extension ephemeral state storage."""
+
+ allow_browser_login = True
+ route_base = "/api/v1/extensions/storage"
+
+ def response(self, status_code: int, **kwargs: Any) -> Response:
+ """Helper method to create JSON responses."""
+ from flask import jsonify
+
+ return jsonify(kwargs), status_code
+
+ def response_404(self, message: str = "Not found") -> Response:
+ """Helper method to create 404 responses."""
+ from flask import jsonify
+
+ return jsonify({"message": message}), 404
+
+ def response_400(self, message: str) -> Response:
+ """Helper method to create 400 responses."""
+ from flask import jsonify
+
+ return jsonify({"message": message}), 400
+
+ # =========================================================================
+ # User-Scoped Ephemeral State Endpoints (Default)
+ # =========================================================================
+
+ @protect()
+ @safe
+ @expose("/ephemeral/<extension_id>/<key>", methods=("GET",))
+ def get_ephemeral(self, extension_id: str, key: str, **kwargs: Any) -> Response:
+ """Get a value from user-scoped ephemeral state.
+ ---
+ get:
+ summary: Get a value from user-scoped ephemeral state (default)
+ parameters:
+ - in: path
+ name: extension_id
+ schema:
+ type: string
+ required: true
+ description: Extension ID (publisher.name)
+ - in: path
+ name: key
+ schema:
+ type: string
+ required: true
+ description: Storage key
+ responses:
+ 200:
+ description: Value retrieved successfully
+ content:
+ application/json:
+ schema:
+ type: object
+ properties:
+ result:
+ description: The stored value
+ 404:
+ description: Extension not found
+ """
+ extension = _get_extension_or_404(extension_id)
+ if not extension:
+ return self.response_404("Extension not found")
+
+ user_id = g.user.id
+ cache_key = _build_cache_key(KEY_PREFIX, extension_id, "user", user_id, key)
+ value = cache_manager.extension_ephemeral_state_cache.get(cache_key)
+
+ return self.response(200, result=value)
+
+ @protect()
+ @safe
+ @expose("/ephemeral/<extension_id>/<key>", methods=("PUT",))
+ def set_ephemeral(self, extension_id: str, key: str, **kwargs: Any) -> Response:
+ """Set a value in user-scoped ephemeral state.
+ ---
+ put:
+ summary: Set a value in user-scoped ephemeral state (default)
+ parameters:
+ - in: path
+ name: extension_id
+ schema:
+ type: string
+ required: true
+ description: Extension ID (publisher.name)
+ - in: path
+ name: key
+ schema:
+ type: string
+ required: true
+ description: Storage key
+ requestBody:
+ required: true
+ content:
+ application/json:
+ schema:
+ type: object
+ properties:
+ value:
+ description: The value to store
+ ttl:
+ type: integer
+ description: Time-to-live in seconds (default 3600)
+ responses:
+ 200:
+ description: Value stored successfully
+ 400:
+ description: Invalid request body
+ 404:
+ description: Extension not found
+ """
+ extension = _get_extension_or_404(extension_id)
+ if not extension:
+ return self.response_404("Extension not found")
+
+ body = request.get_json(silent=True) or {}
+ if "value" not in body:
+ return self.response_400("Request body must contain 'value' field")
+
+ value = body["value"]
+ ttl = body.get("ttl", DEFAULT_TTL)
+
+ user_id = g.user.id
+ cache_key = _build_cache_key(KEY_PREFIX, extension_id, "user", user_id, key)
+ cache_manager.extension_ephemeral_state_cache.set(cache_key, value, timeout=ttl)
+
+ return self.response(200, message="Value stored successfully")
+
+ @protect()
+ @safe
+ @expose("/ephemeral/<extension_id>/<key>", methods=("DELETE",))
+ def delete_ephemeral(self, extension_id: str, key: str, **kwargs: Any) -> Response:
+ """Delete a value from user-scoped ephemeral state.
+ ---
+ delete:
+ summary: Delete a value from user-scoped ephemeral state (default)
+ parameters:
+ - in: path
+ name: extension_id
+ schema:
+ type: string
+ required: true
+ description: Extension ID (publisher.name)
+ - in: path
+ name: key
+ schema:
+ type: string
+ required: true
+ description: Storage key
+ responses:
+ 200:
+ description: Value deleted successfully
+ 404:
+ description: Extension not found
+ """
+ extension = _get_extension_or_404(extension_id)
+ if not extension:
+ return self.response_404("Extension not found")
+
+ user_id = g.user.id
+ cache_key = _build_cache_key(KEY_PREFIX, extension_id, "user", user_id, key)
+ cache_manager.extension_ephemeral_state_cache.delete(cache_key)
+
+ return self.response(200, message="Value deleted successfully")
+
+ # =========================================================================
+ # Shared (Global) Ephemeral State Endpoints
+ # =========================================================================
+
+ @protect()
+ @safe
+ @expose("/ephemeral/shared/<extension_id>/<key>", methods=("GET",))
+ def get_ephemeral_shared(
+ self, extension_id: str, key: str, **kwargs: Any
+ ) -> Response:
+ """Get a value from shared ephemeral state.
+ ---
+ get:
+ summary: Get a value from shared (global) ephemeral state
+ parameters:
+ - in: path
+ name: extension_id
+ schema:
+ type: string
+ required: true
+ description: Extension ID (publisher.name)
+ - in: path
+ name: key
+ schema:
+ type: string
+ required: true
+ description: Storage key
+ responses:
+ 200:
+ description: Value retrieved successfully
+ content:
+ application/json:
+ schema:
+ type: object
+ properties:
+ result:
+ description: The stored value
+ 404:
+ description: Extension not found
+ """
+ extension = _get_extension_or_404(extension_id)
+ if not extension:
+ return self.response_404("Extension not found")
+
+ cache_key = _build_cache_key(KEY_PREFIX, extension_id, key)
Review Comment:
**Suggestion:** The shared GET endpoint allows keys like `user:<id>:<key>`, which resolves to the same cache key format used by private user-scoped storage and can leak private values through the shared API. Reject reserved `user:`-prefixed keys before reading shared state. [security]
<details>
<summary><b>Severity Level:</b> Critical π¨</summary>
```mdx
- β Shared GET exposes other users' ephemeral extension state.
- β Breaks user isolation promised in ExtensionStorageRestApi.
- β οΈ Affects all extensions using shared ephemeral storage.
```
</details>
```suggestion
if key.startswith("user:"):
return self.response_400("Invalid key prefix for shared storage")
```
<details>
<summary><b>Steps of Reproduction β
</b></summary>
```mdx
1. Configure at least one extension so `get_extensions()` returns it (see
`superset/extensions/utils.py:252-38` where extensions are loaded from `LOCAL_EXTENSIONS`
/ `EXTENSIONS_PATH` and expose a manifest `id` used as `extension_id`).
2. As user A (with some `g.user.id`), call the user-scoped set endpoint `PUT
/api/v1/extensions/storage/ephemeral/<extension_id>/job_progress` implemented by
`ExtensionStorageRestApi.set_ephemeral` in `superset/extensions/storage/api.py:135-189`
with body `{"value": {"pct": 42}, "ttl": 3600}` so line `186 cache_key =
_build_cache_key(KEY_PREFIX, extension_id, "user", user_id, key)` builds key
`superset-ext:<extension_id>:user:<userA_id>:job_progress` and line `187` stores the
private value in `extension_ephemeral_state_cache`.
3. As a different logged-in user B, call the shared GET endpoint `GET
/api/v1/extensions/storage/ephemeral/shared/<extension_id>/user:<userA_id>:job_progress`
handled by `get_ephemeral_shared` in `superset/extensions/storage/api.py:234-275`; at
lines `272-273` the code `cache_key = _build_cache_key(KEY_PREFIX, extension_id, key)` and
`value = cache_manager.extension_ephemeral_state_cache.get(cache_key)` compute the same
concrete cache key `superset-ext:<extension_id>:user:<userA_id>:job_progress` and fetch
user A's supposedly private value.
4. Observe that the shared endpoint returns `{"result": {"pct": 42}}` (line `275 return
self.response(200, result=value)`), leaking user A's user-scoped ephemeral state through
the shared API due to the shared key `user:<id>:<key>` colliding exactly with the private
key pattern.
```
</details>
<details>
<summary><b>Prompt for AI Agent π€ </b></summary>
```mdx
This is a comment left during a code review.
**Path:** superset/extensions/storage/api.py
**Line:** 272:272
**Comment:**
*Security: The shared GET endpoint allows keys like `user:<id>:<key>`, which resolves to the same cache key format used by private user-scoped storage and can leak private values through the shared API. Reject reserved `user:`-prefixed keys before reading shared state.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F39171&comment_hash=54e3abb42ff5f5a775b4bd6914d5f7192ce9866c11c1dc3f525db5efcf40c310&reaction=like'>π</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F39171&comment_hash=54e3abb42ff5f5a775b4bd6914d5f7192ce9866c11c1dc3f525db5efcf40c310&reaction=dislike'>π</a>
##########
superset/extensions/storage/api.py:
##########
@@ -0,0 +1,372 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""
+REST API for extension storage.
+
+Provides HTTP endpoints for frontend extensions to access server-side
+ephemeral storage without direct backend code.
+
+By default, all operations are user-scoped (private to the current user).
+Use the /shared/ endpoints to access state visible to all users.
+"""
+
+from __future__ import annotations
+
+from typing import Any
+
+from flask import g, request
+from flask.wrappers import Response
+from flask_appbuilder.api import BaseApi, expose, protect, safe
+
+from superset.extensions import cache_manager
+from superset.extensions.types import LoadedExtension
+from superset.extensions.utils import get_extensions
+
+# Key separator
+SEPARATOR = ":"
+
+# Key prefix for extension ephemeral state
+KEY_PREFIX = "superset-ext"
+
+# Default TTL: 1 hour
+DEFAULT_TTL = 3600
+
+
+def _build_cache_key(*parts: Any) -> str:
+ """Build a namespaced cache key from parts."""
+ return SEPARATOR.join(str(part) for part in parts)
+
+
+def _get_extension_or_404(extension_id: str) -> LoadedExtension | None:
+ """Get extension by ID or return None if not found."""
+ extensions = get_extensions()
+ return extensions.get(extension_id)
+
+
+class ExtensionStorageRestApi(BaseApi):
+ """REST API for extension ephemeral state storage."""
+
+ allow_browser_login = True
+ route_base = "/api/v1/extensions/storage"
+
+ def response(self, status_code: int, **kwargs: Any) -> Response:
+ """Helper method to create JSON responses."""
+ from flask import jsonify
+
+ return jsonify(kwargs), status_code
+
+ def response_404(self, message: str = "Not found") -> Response:
+ """Helper method to create 404 responses."""
+ from flask import jsonify
+
+ return jsonify({"message": message}), 404
+
+ def response_400(self, message: str) -> Response:
+ """Helper method to create 400 responses."""
+ from flask import jsonify
+
+ return jsonify({"message": message}), 400
+
+ # =========================================================================
+ # User-Scoped Ephemeral State Endpoints (Default)
+ # =========================================================================
+
+ @protect()
+ @safe
+ @expose("/ephemeral/<extension_id>/<key>", methods=("GET",))
+ def get_ephemeral(self, extension_id: str, key: str, **kwargs: Any) -> Response:
+ """Get a value from user-scoped ephemeral state.
+ ---
+ get:
+ summary: Get a value from user-scoped ephemeral state (default)
+ parameters:
+ - in: path
+ name: extension_id
+ schema:
+ type: string
+ required: true
+ description: Extension ID (publisher.name)
+ - in: path
+ name: key
+ schema:
+ type: string
+ required: true
+ description: Storage key
+ responses:
+ 200:
+ description: Value retrieved successfully
+ content:
+ application/json:
+ schema:
+ type: object
+ properties:
+ result:
+ description: The stored value
+ 404:
+ description: Extension not found
+ """
+ extension = _get_extension_or_404(extension_id)
+ if not extension:
+ return self.response_404("Extension not found")
+
+ user_id = g.user.id
+ cache_key = _build_cache_key(KEY_PREFIX, extension_id, "user", user_id, key)
+ value = cache_manager.extension_ephemeral_state_cache.get(cache_key)
+
+ return self.response(200, result=value)
+
+ @protect()
+ @safe
+ @expose("/ephemeral/<extension_id>/<key>", methods=("PUT",))
+ def set_ephemeral(self, extension_id: str, key: str, **kwargs: Any) -> Response:
+ """Set a value in user-scoped ephemeral state.
+ ---
+ put:
+ summary: Set a value in user-scoped ephemeral state (default)
+ parameters:
+ - in: path
+ name: extension_id
+ schema:
+ type: string
+ required: true
+ description: Extension ID (publisher.name)
+ - in: path
+ name: key
+ schema:
+ type: string
+ required: true
+ description: Storage key
+ requestBody:
+ required: true
+ content:
+ application/json:
+ schema:
+ type: object
+ properties:
+ value:
+ description: The value to store
+ ttl:
+ type: integer
+ description: Time-to-live in seconds (default 3600)
+ responses:
+ 200:
+ description: Value stored successfully
+ 400:
+ description: Invalid request body
+ 404:
+ description: Extension not found
+ """
+ extension = _get_extension_or_404(extension_id)
+ if not extension:
+ return self.response_404("Extension not found")
+
+ body = request.get_json(silent=True) or {}
+ if "value" not in body:
+ return self.response_400("Request body must contain 'value' field")
+
+ value = body["value"]
+ ttl = body.get("ttl", DEFAULT_TTL)
+
+ user_id = g.user.id
+ cache_key = _build_cache_key(KEY_PREFIX, extension_id, "user", user_id, key)
+ cache_manager.extension_ephemeral_state_cache.set(cache_key, value, timeout=ttl)
+
+ return self.response(200, message="Value stored successfully")
+
+ @protect()
+ @safe
+ @expose("/ephemeral/<extension_id>/<key>", methods=("DELETE",))
+ def delete_ephemeral(self, extension_id: str, key: str, **kwargs: Any) -> Response:
+ """Delete a value from user-scoped ephemeral state.
+ ---
+ delete:
+ summary: Delete a value from user-scoped ephemeral state (default)
+ parameters:
+ - in: path
+ name: extension_id
+ schema:
+ type: string
+ required: true
+ description: Extension ID (publisher.name)
+ - in: path
+ name: key
+ schema:
+ type: string
+ required: true
+ description: Storage key
+ responses:
+ 200:
+ description: Value deleted successfully
+ 404:
+ description: Extension not found
+ """
+ extension = _get_extension_or_404(extension_id)
+ if not extension:
+ return self.response_404("Extension not found")
+
+ user_id = g.user.id
+ cache_key = _build_cache_key(KEY_PREFIX, extension_id, "user", user_id, key)
+ cache_manager.extension_ephemeral_state_cache.delete(cache_key)
+
+ return self.response(200, message="Value deleted successfully")
+
+ # =========================================================================
+ # Shared (Global) Ephemeral State Endpoints
+ # =========================================================================
+
+ @protect()
+ @safe
+ @expose("/ephemeral/shared/<extension_id>/<key>", methods=("GET",))
+ def get_ephemeral_shared(
+ self, extension_id: str, key: str, **kwargs: Any
+ ) -> Response:
+ """Get a value from shared ephemeral state.
+ ---
+ get:
+ summary: Get a value from shared (global) ephemeral state
+ parameters:
+ - in: path
+ name: extension_id
+ schema:
+ type: string
+ required: true
+ description: Extension ID (publisher.name)
+ - in: path
+ name: key
+ schema:
+ type: string
+ required: true
+ description: Storage key
+ responses:
+ 200:
+ description: Value retrieved successfully
+ content:
+ application/json:
+ schema:
+ type: object
+ properties:
+ result:
+ description: The stored value
+ 404:
+ description: Extension not found
+ """
+ extension = _get_extension_or_404(extension_id)
+ if not extension:
+ return self.response_404("Extension not found")
+
+ cache_key = _build_cache_key(KEY_PREFIX, extension_id, key)
+ value = cache_manager.extension_ephemeral_state_cache.get(cache_key)
+
+ return self.response(200, result=value)
+
+ @protect()
+ @safe
+ @expose("/ephemeral/shared/<extension_id>/<key>", methods=("PUT",))
+ def set_ephemeral_shared(
+ self, extension_id: str, key: str, **kwargs: Any
+ ) -> Response:
+ """Set a value in shared ephemeral state.
+ ---
+ put:
+ summary: Set a value in shared (global) ephemeral state
+ parameters:
+ - in: path
+ name: extension_id
+ schema:
+ type: string
+ required: true
+ description: Extension ID (publisher.name)
+ - in: path
+ name: key
+ schema:
+ type: string
+ required: true
+ description: Storage key
+ requestBody:
+ required: true
+ content:
+ application/json:
+ schema:
+ type: object
+ properties:
+ value:
+ description: The value to store
+ ttl:
+ type: integer
+ description: Time-to-live in seconds (default 3600)
+ responses:
+ 200:
+ description: Value stored successfully
+ 400:
+ description: Invalid request body
+ 404:
+ description: Extension not found
+ """
+ extension = _get_extension_or_404(extension_id)
+ if not extension:
+ return self.response_404("Extension not found")
+
+ body = request.get_json(silent=True) or {}
+ if "value" not in body:
+ return self.response_400("Request body must contain 'value' field")
+
+ value = body["value"]
+ ttl = body.get("ttl", DEFAULT_TTL)
+
+ cache_key = _build_cache_key(KEY_PREFIX, extension_id, key)
+ cache_manager.extension_ephemeral_state_cache.set(cache_key, value, timeout=ttl)
+
+ return self.response(200, message="Value stored successfully")
+
+ @protect()
+ @safe
+ @expose("/ephemeral/shared/<extension_id>/<key>", methods=("DELETE",))
+ def delete_ephemeral_shared(
+ self, extension_id: str, key: str, **kwargs: Any
+ ) -> Response:
+ """Delete a value from shared ephemeral state.
+ ---
+ delete:
+ summary: Delete a value from shared (global) ephemeral state
+ parameters:
+ - in: path
+ name: extension_id
+ schema:
+ type: string
+ required: true
+ description: Extension ID (publisher.name)
+ - in: path
+ name: key
+ schema:
+ type: string
+ required: true
+ description: Storage key
+ responses:
+ 200:
+ description: Value deleted successfully
+ 404:
+ description: Extension not found
+ """
+ extension = _get_extension_or_404(extension_id)
+ if not extension:
+ return self.response_404("Extension not found")
+
+ cache_key = _build_cache_key(KEY_PREFIX, extension_id, key)
Review Comment:
**Suggestion:** The shared DELETE endpoint can delete private user-scoped values via crafted `user:`-prefixed keys because shared and private key formats overlap. Block reserved `user:` keys before deletion. [security]
<details>
<summary><b>Severity Level:</b> Critical π¨</summary>
```mdx
- β Shared DELETE can erase other users' ephemeral data.
- β Violates isolation between shared and user-scoped storage.
- β οΈ Causes unpredictable behavior in extensions using ephemeral state.
```
</details>
```suggestion
if key.startswith("user:"):
return self.response_400("Invalid key prefix for shared storage")
```
<details>
<summary><b>Steps of Reproduction β
</b></summary>
```mdx
1. Register an extension so `extension_id` resolves via `_get_extension_or_404` in
`superset/extensions/storage/api.py:55-58`, and configure
`EXTENSIONS_STORAGE['EPHEMERAL']` so `CacheManager.init_app` wires
`extension_ephemeral_state_cache` (see `superset/utils/cache_manager.py:58-76`).
2. As user A, create a user-scoped entry by calling `PUT
/api/v1/extensions/storage/ephemeral/<extension_id>/job_progress` handled by
`set_ephemeral` in `superset/extensions/storage/api.py:135-189`; line `186` builds
`cache_key = _build_cache_key(KEY_PREFIX, extension_id, "user", user_id, key)` and line
`187` stores the value under `superset-ext:<extension_id>:user:<userA_id>:job_progress`.
3. As user B, invoke the shared DELETE endpoint `DELETE
/api/v1/extensions/storage/ephemeral/shared/<extension_id>/user:<userA_id>:job_progress`
implemented by `delete_ephemeral_shared` in `superset/extensions/storage/api.py:79-113`;
after extension validation at lines `106-108`, lines `110-111` (full file lines `369-370`)
execute `cache_key = _build_cache_key(KEY_PREFIX, extension_id, key)` and
`cache_manager.extension_ephemeral_state_cache.delete(cache_key)`, computing the exact
same underlying key `superset-ext:<extension_id>:user:<userA_id>:job_progress`.
4. A subsequent call by user A to `GET
/api/v1/extensions/storage/ephemeral/<extension_id>/job_progress` via `get_ephemeral` at
`superset/extensions/storage/api.py:91-131` will return `{"result": null}` because the
shared delete removed A's private entry, demonstrating that crafted shared keys
`user:<id>:<key>` can delete user-scoped state.
```
</details>
<details>
<summary><b>Prompt for AI Agent π€ </b></summary>
```mdx
This is a comment left during a code review.
**Path:** superset/extensions/storage/api.py
**Line:** 369:369
**Comment:**
*Security: The shared DELETE endpoint can delete private user-scoped values via crafted `user:`-prefixed keys because shared and private key formats overlap. Block reserved `user:` keys before deletion.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F39171&comment_hash=34b7659f50be080c35bbb50739f2ff8ceb1363bb267290d43bed19ee4ee3d05f&reaction=like'>π</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F39171&comment_hash=34b7659f50be080c35bbb50739f2ff8ceb1363bb267290d43bed19ee4ee3d05f&reaction=dislike'>π</a>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(extensions): add a Storage API for extensions [superset]
Posted by "michael-s-molina (via GitHub)" <gi...@apache.org>.
michael-s-molina commented on code in PR #39171:
URL: https://github.com/apache/superset/pull/39171#discussion_r3560961803
##########
superset/extensions/storage/persistent_dao.py:
##########
@@ -0,0 +1,313 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from __future__ import annotations
+
+import hashlib
+import hmac
+import logging
+
+from flask import current_app
+from flask_babel import gettext as _
+from sqlalchemy import and_, func, LargeBinary
+
+from superset import db
+from superset.daos.base import BaseDAO
+from superset.exceptions import SupersetGenericErrorException
+from superset.extensions.storage.filters import ExtensionStorageFilter
+from superset.extensions.storage.persistent_model import ExtensionStorage
+from superset.utils.encrypt import (
+ DEFAULT_ENCRYPTION_ENGINE_NAME,
+ EncryptedType,
+ resolve_encryption_engine,
+)
+
+logger = logging.getLogger(__name__)
+
+
+class ExtensionStorageQuotaExceeded(SupersetGenericErrorException):
+ """Raised when a write would exceed an extension's persistent storage quota."""
+
+ status = 413
+
+ def __init__(self, extension_id: str, quota: int) -> None:
+ super().__init__(
+ message=_(
+ "Extension '%(extension_id)s' has exceeded its persistent "
+ "storage quota of %(quota)d bytes.",
+ extension_id=extension_id,
+ quota=quota,
+ ),
+ )
+
+
+def _derive_key(secret: str, user_fk: int) -> str:
+ """Derive a per-user encryption key from the master secret and user ID.
+
+ Uses HMAC-SHA256 so the derived key is cryptographically bound to both.
+ Ciphertext encrypted for one user cannot be decrypted as another.
+ """
+ return hmac.new(
+ secret.encode(),
+ str(user_fk).encode(),
+ hashlib.sha256,
+ ).hexdigest()
+
+
+def _enc_type(user_fk: int | None) -> EncryptedType:
+ """Return an EncryptedType for the given scope.
+
+ User-scoped rows use a key derived from SECRET_KEY and the user ID, so
+ ciphertext written for one user cannot be decrypted as another.
+ Shared rows (user_fk=None) use SECRET_KEY directly.
+ """
+ secret = current_app.config["SECRET_KEY"]
+ key = _derive_key(secret, user_fk) if user_fk is not None else secret
+ engine_name = current_app.config.get(
+ "SQLALCHEMY_ENCRYPTED_FIELD_ENGINE", DEFAULT_ENCRYPTION_ENGINE_NAME
+ )
+ return EncryptedType(
+ LargeBinary,
+ key=key,
+ engine=resolve_encryption_engine(engine_name),
+ )
+
+
+def _get_quota() -> int | None:
+ """Return the configured per-extension persistent storage quota in bytes.
+
+ Returns None when no quota is configured (unlimited).
+ """
+ config = current_app.config.get("EXTENSIONS_PERSISTENT_STORAGE", {})
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(extensions): add a Storage API for extensions [superset]
Posted by "michael-s-molina (via GitHub)" <gi...@apache.org>.
michael-s-molina commented on code in PR #39171:
URL: https://github.com/apache/superset/pull/39171#discussion_r3560991600
##########
superset/extensions/storage/persistent_dao.py:
##########
@@ -0,0 +1,313 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from __future__ import annotations
+
+import hashlib
+import hmac
+import logging
+
+from flask import current_app
+from flask_babel import gettext as _
+from sqlalchemy import and_, func, LargeBinary
+
+from superset import db
+from superset.daos.base import BaseDAO
+from superset.exceptions import SupersetGenericErrorException
+from superset.extensions.storage.filters import ExtensionStorageFilter
+from superset.extensions.storage.persistent_model import ExtensionStorage
+from superset.utils.encrypt import (
+ DEFAULT_ENCRYPTION_ENGINE_NAME,
+ EncryptedType,
+ resolve_encryption_engine,
+)
+
+logger = logging.getLogger(__name__)
+
+
+class ExtensionStorageQuotaExceeded(SupersetGenericErrorException):
+ """Raised when a write would exceed an extension's persistent storage quota."""
+
+ status = 413
+
+ def __init__(self, extension_id: str, quota: int) -> None:
+ super().__init__(
+ message=_(
+ "Extension '%(extension_id)s' has exceeded its persistent "
+ "storage quota of %(quota)d bytes.",
+ extension_id=extension_id,
+ quota=quota,
+ ),
+ )
+
+
+def _derive_key(secret: str, user_fk: int) -> str:
+ """Derive a per-user encryption key from the master secret and user ID.
+
+ Uses HMAC-SHA256 so the derived key is cryptographically bound to both.
+ Ciphertext encrypted for one user cannot be decrypted as another.
+ """
+ return hmac.new(
+ secret.encode(),
+ str(user_fk).encode(),
+ hashlib.sha256,
+ ).hexdigest()
+
+
+def _enc_type(user_fk: int | None) -> EncryptedType:
+ """Return an EncryptedType for the given scope.
+
+ User-scoped rows use a key derived from SECRET_KEY and the user ID, so
+ ciphertext written for one user cannot be decrypted as another.
+ Shared rows (user_fk=None) use SECRET_KEY directly.
+ """
+ secret = current_app.config["SECRET_KEY"]
+ key = _derive_key(secret, user_fk) if user_fk is not None else secret
+ engine_name = current_app.config.get(
+ "SQLALCHEMY_ENCRYPTED_FIELD_ENGINE", DEFAULT_ENCRYPTION_ENGINE_NAME
+ )
+ return EncryptedType(
+ LargeBinary,
+ key=key,
+ engine=resolve_encryption_engine(engine_name),
+ )
+
+
+def _get_quota() -> int | None:
+ """Return the configured per-extension persistent storage quota in bytes.
+
+ Returns None when no quota is configured (unlimited).
+ """
+ config = current_app.config.get("EXTENSIONS_PERSISTENT_STORAGE", {})
+ return config.get("QUOTA_PER_EXTENSION")
+
+
+def _check_quota(extension_id: str, new_size: int, existing_size: int = 0) -> None:
+ """Raise if writing `new_size` bytes would push the extension over quota.
+
+ `existing_size` is the size of the row being replaced (0 for inserts),
+ subtracted from current usage so overwriting a key doesn't double-count
+ it against the quota it already occupies.
+ """
+ quota = _get_quota()
+ if quota is None:
+ return
+ current_usage = (
+ db.session.query(
+ func.coalesce(func.sum(func.length(ExtensionStorage.value)), 0)
+ )
+ .filter(ExtensionStorage.extension_id == extension_id)
+ .scalar()
+ )
+ if current_usage - existing_size + new_size > quota:
+ raise ExtensionStorageQuotaExceeded(extension_id, quota)
Review Comment:
To optimize quota calculations, I updated our strategy to store sizes at the time of insertion. I evaluated using a separate table or Redis, but the added complexity wasn't worth it. A simple SUM on a numeric column gives us the efficiency we need.
##########
superset/extensions/storage/persistent_model.py:
##########
@@ -0,0 +1,129 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import uuid as uuid_module
+
+from flask_appbuilder import Model
+from sqlalchemy import (
+ Boolean,
+ Column,
+ ForeignKey,
+ Index,
+ Integer,
+ LargeBinary,
+ String,
+ Text,
+ UniqueConstraint,
+)
+from sqlalchemy.orm import backref, relationship
+from sqlalchemy_utils import UUIDType
+from superset_core.extensions.storage.models import (
+ ExtensionStorageEntry as CoreExtensionStorageEntry,
+)
+
+from superset.models.helpers import AuditMixinNullable
+
+# 16 MB β matches the KeyValue store limit.
+EXTENSION_STORAGE_MAX_SIZE = 2**24 - 1
+
+
+class ExtensionStorage(CoreExtensionStorageEntry, AuditMixinNullable, Model):
+ """Generic persistent key-value storage for extensions (Tier 3).
+
+ Each row is identified by (extension_id, user_fk, resource_type,
+ resource_uuid, key):
+
+ * Global scope β user_fk IS NULL, resource_type IS NULL
+ * User scope β user_fk set, resource_type IS NULL
+ * Resource scope β resource_type + resource_uuid set (user_fk optional)
+
+ The payload is stored as raw bytes (value) with a MIME-type hint
+ (value_type). When is_encrypted is True the value has been encrypted
+ at the DAO layer using Fernet and must be decrypted before use.
+ """
+
+ __tablename__ = "extension_storage"
+
+ id = Column(Integer, primary_key=True, autoincrement=True)
+ uuid = Column(
+ UUIDType(binary=True),
+ default=uuid_module.uuid4,
+ unique=True,
+ nullable=False,
+ )
+
+ # Extension identity
+ extension_id = Column(String(255), nullable=False)
+
+ # Scope discriminators β all nullable; NULLs define the scope (see docstring)
+ user_fk = Column(
+ Integer,
+ ForeignKey(
+ "ab_user.id",
+ ondelete="SET NULL",
+ name="fk_extension_storage_user_fk_ab_user",
+ ),
+ nullable=True,
+ )
+ resource_type = Column(String(64), nullable=True)
+ resource_uuid = Column(String(36), nullable=True)
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(extensions): add a Storage API for extensions [superset]
Posted by "michael-s-molina (via GitHub)" <gi...@apache.org>.
michael-s-molina commented on code in PR #39171:
URL: https://github.com/apache/superset/pull/39171#discussion_r3561219232
##########
superset-frontend/packages/superset-core/src/storage/index.ts:
##########
@@ -0,0 +1,208 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/**
+ * @fileoverview Shared types for extension storage APIs.
+ *
+ * These types are shared across all storage tiers (local, session, ephemeral,
+ * persistent) to ensure a consistent API pattern.
+ */
+
+/**
+ * JSON-compatible value type.
+ * These are the only values that can be safely serialized/deserialized via JSON.
+ */
+export type JsonValue =
+ string | number | boolean | null | JsonValue[] | { [key: string]: JsonValue };
+
+/**
+ * Base interface for a storage accessor.
+ * All storage tiers implement this interface for both user-scoped and shared access.
+ */
+export interface StorageAccessor {
+ /**
+ * Get a value from storage.
+ *
+ * @param key The key to retrieve.
+ * @returns The stored value, or null if not found.
+ */
+ get<T = JsonValue>(key: string): Promise<T | null>;
+
+ /**
+ * Set a value in storage.
+ *
+ * @param key The key to store.
+ * @param value The value to store (must be JSON-serializable).
+ */
+ set<T = JsonValue>(key: string, value: T): Promise<void>;
Review Comment:
Not a bug β by design. `T` here is a generic default (`T = JsonValue`), not a constraint. It's intentionally unconstrained because `set()` supports non-JSON values via the `codec` option (`base64` for binary, `pickle` for arbitrary Python objects on the backend) β constraining `T extends JsonValue` would make those codec paths untypeable. See `PersistentSetOptions.codec`/`EphemeralSetOptions.codec`.
##########
superset-frontend/src/core/storage/binaryCodec.ts:
##########
@@ -0,0 +1,70 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/** Convert bytes to a base64 string, for transport over the JSON REST API. */
+export function bytesToBase64(bytes: Uint8Array): string {
+ let binary = '';
+ bytes.forEach(byte => {
+ binary += String.fromCharCode(byte);
+ });
+ return btoa(binary);
+}
+
+function isBinaryValue(value: unknown): value is Uint8Array | ArrayBuffer {
+ return value instanceof Uint8Array || value instanceof ArrayBuffer;
+}
+
+/**
+ * Infer which codec to use for `value` when the caller didn't specify one.
+ *
+ * "json" names the serialization format applied to encode the value for
+ * storage/transport -- it is not a claim about the value's shape. Per
+ * RFC 8259, a JSON text may be any JSON value (object, array, string,
+ * number, boolean, or null), so scalars like `1` or `"foo"` are encoded
+ * with the "json" codec too, alongside objects and arrays.
+ *
+ * Binary values (Uint8Array/ArrayBuffer) are the one case JSON cannot
+ * represent, so they're base64-encoded instead.
+ */
+function inferDefaultCodec(value: unknown): 'json' | 'base64' {
+ return isBinaryValue(value) ? 'base64' : 'json';
+}
+
+/**
+ * Resolve the codec and request-body value for a `set()` call.
+ *
+ * If the caller explicitly set `options.codec`, it is always respected
+ * as-is and `value` is passed through untouched -- the caller is
+ * responsible for encoding it correctly (e.g. base64-encoding it
+ * themselves before calling `set()`). Auto base64-encoding only happens
+ * when no codec was specified and `value` is binary.
+ */
+export function resolveSetPayload<T>(
+ value: T,
+ explicitCodec: string | undefined,
+): { value: unknown; codec: string } {
+ if (explicitCodec !== undefined) {
+ return { value, codec: explicitCodec };
Review Comment:
Not a bug β this is the intended contract. `resolveSetPayload` only auto-encodes binary values when the caller leaves `codec` unset; once a caller passes an explicit `codec`, we respect it as-is and never reinterpret `value`, so the caller stays fully in control of encoding (including for `pickle`, which isn't JSON/base64-shaped at all). This is documented in the function's own docstring and covered by the test `'does not double-encode when the caller already base64-encoded a binary value'` in `binaryCodec.test.ts`.
##########
superset-frontend/src/core/storage/ephemeralState.test.ts:
##########
@@ -0,0 +1,144 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import { SupersetClient } from '@superset-ui/core';
+import { createEphemeralState } from './ephemeralState';
+
+jest.mock('@superset-ui/core', () => ({
+ SupersetClient: {
+ get: jest.fn(),
+ put: jest.fn(),
+ delete: jest.fn(),
+ },
+}));
+
+const mockGet = SupersetClient.get as jest.Mock;
+const mockPut = SupersetClient.put as jest.Mock;
+const mockDelete = SupersetClient.delete as jest.Mock;
+
+beforeEach(() => {
+ jest.clearAllMocks();
+ mockGet.mockResolvedValue({ json: { result: null } });
+ mockPut.mockResolvedValue({});
+ mockDelete.mockResolvedValue({});
+});
+
+test('get calls correct URL with publisher/name pattern', async () => {
+ const store = createEphemeralState('myorg.myext');
+ await store.get('job_progress');
+ expect(mockGet).toHaveBeenCalledWith({
+ endpoint: '/api/v1/extensions/myorg/myext/storage/ephemeral/job_progress',
+ });
+});
+
+test('get returns result from response', async () => {
+ mockGet.mockResolvedValue({ json: { result: { pct: 42 } } });
+ const store = createEphemeralState('myorg.myext');
+ const result = await store.get('job_progress');
+ expect(result).toEqual({ pct: 42 });
+});
+
+test('get returns null when result is absent', async () => {
+ mockGet.mockResolvedValue({ json: {} });
+ const store = createEphemeralState('myorg.myext');
+ expect(await store.get('key')).toBeNull();
+});
+
+test('set calls correct URL and includes value and ttl in body', async () => {
+ const store = createEphemeralState('myorg.myext');
+ await store.set('job_progress', { pct: 42 }, { ttl: 300 });
+ expect(mockPut).toHaveBeenCalledWith({
+ endpoint: '/api/v1/extensions/myorg/myext/storage/ephemeral/job_progress',
+ body: JSON.stringify({ value: { pct: 42 }, ttl: 300, codec: 'json' }),
+ headers: { 'Content-Type': 'application/json' },
+ });
+});
+
+test('set passes codec from options', async () => {
+ const store = createEphemeralState('myorg.myext');
+ await store.set('job_progress', 'sk-...', { ttl: 300, codec: 'pickle' });
+ expect(mockPut).toHaveBeenCalledWith({
+ endpoint: '/api/v1/extensions/myorg/myext/storage/ephemeral/job_progress',
+ body: JSON.stringify({ value: 'sk-...', ttl: 300, codec: 'pickle' }),
+ headers: { 'Content-Type': 'application/json' },
+ });
Review Comment:
Not a bug β this test exercises the SDK's codec-passthrough behavior (what gets sent in the request), not a live round-trip against the server's `SAFE_CODECS` allowlist. Asserting that an explicit `codec: 'pickle'` is forwarded as given is correct per the client's contract (see `binaryCodec.ts`'s "explicit codec always respected" design). Whether the server accepts `pickle` for a given call depends on which path is used β it's rejected over REST but valid for direct backend calls β which is a separate, already-covered concern on the server side (`SAFE_CODECS` in `superset/extensions/storage/codecs.py`).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(extensions): add a Storage API for extensions [superset]
Posted by "codeant-ai-for-open-source[bot] (via GitHub)" <gi...@apache.org>.
codeant-ai-for-open-source[bot] commented on code in PR #39171:
URL: https://github.com/apache/superset/pull/39171#discussion_r3561088602
##########
superset/extensions/storage/persistent_dao.py:
##########
@@ -0,0 +1,527 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from __future__ import annotations
+
+import hashlib
+import hmac
+import logging
+from dataclasses import dataclass
+from typing import Any
+
+from flask import current_app
+from flask_babel import gettext as _
+from sqlalchemy import and_, desc, func, LargeBinary
+
+from superset import db
+from superset.daos.base import BaseDAO
+from superset.exceptions import SupersetGenericErrorException
+from superset.extensions.storage.codecs import DEFAULT_CODEC, get_codec
+from superset.extensions.storage.filters import ExtensionStorageFilter
+from superset.extensions.storage.persistent_model import ExtensionStorage
+from superset.utils.encrypt import (
+ DEFAULT_ENCRYPTION_ENGINE_NAME,
+ EncryptedType,
+ resolve_encryption_engine,
+)
+
+logger = logging.getLogger(__name__)
+
+
+class ExtensionStorageQuotaExceeded(SupersetGenericErrorException):
+ """Raised when a write would exceed an extension's persistent storage quota."""
+
+ status = 413
+
+ def __init__(self, extension_id: str, quota: int) -> None:
+ super().__init__(
+ message=_(
+ "Extension '%(extension_id)s' has exceeded its persistent "
+ "storage quota of %(quota)d bytes.",
+ extension_id=extension_id,
+ quota=quota,
+ ),
+ )
+
+
+class ExtensionStorageValueTooLarge(SupersetGenericErrorException):
+ """Raised when a `set` call's value exceeds MAX_VALUE_SIZE."""
+
+ status = 400
+
+ def __init__(self, max_size: int) -> None:
+ super().__init__(
+ message=_(
+ "Value exceeds the maximum allowed size of %(max_size)d bytes.",
+ max_size=max_size,
+ ),
+ )
+
+
+class ExtensionStorageListPayloadTooLarge(SupersetGenericErrorException):
+ """Raised when a `list` page's combined value size exceeds
+ MAX_LIST_PAYLOAD_SIZE."""
+
+ status = 400
+
+ def __init__(self, requested_size: int, max_size: int) -> None:
+ super().__init__(
+ message=_(
+ "Requested page would return %(requested_size)d bytes, "
+ "exceeding the maximum allowed list payload size of "
+ "%(max_size)d bytes. Reduce page_size and try again.",
+ requested_size=requested_size,
+ max_size=max_size,
+ ),
+ )
+
+
+def _derive_key(secret: str, user_fk: int) -> str:
+ """Derive a per-user encryption key from the master secret and user ID.
+
+ Uses HMAC-SHA256 so the derived key is cryptographically bound to both.
+ Ciphertext encrypted for one user cannot be decrypted as another.
+ """
+ return hmac.new(
+ secret.encode(),
+ str(user_fk).encode(),
+ hashlib.sha256,
+ ).hexdigest()
+
+
+def _enc_type(user_fk: int | None) -> EncryptedType:
+ """Return an EncryptedType for the given scope.
+
+ User-scoped rows use a key derived from SECRET_KEY and the user ID, so
+ ciphertext written for one user cannot be decrypted as another.
+ Shared rows (user_fk=None) use SECRET_KEY directly.
+ """
+ secret = current_app.config["SECRET_KEY"]
+ key = _derive_key(secret, user_fk) if user_fk is not None else secret
+ engine_name = current_app.config.get(
+ "SQLALCHEMY_ENCRYPTED_FIELD_ENGINE", DEFAULT_ENCRYPTION_ENGINE_NAME
+ )
+ return EncryptedType(
+ LargeBinary,
+ key=key,
+ engine=resolve_encryption_engine(engine_name),
+ )
+
+
+def _get_quota() -> int | None:
+ """Return the configured per-extension persistent storage quota in bytes.
+
+ Returns None when no quota is configured (unlimited).
+ """
+ config = current_app.config["EXTENSIONS_PERSISTENT_STORAGE"]
+ return config.get("QUOTA_PER_EXTENSION")
+
+
+def _get_max_list_payload_size() -> int | None:
+ """Return the configured max combined value size for a `list()` page.
+
+ Returns None when no limit is configured (unlimited).
+ """
+ config = current_app.config["EXTENSIONS_PERSISTENT_STORAGE"]
+ return config.get("MAX_LIST_PAYLOAD_SIZE")
+
+
+def _validate_value_size(encoded: bytes) -> None:
+ """Validate encoded value bytes against MAX_VALUE_SIZE. Raises if over.
+
+ Enforced independently of `QUOTA_PER_EXTENSION`: the quota bounds an
+ extension's total storage, while this bounds a single value. A
+ `list_entries()` page's combined size is bounded separately by
+ MAX_LIST_PAYLOAD_SIZE (see `ExtensionStorageDAO.list_entries`).
+ """
+ config = current_app.config["EXTENSIONS_PERSISTENT_STORAGE"]
+ max_size = config.get("MAX_VALUE_SIZE")
+ if max_size is None:
+ return
+ if len(encoded) > max_size:
+ raise ExtensionStorageValueTooLarge(max_size)
+
+
+def _check_quota(extension_id: str, new_size: int, existing_size: int = 0) -> None:
+ """Raise if writing `new_size` bytes would push the extension over quota.
+
+ `existing_size` is the size of the row being replaced (0 for inserts),
+ subtracted from current usage so overwriting a key doesn't double-count
+ it against the quota it already occupies.
+
+ The usage query sums the `value_size` column via the covering
+ `ix_ext_storage_extension_id` index (on `extension_id, value_size`)
+ instead of `LENGTH(value)`, so it can run as an index-only scan rather
+ than reading every one of the extension's stored (and potentially
+ TOASTed) blobs. It is still O(rows) for the extension, so write latency
+ grows with row count β acceptable at the row counts a single extension
+ is expected to accumulate, but not a fully constant-time check.
+ """
+ quota = _get_quota()
+ if quota is None:
+ return
+ current_usage = (
+ db.session.query(func.coalesce(func.sum(ExtensionStorage.value_size), 0))
+ .filter(ExtensionStorage.extension_id == extension_id)
+ .scalar()
+ )
+ if current_usage - existing_size + new_size > quota:
+ raise ExtensionStorageQuotaExceeded(extension_id, quota)
+
+
+def _decrypt_if_needed(
+ entry: ExtensionStorage, extension_id: str, key: str
+) -> bytes | None:
+ """Return `entry.value`, decrypted if `entry.is_encrypted`.
+
+ Returns None (and logs) if decryption fails, e.g. after key rotation.
+ """
+ if not entry.is_encrypted:
+ return entry.value
+ try:
+ return _enc_type(entry.user_fk).process_result_value(
+ entry.value, db.engine.dialect
+ )
+ except Exception: # noqa: BLE001
+ logger.error(
+ "Failed to decrypt extension storage value for "
+ "extension_id=%s key=%s β possible key rotation issue",
+ extension_id,
+ key,
+ )
+ return None
+
+
+def _scope_filter(
+ extension_id: str,
+ key: str,
+ user_fk: int | None = None,
+ resource_type: str | None = None,
+ resource_uuid: str | None = None,
+) -> list[object]:
+ """Build the SQLAlchemy filter list for a scoped lookup."""
+ filters: list[object] = [
+ ExtensionStorage.extension_id == extension_id,
+ ExtensionStorage.key == key,
+ ]
+ if user_fk is None:
+ filters.append(ExtensionStorage.user_fk.is_(None))
+ else:
+ filters.append(ExtensionStorage.user_fk == user_fk)
+ if resource_type is None:
+ filters.append(ExtensionStorage.resource_type.is_(None))
+ else:
+ filters.append(ExtensionStorage.resource_type == resource_type)
+ if resource_uuid is None:
+ filters.append(ExtensionStorage.resource_uuid.is_(None))
+ else:
+ filters.append(ExtensionStorage.resource_uuid == resource_uuid)
+ return filters
+
+
+def _list_scope_filter(
+ extension_id: str,
+ user_fk: int | None = None,
+ resource_type: str | None = None,
+ resource_uuid: str | None = None,
+) -> list[object]:
+ """Build the SQLAlchemy filter list for a `list()` call.
+
+ Same scope semantics as `_scope_filter` (an explicit None means "match
+ the global/unset scope"), minus the `key` filter.
+ """
+ filters: list[object] = [ExtensionStorage.extension_id == extension_id]
+ if user_fk is None:
+ filters.append(ExtensionStorage.user_fk.is_(None))
+ else:
+ filters.append(ExtensionStorage.user_fk == user_fk)
+ if resource_type is None:
+ filters.append(ExtensionStorage.resource_type.is_(None))
+ else:
+ filters.append(ExtensionStorage.resource_type == resource_type)
+ if resource_uuid is None:
+ filters.append(ExtensionStorage.resource_uuid.is_(None))
+ else:
+ filters.append(ExtensionStorage.resource_uuid == resource_uuid)
+ return filters
+
+
+@dataclass
+class ExtensionStorageListEntry:
+ """A single row returned by `ExtensionStorageDAO.list_entries`.
+
+ `value` is the raw (decrypted) payload, mirroring `get_value()` β it is
+ not decoded with `codec` here, since REST callers must first check
+ `codec` against `SAFE_CODECS` before deciding whether decoding it is
+ safe to expose over the API, while ambient backend callers (no such
+ restriction) decode unconditionally, same as `get_decoded_value()`.
+ """
+
+ key: str
+ value: bytes | None
+ codec: str
+ is_encrypted: bool
+
+
+class ExtensionStorageDAO(BaseDAO[ExtensionStorage]):
+ """Persistent key-value store for extensions (Tier 3).
+
+ Every query and mutation is scoped to the extension currently executing
+ (resolved from ambient extension context by `ExtensionStorageFilter`),
+ so an extension can never read or modify another extension's storage
+ through this DAO.
+
+ Two access shapes are supported:
+
+ * `get`/`set`/`delete_by_key`/`list_entries` β single-key or
+ paginated-listing operations with explicit scope discriminators
+ (`user_fk`, `resource_type`, `resource_uuid`), used by the ambient
+ `persistent_state`/`ephemeral_state` accessors and the REST API (both
+ of which establish the ambient extension context themselves before
+ calling in).
+ * `find_all`/`find_one_or_none`/`filter_by`/`query`/`create`/`update`/
+ `delete` (inherited from `BaseDAO`) β general-purpose DAO access for
+ backend extension code that needs to enumerate or bulk-manage its own
+ rows, exposed to extensions via `superset_core.extensions.storage.dao`.
+
+ Scope is one of:
+
+ * Global scope β user_fk=None, resource_type=None
+ * User scope β user_fk=<id>, resource_type=None
+ * Resource scope β resource_type + resource_uuid set
+ """
+
+ base_filter = ExtensionStorageFilter
+
+ # ββ Read βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
+
+ @staticmethod
+ def get(
+ extension_id: str,
+ key: str,
+ user_fk: int | None = None,
+ resource_type: str | None = None,
+ resource_uuid: str | None = None,
+ ) -> ExtensionStorage | None:
+ """Return the raw storage entry. The value field may be encrypted."""
+ entry = (
+ db.session.query(ExtensionStorage)
+ .filter(
+ and_(
+ *_scope_filter(
+ extension_id, key, user_fk, resource_type, resource_uuid
+ )
+ )
+ )
+ .first()
+ )
+ return entry
+
+ @staticmethod
+ def get_value(
+ extension_id: str,
+ key: str,
+ user_fk: int | None = None,
+ resource_type: str | None = None,
+ resource_uuid: str | None = None,
+ ) -> bytes | None:
+ """Return the raw (decrypted) value bytes, or None if not found."""
+ entry = ExtensionStorageDAO.get(
+ extension_id, key, user_fk, resource_type, resource_uuid
+ )
+ if entry is None:
+ return None
+ return _decrypt_if_needed(entry, extension_id, key)
+
+ @staticmethod
+ def get_decoded_value(
+ extension_id: str,
+ key: str,
+ user_fk: int | None = None,
+ resource_type: str | None = None,
+ resource_uuid: str | None = None,
+ ) -> Any:
+ """Return the value decoded with the codec it was written with.
+
+ :returns: The decoded value, or None if not found.
+ """
+ entry = ExtensionStorageDAO.get(
+ extension_id, key, user_fk, resource_type, resource_uuid
+ )
+ if entry is None:
+ return None
+ raw = _decrypt_if_needed(entry, extension_id, key)
+ if raw is None:
+ return None
+ return get_codec(entry.codec).decode(raw)
+
+ @staticmethod
+ def list_entries(
+ extension_id: str,
+ user_fk: int | None = None,
+ resource_type: str | None = None,
+ resource_uuid: str | None = None,
+ page: int = 0,
+ page_size: int = 10,
+ ) -> tuple[list[ExtensionStorageListEntry], int]:
+ """List entries in the given scope, most recently changed first.
+
+ Named `list_entries` rather than `list` (unlike the single-key
+ `get`/`set`/`delete_by_key` shape) because `BaseDAO` already
+ defines a generic `list()` with a different, incompatible
+ signature (search/sort/column-selection over arbitrary models);
+ overriding it here would break that contract for callers relying
+ on polymorphic DAO access.
+
+ Callers may request any `page_size` β there is no row-count
+ ceiling β but the combined `value_size` of the requested page is
+ checked against MAX_LIST_PAYLOAD_SIZE *before* the (potentially
+ large) `value` column is fetched for any row, so an oversized
+ request fails fast rather than reading data it will then reject.
+
+ :returns: (entries, total_count). `entries[i].value` is the raw
+ (decrypted) payload β see `ExtensionStorageListEntry`.
+ :raises ExtensionStorageListPayloadTooLarge: if the requested page's
+ combined value size exceeds MAX_LIST_PAYLOAD_SIZE.
+ """
+ filters = _list_scope_filter(
+ extension_id, user_fk, resource_type, resource_uuid
+ )
+ base_query = db.session.query(ExtensionStorage).filter(and_(*filters))
+ total_count = base_query.count()
+
+ page = max(page, 0)
+ page_size = max(page_size, 1)
+ page_query = (
+ base_query.order_by(desc(ExtensionStorage.changed_on))
+ .offset(page * page_size)
+ .limit(page_size)
+ )
+
+ if (max_payload_size := _get_max_list_payload_size()) is not None:
+ # Sum value_size for exactly this page's rows, without touching
+ # the `value` column, by reusing the same ordered/offset/limited
+ # query as a subquery.
+ subquery = page_query.with_entities(ExtensionStorage.value_size).subquery()
+ requested_size = db.session.query(
+ func.coalesce(func.sum(subquery.c.value_size), 0)
+ ).scalar()
+ if requested_size > max_payload_size:
+ raise ExtensionStorageListPayloadTooLarge(
+ requested_size, max_payload_size
+ )
+
+ entries = [
+ ExtensionStorageListEntry(
+ key=row.key,
+ value=_decrypt_if_needed(row, extension_id, row.key),
+ codec=row.codec,
+ is_encrypted=row.is_encrypted,
+ )
+ for row in page_query.all()
+ ]
+ return entries, total_count
+
+ # ββ Write (upsert) ββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
+
+ @staticmethod
+ def set(
+ extension_id: str,
+ key: str,
+ value: bytes,
+ codec: str = DEFAULT_CODEC,
+ user_fk: int | None = None,
+ resource_type: str | None = None,
+ resource_uuid: str | None = None,
+ encrypt: bool = False,
+ ) -> ExtensionStorage:
+ """Upsert a storage entry. Encrypts value when encrypt=True.
+
+ :raises ExtensionStorageValueTooLarge: if `value` exceeds
+ MAX_VALUE_SIZE.
+ :raises ExtensionStorageQuotaExceeded: if writing `value` would push
+ the extension over its total storage quota.
+ """
+ _validate_value_size(value)
+ stored_value = (
+ _enc_type(user_fk).process_bind_param(value, db.engine.dialect)
+ if encrypt
+ else value
+ )
+
+ entry = (
+ db.session.query(ExtensionStorage)
+ .filter(
+ and_(
+ *_scope_filter(
+ extension_id, key, user_fk, resource_type, resource_uuid
+ )
+ )
+ )
+ .first()
+ )
+ new_size = len(stored_value)
+ existing_size = entry.value_size if entry is not None else 0
+ _check_quota(extension_id, new_size, existing_size)
+ if entry is not None:
+ entry.value = stored_value
+ entry.value_size = new_size
+ entry.codec = codec
+ entry.is_encrypted = encrypt
+ else:
+ entry = ExtensionStorage(
+ extension_id=extension_id,
+ key=key,
+ value=stored_value,
+ value_size=new_size,
+ codec=codec,
+ user_fk=user_fk,
+ resource_type=resource_type,
+ resource_uuid=resource_uuid,
+ is_encrypted=encrypt,
+ )
+ db.session.add(entry)
+ db.session.flush()
+ return entry
Review Comment:
**π Architect Review β HIGH**
Persistent storage key length is only enforced in the frontend SDK, while the DAO/API paths do not validate key length before writing. Backend extension code or direct API callers can use keys longer than the database column limit (String(255)), leading to database-level errors/truncation instead of a controlled storage error response.
**Suggestion:** Add key-length validation at the ExtensionStorageDAO boundary (and reuse it from the REST API and backend accessors) so all callers receive a consistent, explicit error when keys exceed 255 characters, rather than hitting database integrity or truncation behavior.
[](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=9e725d0f3d5a4ddd909c4520a325acd3&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=9e725d0f3d5a4ddd909c4520a325acd3&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset)
*(Use Cmd/Ctrl + Click for best experience)*
<details>
<summary><b>Prompt for AI Agent π€ </b></summary>
```mdx
This is an **Architect / Logical Review** comment left during a code review. These reviews are first-class, important findings β not optional suggestions. Do NOT dismiss this as a 'big architectural change' just because the title says architect review; most of these can be resolved with a small, localized fix once the intent is understood.
**Path:** superset/extensions/storage/persistent_dao.py
**Line:** 442:499
**Comment:**
*HIGH: Persistent storage key length is only enforced in the frontend SDK, while the DAO/API paths do not validate key length before writing. Backend extension code or direct API callers can use keys longer than the database column limit (String(255)), leading to database-level errors/truncation instead of a controlled storage error response.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
If a suggested approach is provided above, use it as the authoritative instruction. If no explicit code suggestion is given, you MUST still draft and apply your own minimal, localized fix β do not punt back with 'no suggestion provided, review manually'. Keep the change as small as possible: add a guard clause, gate on a loading state, reorder an await, wrap in a conditional, etc. Do not refactor surrounding code or expand scope beyond the finding.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(extensions): add a Storage API for extensions [superset]
Posted by "codeant-ai-for-open-source[bot] (via GitHub)" <gi...@apache.org>.
codeant-ai-for-open-source[bot] commented on PR #39171:
URL: https://github.com/apache/superset/pull/39171#issuecomment-4938192083
---
### Thanks for using CodeAnt! π
We're free for open-source projects. if you're enjoying it, help us grow by sharing.
[Share on X](https://twitter.com/intent/tweet?text=Just%20tried%20%40CodeAntAI%20for%20automated%20code%20review%20and%20I%27m%20impressed%21%20Free%20for%20open%20source%20with%20a%20free%20trial%20for%20private%20repos.%20Worth%20checking%20out%3A&url=https%3A//codeant.ai) Β·
[Reddit](https://www.reddit.com/submit?title=Check%20out%20CodeAnt%20for%20automated%20code%20review&text=Just%20tried%20CodeAnt%20for%20automated%20code%20review%20and%20I%27m%20impressed%21%20Free%20for%20open%20source%20with%20a%20free%20trial%20for%20private%20repos.%20Worth%20checking%20out%3A%20https%3A//codeant.ai) Β·
[LinkedIn](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fcodeant.ai&mini=true&title=Check%20out%20CodeAnt%20for%20automated%20code%20review&summary=Just%20tried%20CodeAnt%20for%20automated%20code%20review%20and%20I%27m%20impressed%21%20Free%20for%20open%20source%20with%20a%20free%20trial%20for%20private%20repos)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(extensions): add a Storage API for extensions [superset]
Posted by "michael-s-molina (via GitHub)" <gi...@apache.org>.
michael-s-molina commented on PR #39171:
URL: https://github.com/apache/superset/pull/39171#issuecomment-4444310594
@matebenyovszky @rusackas Thanks for your contributions! Iβll pick this back up after my PTOβthere are still several updates needed based on our recent extensions meeting. Rest assured, weβll finish the work in time for the next Superset release, since this feature is essential for extensions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(extensions): add a Storage API for extensions [superset]
Posted by "codeant-ai-for-open-source[bot] (via GitHub)" <gi...@apache.org>.
codeant-ai-for-open-source[bot] commented on code in PR #39171:
URL: https://github.com/apache/superset/pull/39171#discussion_r3622757568
##########
superset-core/src/superset_core/extensions/context.py:
##########
@@ -0,0 +1,136 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""
+Extension Context API for superset-core extensions.
+
+Provides access to the current extension's context, including metadata
+and scoped resources like storage. Extensions call `get_context()` to
+access their context during execution.
+
+The context is set by the host (Superset) during extension loading and
+is only available within extension code.
+
+Usage:
+ from superset_core.extensions.context import get_context
+
+ def setup():
+ ctx = get_context()
+
+ # Access extension metadata
+ print(f"Running {ctx.extension.displayName} v{ctx.extension.version}")
+
+ # Access extension-scoped storage
+ ctx.storage.ephemeral.set("lastRun", time.time())
+ data = ctx.storage.ephemeral.get("cachedData")
+"""
+
+from __future__ import annotations
+
+from typing import Any, Protocol, TYPE_CHECKING
+
+if TYPE_CHECKING:
+ from superset_core.extensions.types import BaseExtension
+
+
+class StorageAccessor(Protocol):
+ """Protocol for storage access with user-scoped and shared modes."""
+
+ def get(self, key: str) -> Any:
+ """Get a value from storage."""
+ ...
+
+ def set(self, key: str, value: Any, ttl: int = 3600) -> None:
+ """Set a value in storage with optional TTL."""
+ ...
+
+ def remove(self, key: str) -> None:
+ """Remove a value from storage."""
+ ...
+
+ @property
+ def shared(self) -> "StorageAccessor":
+ """Shared (cross-user) storage accessor."""
+ ...
+
+
+class ExtensionStorage(Protocol):
+ """Extension-scoped storage accessor for all available tiers."""
+
+ @property
+ def ephemeral(self) -> StorageAccessor:
+ """Server-side cache (Redis/Memcached) with TTL."""
+ ...
+
+ @property
+ def persistent(self) -> StorageAccessor:
+ """Database-backed persistent storage."""
+ ...
+
+
+class ExtensionContext(Protocol):
+ """
+ Context object providing extension-specific resources.
+
+ This context is only available during extension execution.
+ Calling `get_context()` outside of an extension will raise an error.
+ """
+
+ @property
+ def extension(self) -> "BaseExtension":
+ """Metadata about the current extension."""
+ ...
+
+ @property
+ def storage(self) -> ExtensionStorage:
+ """Extension-scoped storage across all available tiers."""
+ ...
+
+
+def get_context() -> ExtensionContext:
+ """
+ Get the current extension's context.
+
+ This function returns the context for the currently executing extension,
+ providing access to extension metadata and scoped resources like storage.
+
+ Host implementations will replace this function during initialization
+ with a concrete implementation providing actual functionality.
+
+ :returns: The current extension's context.
+ :raises RuntimeError: If called outside of an extension context.
+
+ Example:
+ from superset_core.extensions.context import get_context
+
+ ctx = get_context()
+
+ # Access extension metadata
+ print(f"Extension: {ctx.extension.id}")
+ print(f"Version: {ctx.extension.version}")
+
+ # Access extension-scoped storage
+ ctx.storage.ephemeral.set("tempData", data, ttl=3600)
+ value = ctx.storage.ephemeral.get("tempData")
+
+ # Access shared (cross-user) storage
+ ctx.storage.ephemeral.shared.set("globalCounter", count)
+ """
+ raise NotImplementedError(
+ "get_context() must be called within an extension context. "
+ "This function is replaced by the host during extension loading."
+ )
Review Comment:
β
**Customized review instruction saved!**
**Instruction:**
> Do not flag docstring/exception mismatches in stub functions whose documented behavior describes the host-replaced implementation contract, even if the stub itself raises a different placeholder exception.
**Applied to:**
- `superset-core/src/superset_core/extensions/**`
---
π‘ *To manage or update this instruction, visit: [CodeAnt AI Settings](https://app.codeant.ai/org/settings/learnings)*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(extensions): add a Storage API for extensions [superset]
Posted by "michael-s-molina (via GitHub)" <gi...@apache.org>.
michael-s-molina commented on code in PR #39171:
URL: https://github.com/apache/superset/pull/39171#discussion_r3624936583
##########
superset-core/src/superset_core/extensions/context.py:
##########
@@ -0,0 +1,117 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""
+Extension Context API for superset-core extensions.
+
+Provides access to the current extension's context, including metadata
+and scoped resources like storage. Extensions call `get_context()` to
+access their context during execution.
+
+The context is set by the host (Superset) during extension loading and
+is only available within extension code.
+
+Usage:
+ from superset_core.extensions.context import get_context
+
+ def setup():
+ ctx = get_context()
+
+ # Access extension metadata
+ print(f"Running {ctx.extension.displayName} v{ctx.extension.version}")
+
+ # Access extension-scoped storage
+ ctx.storage.ephemeral.set("lastRun", time.time(), EphemeralSetOptions(ttl=3600))
+ data = ctx.storage.ephemeral.get("cachedData")
+"""
+
+from __future__ import annotations
+
+from typing import Protocol, TYPE_CHECKING
+
+if TYPE_CHECKING:
+ from superset_core.extensions.storage.ephemeral import EphemeralStateAccessor
+ from superset_core.extensions.storage.persistent import PersistentStateAccessor
+ from superset_core.extensions.types import BaseExtension
+
+
+class ExtensionStorage(Protocol):
+ """Extension-scoped storage accessor for all available tiers."""
+
+ @property
+ def ephemeral(self) -> "EphemeralStateAccessor":
+ """Server-side cache (Redis/Memcached) with TTL."""
+ ...
+
+ @property
+ def persistent(self) -> "PersistentStateAccessor":
+ """Database-backed persistent storage."""
+ ...
+
+
+class ExtensionContext(Protocol):
+ """
+ Context object providing extension-specific resources.
+
+ This context is only available during extension execution.
+ Calling `get_context()` outside of an extension will raise an error.
+ """
+
+ @property
+ def extension(self) -> "BaseExtension":
+ """Metadata about the current extension."""
+ ...
+
+ @property
+ def storage(self) -> ExtensionStorage:
+ """Extension-scoped storage across all available tiers."""
+ ...
+
+
+def get_context() -> ExtensionContext:
+ """
+ Get the current extension's context.
+
+ This function returns the context for the currently executing extension,
+ providing access to extension metadata and scoped resources like storage.
+
+ Host implementations will replace this function during initialization
+ with a concrete implementation providing actual functionality.
+
+ :returns: The current extension's context.
+ :raises RuntimeError: If called outside of an extension context.
+
+ Example:
+ from superset_core.extensions.context import get_context
+
+ ctx = get_context()
+
+ # Access extension metadata
+ print(f"Extension: {ctx.extension.id}")
+ print(f"Version: {ctx.extension.version}")
+
+ # Access extension-scoped storage
+ ctx.storage.ephemeral.set("tempData", data, ttl=3600)
Review Comment:
Fixed β updated `get_context()`'s own docstring example to use `EphemeralSetOptions(ttl=3600)` for both the private and shared `set()` calls, matching the real Tier 2 signature (the module-level docstring above it was already correct; this was a second, separate copy of the same example).
##########
superset-core/src/superset_core/extensions/storage/ephemeral.py:
##########
@@ -0,0 +1,144 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""
+Ephemeral State API for superset-core extensions (Tier 2 Storage).
+
+Provides short-lived KV storage backed by the configured server-side cache
+backend (Redis, Memcached, or filesystem). Automatically expires based on TTL.
+Not guaranteed to survive server restarts.
+
+Host implementations will replace the EphemeralState class during initialization
+with a concrete implementation providing actual functionality.
+
+Cache keys are namespaced automatically:
+- User-scoped (default): superset-ext:{extension_id}:user:{user_id}:{key}
+- Shared (global): superset-ext:{extension_id}:shared:{key}
+
+Usage (via extension context - preferred):
+ from superset_core.extensions.context import get_context
+
+ ctx = get_context()
+
+ # User-scoped state (default - private to current user)
+ ctx.storage.ephemeral.get('preference')
+ ctx.storage.ephemeral.set('preference', 'compact', EphemeralSetOptions(ttl=3600))
+ ctx.storage.ephemeral.remove('preference')
+
+ # Shared state (explicit opt-in - visible to all users)
+ ctx.storage.ephemeral.shared.get('job_progress')
+ ctx.storage.ephemeral.shared.set(
+ 'job_progress', {'pct': 42}, EphemeralSetOptions(ttl=3600)
+ )
+ ctx.storage.ephemeral.shared.remove('job_progress')
+"""
+
+from dataclasses import dataclass
+from typing import Any, ClassVar, Protocol
+
+
+@dataclass(frozen=True)
+class EphemeralSetOptions:
+ """
+ Options for an ephemeral state `set` call.
+
+ `ttl` is required (no default): every write must have an explicit
+ expiration, matching the frontend `EphemeralSetOptions` and the REST API.
+ """
+
+ ttl: int
+ #: Name of the codec used to encode `value`, e.g. "json" (default).
+ codec: str = "json"
+
+
+class EphemeralStateAccessor(Protocol):
+ """Protocol for scoped ephemeral state access."""
+
+ def get(self, key: str) -> Any:
+ """Get a value from ephemeral state."""
+ ...
+
+ def set(self, key: str, value: Any, options: EphemeralSetOptions) -> None:
+ """Set a value in ephemeral state with TTL."""
+ ...
+
+ def remove(self, key: str) -> None:
+ """Remove a value from ephemeral state."""
+ ...
+
+
+class EphemeralState:
+ """
+ Tier 2 ephemeral state storage for extensions.
+
+ Backed by the configured server-side cache (Redis, Memcached, or filesystem).
+ Data expires based on TTL and is not guaranteed to survive server restarts.
+
+ Host implementations will replace this class during initialization
+ with a concrete implementation providing actual functionality.
+
+ All operations are user-scoped by default (private to the current user).
+ Use `.shared` to access state that is visible to all users.
+ """
+
+ @staticmethod
+ def get(key: str) -> Any:
+ """
+ Get a value from user-scoped ephemeral state.
+
+ Data is automatically scoped to the current authenticated user.
+ Other users cannot see or modify this data.
+
+ :param key: The key to retrieve.
+ :returns: The stored value, or None if not found or expired.
+ """
+ raise NotImplementedError("Class will be replaced during initialization")
+
+ @staticmethod
+ def set(key: str, value: Any, options: EphemeralSetOptions) -> None:
+ """
+ Set a value in user-scoped ephemeral state with TTL.
+
+ Data is automatically scoped to the current authenticated user.
+ Other users cannot see or modify this data.
+
+ :param key: The key to store.
+ :param value: The value to store, encoded with `options.codec`
+ (default "json"). The encoded value must not exceed
+ MAX_VALUE_SIZE from config.
+ :param options: `EphemeralSetOptions`, e.g. `ttl=3600`. Required β
+ `ttl` must not exceed MAX_TTL from config. `codec="pickle"`
+ stores a value that isn't JSON-serializable.
+ """
+ raise NotImplementedError("Class will be replaced during initialization")
+
+ @staticmethod
+ def remove(key: str) -> None:
+ """
+ Remove a value from user-scoped ephemeral state.
+
+ Data is automatically scoped to the current authenticated user.
+
+ :param key: The key to remove.
+ """
+ raise NotImplementedError("Class will be replaced during initialization")
+
+ #: Shared (global) ephemeral state accessor.
+ #: Data stored via this accessor is visible to all users of the extension.
+ #: WARNING: Do not store user-specific or sensitive data here.
+ #: Host implementations will set this during initialization.
+ shared: ClassVar[EphemeralStateAccessor]
Review Comment:
Fixed β `EphemeralState.shared` (and the identical pattern on `PersistentState.shared` in the sibling module) now defaults to a placeholder accessor whose methods raise the same `NotImplementedError("Class will be replaced during initialization")` as every other stub method, instead of a bare `AttributeError` from the unassigned annotation.
##########
superset/extensions/storage/api.py:
##########
@@ -0,0 +1,740 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""
+REST API for extension storage.
+
+Provides HTTP endpoints for frontend extensions to access server-side
+ephemeral storage without direct backend code.
+
+All operations are user-scoped by default. Use `?shared=true` query param
+to access shared state visible to all users.
+"""
+
+from __future__ import annotations
+
+import base64
+from typing import Any
+
+from flask import g, request
+from flask.wrappers import Response
+from flask_appbuilder.api import BaseApi, expose, protect, safe
+
+from superset.extensions.storage.codecs import DEFAULT_CODEC, get_codec, SAFE_CODECS
+from superset.extensions.storage.ephemeral_dao import (
+ ExtensionEphemeralDAO,
+ ExtensionEphemeralTTLInvalid,
+ ExtensionEphemeralValueTooLarge,
+)
+from superset.extensions.storage.persistent_dao import (
+ ExtensionStorageDAO,
+ ExtensionStorageKeyTooLong,
+ ExtensionStorageListPayloadTooLarge,
+ ExtensionStorageQuotaExceeded,
+ ExtensionStorageValueTooLarge,
+)
+from superset.extensions.storage.utils import get_extension_or_404, parse_ttl
+from superset.key_value.exceptions import KeyValueCodecEncodeException
+from superset.utils.decorators import transaction
+
+
+def _decoded_result_for_wire(decoded: Any) -> tuple[Any, bool]:
+ """Convert a codec-decoded value into its JSON wire representation.
+
+ JSON has no byte type, so a raw `bytes` value is base64-encoded to a
+ string for the response and flagged as such; every other value is
+ already JSON-representable as-is. Checked on the decoded value's
+ actual type, not the codec's name, so this keeps working for a codec
+ this module didn't define.
+
+ :returns: (wire_value, is_binary)
+ """
+ if isinstance(decoded, bytes):
+ return base64.b64encode(decoded).decode("ascii"), True
+ return decoded, False
+
+
+def _wire_value_for_request(value: Any, is_binary: bool) -> Any:
+ """Convert a request body's JSON `value` into a codec's input type.
+
+ `is_binary` is an explicit flag from the caller: JSON has no byte
+ type, so there is no way to tell, from the JSON `value` alone,
+ whether it is a base64 string that must be decoded to bytes before
+ being handed to the codec's `encode`, or a literal value to pass
+ through as-is. Only the caller knows which it sent.
+
+ :raises ValueError: if `is_binary` is set and `value` is not a valid
+ base64 string.
+ """
+ if is_binary:
+ return base64.b64decode(value, validate=True)
+ return value
+
+
+class ExtensionStorageRestApi(BaseApi):
+ """REST API for extension ephemeral (Tier 2) and persistent (Tier 3) storage."""
+
+ allow_browser_login = True
+ route_base = "/api/v1/extensions"
+
+ def response(self, status_code: int, **kwargs: Any) -> Response:
+ """Helper method to create JSON responses."""
+ from flask import jsonify
+
+ return jsonify(kwargs), status_code
+
+ def response_404(self, message: str = "Not found") -> Response:
+ """Helper method to create 404 responses."""
+ from flask import jsonify
+
+ return jsonify({"message": message}), 404
+
+ def response_400(self, message: str) -> Response:
+ """Helper method to create 400 responses."""
+ from flask import jsonify
+
+ return jsonify({"message": message}), 400
+
+ @protect()
+ @safe
+ @expose("/<publisher>/<name>/storage/ephemeral/<key>", methods=("GET",))
+ def get_ephemeral(
+ self, publisher: str, name: str, key: str, **kwargs: Any
+ ) -> Response:
+ """Get a value from ephemeral state.
+ ---
+ get:
+ summary: Get a value from ephemeral state
+ parameters:
+ - in: path
+ name: publisher
+ schema:
+ type: string
+ required: true
+ description: Extension publisher
+ - in: path
+ name: name
+ schema:
+ type: string
+ required: true
+ description: Extension name
+ - in: path
+ name: key
+ schema:
+ type: string
+ required: true
+ description: Storage key
+ - in: query
+ name: shared
+ schema:
+ type: boolean
+ required: false
+ description: If true, read from shared state visible to all users
+ responses:
+ 200:
+ description: Value retrieved successfully
+ content:
+ application/json:
+ schema:
+ type: object
+ properties:
+ result:
+ description: The stored value. When 'isBinary' is
+ true, this is a base64 string that must be
+ decoded to get the actual value.
+ codec:
+ type: string
+ description: Name of the codec 'result' was encoded
+ with, e.g. "json" (default) or "binary"
+ isBinary:
+ type: boolean
+ description: Whether the stored value is binary
+ data
+ 400:
+ description: Value was stored with a codec unavailable over the API
+ 404:
+ description: Extension not found
+ """
+ extension_id = f"{publisher}.{name}"
+ extension = get_extension_or_404(extension_id)
+ if not extension:
+ return self.response_404("Extension not found")
+
+ shared = request.args.get("shared", "false").lower() == "true"
+ raw = ExtensionEphemeralDAO.get_raw(extension_id, key, shared=shared)
+ if raw is None:
+ return self.response(200, result=None)
+ value, codec = raw
+ if codec not in SAFE_CODECS:
+ return self.response_400(
+ f"Value was stored with codec '{codec}', which cannot be "
+ "read over the REST API."
+ )
+
+ result, is_binary = _decoded_result_for_wire(get_codec(codec).decode(value))
+ return self.response(200, result=result, codec=codec, isBinary=is_binary)
+
+ @protect()
+ @safe
+ @expose("/<publisher>/<name>/storage/ephemeral/<key>", methods=("PUT",))
+ def set_ephemeral(
+ self, publisher: str, name: str, key: str, **kwargs: Any
+ ) -> Response:
+ """Set a value in ephemeral state.
+ ---
+ put:
+ summary: Set a value in ephemeral state
+ parameters:
+ - in: path
+ name: publisher
+ schema:
+ type: string
+ required: true
+ description: Extension publisher
+ - in: path
+ name: name
+ schema:
+ type: string
+ required: true
+ description: Extension name
+ - in: path
+ name: key
+ schema:
+ type: string
+ required: true
+ description: Storage key
+ - in: query
+ name: shared
+ schema:
+ type: boolean
+ required: false
+ description: If true, store as shared state visible to all users
+ requestBody:
+ required: true
+ content:
+ application/json:
+ schema:
+ type: object
+ required:
+ - value
+ - ttl
+ properties:
+ value:
+ description: The value to store (must not exceed
+ MAX_VALUE_SIZE bytes once encoded with 'codec').
+ When 'isBinary' is true, this must be a base64
+ string, decoded before being handed to 'codec'.
+ codec:
+ type: string
+ description: Name of the codec used to encode 'value',
+ e.g. "json" (default). Must be one of the codecs
+ allowed over the REST API.
+ isBinary:
+ type: boolean
+ description: Whether 'value' is binary data
+ ttl:
+ type: integer
+ description: Time-to-live in seconds (must be a positive
+ integer not exceeding MAX_TTL)
+ responses:
+ 200:
+ description: Value stored successfully
+ 400:
+ description: Invalid request body, or codec not allowed over the API
+ 404:
+ description: Extension not found
+ """
+ extension_id = f"{publisher}.{name}"
+ extension = get_extension_or_404(extension_id)
+ if not extension:
+ return self.response_404("Extension not found")
+
+ body = request.get_json(silent=True) or {}
+ if "value" not in body:
+ return self.response_400("Request body must contain 'value' field")
+
+ codec = body.get("codec", DEFAULT_CODEC)
Review Comment:
Fixed β `set_ephemeral` now checks `isinstance(body, dict)` and returns 400 before calling `.get()`/`in` on the parsed JSON, so a non-object payload like `["value"]` can no longer reach `body.get(...)` and crash with an unhandled `AttributeError`.
##########
superset/extensions/storage/api.py:
##########
@@ -0,0 +1,740 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""
+REST API for extension storage.
+
+Provides HTTP endpoints for frontend extensions to access server-side
+ephemeral storage without direct backend code.
+
+All operations are user-scoped by default. Use `?shared=true` query param
+to access shared state visible to all users.
+"""
+
+from __future__ import annotations
+
+import base64
+from typing import Any
+
+from flask import g, request
+from flask.wrappers import Response
+from flask_appbuilder.api import BaseApi, expose, protect, safe
+
+from superset.extensions.storage.codecs import DEFAULT_CODEC, get_codec, SAFE_CODECS
+from superset.extensions.storage.ephemeral_dao import (
+ ExtensionEphemeralDAO,
+ ExtensionEphemeralTTLInvalid,
+ ExtensionEphemeralValueTooLarge,
+)
+from superset.extensions.storage.persistent_dao import (
+ ExtensionStorageDAO,
+ ExtensionStorageKeyTooLong,
+ ExtensionStorageListPayloadTooLarge,
+ ExtensionStorageQuotaExceeded,
+ ExtensionStorageValueTooLarge,
+)
+from superset.extensions.storage.utils import get_extension_or_404, parse_ttl
+from superset.key_value.exceptions import KeyValueCodecEncodeException
+from superset.utils.decorators import transaction
+
+
+def _decoded_result_for_wire(decoded: Any) -> tuple[Any, bool]:
+ """Convert a codec-decoded value into its JSON wire representation.
+
+ JSON has no byte type, so a raw `bytes` value is base64-encoded to a
+ string for the response and flagged as such; every other value is
+ already JSON-representable as-is. Checked on the decoded value's
+ actual type, not the codec's name, so this keeps working for a codec
+ this module didn't define.
+
+ :returns: (wire_value, is_binary)
+ """
+ if isinstance(decoded, bytes):
+ return base64.b64encode(decoded).decode("ascii"), True
+ return decoded, False
+
+
+def _wire_value_for_request(value: Any, is_binary: bool) -> Any:
+ """Convert a request body's JSON `value` into a codec's input type.
+
+ `is_binary` is an explicit flag from the caller: JSON has no byte
+ type, so there is no way to tell, from the JSON `value` alone,
+ whether it is a base64 string that must be decoded to bytes before
+ being handed to the codec's `encode`, or a literal value to pass
+ through as-is. Only the caller knows which it sent.
+
+ :raises ValueError: if `is_binary` is set and `value` is not a valid
+ base64 string.
+ """
+ if is_binary:
+ return base64.b64decode(value, validate=True)
+ return value
+
+
+class ExtensionStorageRestApi(BaseApi):
+ """REST API for extension ephemeral (Tier 2) and persistent (Tier 3) storage."""
+
+ allow_browser_login = True
+ route_base = "/api/v1/extensions"
+
+ def response(self, status_code: int, **kwargs: Any) -> Response:
+ """Helper method to create JSON responses."""
+ from flask import jsonify
+
+ return jsonify(kwargs), status_code
+
+ def response_404(self, message: str = "Not found") -> Response:
+ """Helper method to create 404 responses."""
+ from flask import jsonify
+
+ return jsonify({"message": message}), 404
+
+ def response_400(self, message: str) -> Response:
+ """Helper method to create 400 responses."""
+ from flask import jsonify
+
+ return jsonify({"message": message}), 400
+
+ @protect()
+ @safe
+ @expose("/<publisher>/<name>/storage/ephemeral/<key>", methods=("GET",))
+ def get_ephemeral(
+ self, publisher: str, name: str, key: str, **kwargs: Any
+ ) -> Response:
+ """Get a value from ephemeral state.
+ ---
+ get:
+ summary: Get a value from ephemeral state
+ parameters:
+ - in: path
+ name: publisher
+ schema:
+ type: string
+ required: true
+ description: Extension publisher
+ - in: path
+ name: name
+ schema:
+ type: string
+ required: true
+ description: Extension name
+ - in: path
+ name: key
+ schema:
+ type: string
+ required: true
+ description: Storage key
+ - in: query
+ name: shared
+ schema:
+ type: boolean
+ required: false
+ description: If true, read from shared state visible to all users
+ responses:
+ 200:
+ description: Value retrieved successfully
+ content:
+ application/json:
+ schema:
+ type: object
+ properties:
+ result:
+ description: The stored value. When 'isBinary' is
+ true, this is a base64 string that must be
+ decoded to get the actual value.
+ codec:
+ type: string
+ description: Name of the codec 'result' was encoded
+ with, e.g. "json" (default) or "binary"
+ isBinary:
+ type: boolean
+ description: Whether the stored value is binary
+ data
+ 400:
+ description: Value was stored with a codec unavailable over the API
+ 404:
+ description: Extension not found
+ """
+ extension_id = f"{publisher}.{name}"
+ extension = get_extension_or_404(extension_id)
+ if not extension:
+ return self.response_404("Extension not found")
+
+ shared = request.args.get("shared", "false").lower() == "true"
+ raw = ExtensionEphemeralDAO.get_raw(extension_id, key, shared=shared)
+ if raw is None:
+ return self.response(200, result=None)
+ value, codec = raw
+ if codec not in SAFE_CODECS:
+ return self.response_400(
+ f"Value was stored with codec '{codec}', which cannot be "
+ "read over the REST API."
+ )
+
+ result, is_binary = _decoded_result_for_wire(get_codec(codec).decode(value))
+ return self.response(200, result=result, codec=codec, isBinary=is_binary)
+
+ @protect()
+ @safe
+ @expose("/<publisher>/<name>/storage/ephemeral/<key>", methods=("PUT",))
+ def set_ephemeral(
+ self, publisher: str, name: str, key: str, **kwargs: Any
+ ) -> Response:
+ """Set a value in ephemeral state.
+ ---
+ put:
+ summary: Set a value in ephemeral state
+ parameters:
+ - in: path
+ name: publisher
+ schema:
+ type: string
+ required: true
+ description: Extension publisher
+ - in: path
+ name: name
+ schema:
+ type: string
+ required: true
+ description: Extension name
+ - in: path
+ name: key
+ schema:
+ type: string
+ required: true
+ description: Storage key
+ - in: query
+ name: shared
+ schema:
+ type: boolean
+ required: false
+ description: If true, store as shared state visible to all users
+ requestBody:
+ required: true
+ content:
+ application/json:
+ schema:
+ type: object
+ required:
+ - value
+ - ttl
+ properties:
+ value:
+ description: The value to store (must not exceed
+ MAX_VALUE_SIZE bytes once encoded with 'codec').
+ When 'isBinary' is true, this must be a base64
+ string, decoded before being handed to 'codec'.
+ codec:
+ type: string
+ description: Name of the codec used to encode 'value',
+ e.g. "json" (default). Must be one of the codecs
+ allowed over the REST API.
+ isBinary:
+ type: boolean
+ description: Whether 'value' is binary data
+ ttl:
+ type: integer
+ description: Time-to-live in seconds (must be a positive
+ integer not exceeding MAX_TTL)
+ responses:
+ 200:
+ description: Value stored successfully
+ 400:
+ description: Invalid request body, or codec not allowed over the API
+ 404:
+ description: Extension not found
+ """
+ extension_id = f"{publisher}.{name}"
+ extension = get_extension_or_404(extension_id)
+ if not extension:
+ return self.response_404("Extension not found")
+
+ body = request.get_json(silent=True) or {}
+ if "value" not in body:
+ return self.response_400("Request body must contain 'value' field")
+
+ codec = body.get("codec", DEFAULT_CODEC)
+ if not isinstance(codec, str) or codec not in SAFE_CODECS:
+ return self.response_400(
+ f"Codec '{codec}' is not allowed over the REST API."
+ )
+
+ is_binary = bool(body.get("isBinary", False))
+ try:
+ value = _wire_value_for_request(body["value"], is_binary)
+ except (ValueError, TypeError):
+ return self.response_400(
+ "Value must be a valid base64 string when 'isBinary' is true."
+ )
+ ttl, error = parse_ttl(body)
+ if error:
+ return self.response_400(error)
+
+ shared = request.args.get("shared", "false").lower() == "true"
+ try:
+ ExtensionEphemeralDAO.set(
+ extension_id, key, value, ttl, codec=codec, shared=shared
+ )
+ except (ExtensionEphemeralTTLInvalid, ExtensionEphemeralValueTooLarge) as ex:
+ return self.response_400(ex.message)
+ except (KeyValueCodecEncodeException, TypeError, ValueError):
+ return self.response_400(
+ f"Value could not be encoded with codec '{codec}'."
+ )
+
+ return self.response(200, message="Value stored successfully")
+
+ @protect()
+ @safe
+ @expose("/<publisher>/<name>/storage/ephemeral/<key>", methods=("DELETE",))
+ def delete_ephemeral(
+ self, publisher: str, name: str, key: str, **kwargs: Any
+ ) -> Response:
+ """Delete a value from ephemeral state.
+ ---
+ delete:
+ summary: Delete a value from ephemeral state
+ parameters:
+ - in: path
+ name: publisher
+ schema:
+ type: string
+ required: true
+ description: Extension publisher
+ - in: path
+ name: name
+ schema:
+ type: string
+ required: true
+ description: Extension name
+ - in: path
+ name: key
+ schema:
+ type: string
+ required: true
+ description: Storage key
+ - in: query
+ name: shared
+ schema:
+ type: boolean
+ required: false
+ description: If true, delete from shared state
+ responses:
+ 200:
+ description: Value deleted successfully
+ 404:
+ description: Extension not found
+ """
+ extension_id = f"{publisher}.{name}"
+ extension = get_extension_or_404(extension_id)
+ if not extension:
+ return self.response_404("Extension not found")
+
+ shared = request.args.get("shared", "false").lower() == "true"
+ ExtensionEphemeralDAO.delete(extension_id, key, shared=shared)
+
+ return self.response(200, message="Value deleted successfully")
+
+ @protect()
+ @safe
+ @expose("/<publisher>/<name>/storage/persistent", methods=("GET",))
+ def list_persistent(self, publisher: str, name: str, **kwargs: Any) -> Response:
+ """List entries in persistent state.
+ ---
+ get:
+ summary: List entries in persistent state
+ parameters:
+ - in: path
+ name: publisher
+ schema:
+ type: string
+ required: true
+ description: Extension publisher
+ - in: path
+ name: name
+ schema:
+ type: string
+ required: true
+ description: Extension name
+ - in: query
+ name: shared
+ schema:
+ type: boolean
+ required: false
+ description: If true, list shared state visible to all users
+ - in: query
+ name: resource_type
+ schema:
+ type: string
+ required: false
+ description: Filter by resource type
+ - in: query
+ name: resource_uuid
+ schema:
+ type: string
+ required: false
+ description: Filter by resource UUID (requires resource_type)
+ - in: query
+ name: page
+ schema:
+ type: integer
+ required: false
+ description: Zero-indexed page number. Defaults to 0.
+ - in: query
+ name: page_size
+ schema:
+ type: integer
+ required: false
+ description: Number of entries per page. Defaults to 10. There
+ is no fixed ceiling, but a page whose combined value size
+ exceeds MAX_LIST_PAYLOAD_SIZE is rejected β reduce page_size
+ and retry if that happens.
+ responses:
+ 200:
+ description: Entries retrieved successfully
+ content:
+ application/json:
+ schema:
+ type: object
+ properties:
+ result:
+ type: array
+ items:
+ type: object
+ properties:
+ key:
+ type: string
+ value:
+ description: The stored value, or null if its
+ codec cannot be read over the REST API.
+ When 'isBinary' is true, this is a base64
+ string that must be decoded to get the
+ actual value.
+ codec:
+ type: string
+ isBinary:
+ type: boolean
+ description: Whether the stored value is
+ binary data
+ count:
+ type: integer
+ description: Total number of entries matching the
+ given scope/filters, across all pages
+ 400:
+ description: The requested page's combined value size exceeds
+ MAX_LIST_PAYLOAD_SIZE
+ 404:
+ description: Extension not found
+ """
+ extension_id = f"{publisher}.{name}"
+ extension = get_extension_or_404(extension_id)
+ if not extension:
+ return self.response_404("Extension not found")
+
+ shared = request.args.get("shared", "false").lower() == "true"
+ user_fk = None if shared else g.user.id
+ resource_type = request.args.get("resource_type")
+ resource_uuid = request.args.get("resource_uuid")
+ try:
+ page = int(request.args.get("page", 0))
+ page_size = int(request.args.get("page_size", 10))
+ except (TypeError, ValueError):
+ return self.response_400("'page' and 'page_size' must be integers")
+
+ try:
+ entries, count = ExtensionStorageDAO.list_entries(
+ extension_id,
+ user_fk=user_fk,
+ resource_type=resource_type,
+ resource_uuid=resource_uuid,
+ page=page,
+ page_size=page_size,
+ )
+ except ExtensionStorageListPayloadTooLarge as ex:
+ return self.response(ex.status, message=ex.message)
+
+ result = []
+ for entry in entries:
+ value: Any = None
+ is_binary = False
+ if entry.codec in SAFE_CODECS and entry.value is not None:
+ decoded = get_codec(entry.codec).decode(entry.value)
+ value, is_binary = _decoded_result_for_wire(decoded)
+ result.append(
+ {
+ "key": entry.key,
+ "value": value,
+ "codec": entry.codec,
+ "isBinary": is_binary,
+ }
+ )
+
+ return self.response(200, result=result, count=count)
+
+ @protect()
+ @safe
+ @expose("/<publisher>/<name>/storage/persistent/<key>", methods=("GET",))
+ def get_persistent(
+ self, publisher: str, name: str, key: str, **kwargs: Any
+ ) -> Response:
+ """Get a value from persistent state.
+ ---
+ get:
+ summary: Get a value from persistent state
+ parameters:
+ - in: path
+ name: publisher
+ schema:
+ type: string
+ required: true
+ description: Extension publisher
+ - in: path
+ name: name
+ schema:
+ type: string
+ required: true
+ description: Extension name
+ - in: path
+ name: key
+ schema:
+ type: string
+ required: true
+ description: Storage key
+ - in: query
+ name: shared
+ schema:
+ type: boolean
+ required: false
+ description: If true, read from shared state visible to all users
+ responses:
+ 200:
+ description: Value retrieved successfully
+ content:
+ application/json:
+ schema:
+ type: object
+ properties:
+ result:
+ description: The stored value. When 'isBinary' is
+ true, this is a base64 string that must be
+ decoded to get the actual value.
+ codec:
+ type: string
+ description: Name of the codec 'result' was encoded
+ with, e.g. "json" (default) or "binary"
+ isBinary:
+ type: boolean
+ description: Whether the stored value is binary
+ data
+ 400:
+ description: Value was stored with a codec unavailable over the API
+ 404:
+ description: Extension not found
+ """
+ extension_id = f"{publisher}.{name}"
+ extension = get_extension_or_404(extension_id)
+ if not extension:
+ return self.response_404("Extension not found")
+
+ shared = request.args.get("shared", "false").lower() == "true"
+ user_fk = None if shared else g.user.id
+ entry = ExtensionStorageDAO.get(extension_id, key, user_fk=user_fk)
+ if entry is None:
+ return self.response(200, result=None)
+ if entry.codec not in SAFE_CODECS:
+ return self.response_400(
+ f"Value was stored with codec '{entry.codec}', which cannot be "
+ "read over the REST API."
+ )
+
+ decoded = ExtensionStorageDAO.decode_entry(entry)
+ result, is_binary = _decoded_result_for_wire(decoded)
+
+ return self.response(200, result=result, codec=entry.codec, isBinary=is_binary)
+
+ @protect()
+ @safe
+ @expose("/<publisher>/<name>/storage/persistent/<key>", methods=("PUT",))
+ @transaction()
+ def set_persistent(
+ self, publisher: str, name: str, key: str, **kwargs: Any
+ ) -> Response:
+ """Set a value in persistent state.
+ ---
+ put:
+ summary: Set a value in persistent state
+ parameters:
+ - in: path
+ name: publisher
+ schema:
+ type: string
+ required: true
+ description: Extension publisher
+ - in: path
+ name: name
+ schema:
+ type: string
+ required: true
+ description: Extension name
+ - in: path
+ name: key
+ schema:
+ type: string
+ required: true
+ description: Storage key
+ - in: query
+ name: shared
+ schema:
+ type: boolean
+ required: false
+ description: If true, store as shared state visible to all users
+ requestBody:
+ required: true
+ content:
+ application/json:
+ schema:
+ type: object
+ required:
+ - value
+ properties:
+ value:
+ description: The value to store. When 'isBinary' is
+ true, this must be a base64 string, decoded
+ before being handed to 'codec'.
+ codec:
+ type: string
+ description: Name of the codec used to encode 'value',
+ e.g. "json" (default). Must be one of the codecs
+ allowed over the REST API.
+ isBinary:
+ type: boolean
+ description: Whether 'value' is binary data
+ encrypt:
+ type: boolean
+ description: If true, the value is encrypted at rest
+ responses:
+ 200:
+ description: Value stored successfully
+ 400:
+ description: Invalid request body, codec not allowed over the
+ API, or value exceeds MAX_VALUE_SIZE
+ 404:
+ description: Extension not found
+ 413:
+ description: Extension persistent storage quota exceeded
+ """
+ extension_id = f"{publisher}.{name}"
+ extension = get_extension_or_404(extension_id)
+ if not extension:
+ return self.response_404("Extension not found")
+
+ body = request.get_json(silent=True) or {}
+ if "value" not in body:
+ return self.response_400("Request body must contain 'value' field")
+
+ codec = body.get("codec", DEFAULT_CODEC)
Review Comment:
Fixed β same `isinstance(body, dict)` check added to `set_persistent` before any `.get()`/`in` access on the parsed body.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(extensions): add a Storage API for extensions [superset]
Posted by "codeant-ai-for-open-source[bot] (via GitHub)" <gi...@apache.org>.
codeant-ai-for-open-source[bot] commented on code in PR #39171:
URL: https://github.com/apache/superset/pull/39171#discussion_r3623003794
##########
superset/extensions/storage/persistent_model.py:
##########
@@ -0,0 +1,133 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import uuid as uuid_module
+
+from flask_appbuilder import Model
+from sqlalchemy import (
+ Boolean,
+ Column,
+ ForeignKey,
+ Index,
+ Integer,
+ LargeBinary,
+ String,
+ UniqueConstraint,
+)
+from sqlalchemy_utils import UUIDType
+from superset_core.extensions.storage.models import (
+ ExtensionStorageEntry as CoreExtensionStorageEntry,
+)
+
+from superset.models.helpers import AuditMixinNullable
+
+
+class ExtensionStorage(CoreExtensionStorageEntry, AuditMixinNullable, Model):
+ """Generic persistent key-value storage for extensions (Tier 3).
+
+ Each row is identified by (extension_id, user_fk, resource_type,
+ resource_uuid, key):
+
+ * Global scope β user_fk IS NULL, resource_type IS NULL
+ * User scope β user_fk set, resource_type IS NULL
+ * Resource scope β resource_type + resource_uuid set (user_fk optional)
+
+ The payload is stored as raw bytes (value) alongside the identifier of
+ the codec used to encode it (codec), so the same bytes can be decoded
+ back into a value on read. When is_encrypted is True the value has been
+ encrypted at the DAO layer using Fernet and must be decrypted before
+ decoding.
+ """
+
+ __tablename__ = "extension_storage"
+
+ id = Column(Integer, primary_key=True, autoincrement=True)
+ uuid = Column(
+ UUIDType(binary=True),
+ default=uuid_module.uuid4,
+ unique=True,
+ nullable=False,
+ )
+
+ # Extension identity
+ extension_id = Column(String(255), nullable=False)
+
+ # Scope discriminators β all nullable; NULLs define the scope (see docstring).
+ # No relationship() is declared for user_fk: extension storage rows are
+ # not deleted when their owning user is (ondelete="SET NULL" demotes them
+ # to global scope instead), and nothing reads through such a relationship,
+ # so there's no ORM object here for a future change to attach a cascade to.
+ user_fk = Column(
+ Integer,
+ ForeignKey(
+ "ab_user.id",
+ ondelete="SET NULL",
+ name="fk_extension_storage_user_fk_ab_user",
+ ),
+ nullable=True,
+ )
+ # No FK here: resource_type is a free-form string naming the referenced
+ # table (a built-in Superset model or a resource type defined by another
+ # extension entirely), so the table a given row's resource_uuid points
+ # into varies per row and can't be fixed at the schema level.
+ resource_type = Column(String(64), nullable=True)
+ resource_uuid = Column(String(36), nullable=True)
+
+ # Storage key within the scope
+ key = Column(String(255), nullable=False)
+
+ # Payload. Size is bounded at the application level by
+ # EXTENSIONS_PERSISTENT_STORAGE["MAX_VALUE_SIZE"], enforced in
+ # ExtensionStorageDAO.set(), rather than by a fixed column length here.
+ value = Column(LargeBinary, nullable=False)
+ # Byte length of `value`, kept in sync at write time so the quota SUM in
+ # _check_quota() can be computed from this fixed-width column instead of
+ # reading (and, once TOASTed, detoasting) every extension's stored blobs.
+ value_size = Column(Integer, nullable=False)
+ codec = Column(String(255), nullable=False, default="json")
+ is_encrypted = Column(Boolean, nullable=False, default=False)
+
+ __table_args__ = (
+ # Unique constraint prevents duplicate rows from concurrent writes
+ UniqueConstraint(
+ "extension_id",
+ "user_fk",
+ "resource_type",
+ "resource_uuid",
+ "key",
+ name="uq_extension_storage_scoped_key",
+ ),
Review Comment:
β
**Customized review instruction saved!**
**Instruction:**
> Avoid recommending partial/filtered unique indexes or sentinel-column workarounds for persistent storage models that must remain portable across MySQL and PostgreSQL and preserve nullable FK + `ondelete=SET NULL` semantics; in this area, enforce concurrency via application-level locking instead.
**Applied to:**
- `superset/extensions/storage/**`
---
π‘ *To manage or update this instruction, visit: [CodeAnt AI Settings](https://app.codeant.ai/org/settings/learnings)*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(extensions): add a Storage API for extensions [superset]
Posted by "bito-code-review[bot] (via GitHub)" <gi...@apache.org>.
bito-code-review[bot] commented on PR #39171:
URL: https://github.com/apache/superset/pull/39171#issuecomment-5036404649
AI Code Review is in progress (usually takes 3 to 15 minutes unless it's a very large PR).
<details>
<summary>
Bito Usage Guide
</summary>
**Commands**
Type the following command in the pull request comment and save the comment.
- `/review` - Manually triggers a full AI review.
- `/pause` - Pauses automatic reviews on this pull request.
- `/resume` - Resumes automatic reviews.
- `/resolve` - Marks all Bito-posted review comments as resolved.
- `/abort` - Cancels all in-progress reviews.
Refer to the <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/available-commands" target="_blank" rel="noopener noreferrer">documentation</a> for additional commands.
**Configuration**
This repository uses `Superset` You can customize the agent settings <a href="https://alpha.bito.ai/home/ai-agents/code-review-agent" target="_blank" rel="noopener noreferrer">here</a> or contact your Bito workspace admin at evan@preset.io.
**Documentation & Help**
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/getting-started/install-run-using-bito-cloud/create-or-customize-an-agent-instance" target="_blank" rel="noopener noreferrer">Customize agent settings</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/implementing-custom-code-review-rules" target="_blank" rel="noopener noreferrer">Review rules</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent" target="_blank" rel="noopener noreferrer">General documentation</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/faqs" target="_blank" rel="noopener noreferrer">FAQ</a>
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(extensions): add Tier 1 and Tier 2 storage APIs [superset]
Posted by "bito-code-review[bot] (via GitHub)" <gi...@apache.org>.
bito-code-review[bot] commented on PR #39171:
URL: https://github.com/apache/superset/pull/39171#issuecomment-4201619229
AI Code Review is in progress (usually takes 3 to 15 minutes unless it's a very large PR).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(extensions): add Tier 1 and Tier 2 storage APIs [superset]
Posted by "codeant-ai-for-open-source[bot] (via GitHub)" <gi...@apache.org>.
codeant-ai-for-open-source[bot] commented on code in PR #39171:
URL: https://github.com/apache/superset/pull/39171#discussion_r3047380468
##########
superset/extensions/storage/ephemeral_state.py:
##########
@@ -0,0 +1,192 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""
+Host implementation for Ephemeral State (Tier 2 Storage).
+
+Provides the concrete cache-backed implementation that is injected into
+superset_core.extensions.storage.ephemeral_state at startup.
+"""
+
+from __future__ import annotations
+
+from typing import Any
+
+from flask import g
+
+from superset.extensions import cache_manager
+from superset.extensions.context import get_current_extension_context
+
+# Default TTL: 1 hour
+DEFAULT_TTL = 3600
+
+# Key separator
+SEPARATOR = ":"
+
+# Key prefix for extension ephemeral state
+KEY_PREFIX = "superset-ext"
+
+
+def _get_extension_id() -> str:
+ """Get the current extension ID from context."""
+ context = get_current_extension_context()
+ if context is None:
+ raise RuntimeError(
+ "ephemeral_state can only be used within an extension context. "
+ "Ensure this code is being executed during extension loading or "
+ "within an extension API request handler."
+ )
+ return context.manifest.id
+
+
+def _get_current_user_id() -> int:
+ """Get the current authenticated user's ID."""
+ user = getattr(g, "user", None)
+ if user is None or not hasattr(user, "id"):
+ raise RuntimeError(
+ "ephemeral_state requires an authenticated user. "
+ "Ensure the request has been authenticated."
+ )
+ return user.id
+
+
+def _build_cache_key(*parts: Any) -> str:
+ """Build a namespaced cache key from parts."""
+ return SEPARATOR.join(str(part) for part in parts)
+
+
+class SharedEphemeralStateAccessor:
+ """
+ Accessor for shared (global) ephemeral state.
+
+ Data stored via this accessor is visible to all users of the extension.
+ """
+
+ def __init__(self, extension_id: str):
+ self._extension_id = extension_id
+
+ def _build_key(self, key: str) -> str:
+ """Build a shared (global) cache key."""
+ return _build_cache_key(KEY_PREFIX, self._extension_id, key)
Review Comment:
**Suggestion:** Shared keys are missing a dedicated scope segment, so a shared key like `user:42:foo` can collide with a private user-scoped key for user 42 (`superset-ext:<ext>:user:42:foo`). This breaks isolation guarantees and can leak or overwrite user-specific data through shared storage. Add an explicit `shared` namespace component to shared keys. [security]
<details>
<summary><b>Severity Level:</b> Major β οΈ</summary>
```mdx
- β Extension REST storage API mixes shared and user entries.
- β οΈ User-scoped ephemeral values can be overwritten by shared writes.
- β οΈ Violates Tier 2 storage isolation guarantees for extensions.
```
</details>
```suggestion
return _build_cache_key(KEY_PREFIX, self._extension_id, "shared", key)
```
<details>
<summary><b>Steps of Reproduction β
</b></summary>
```mdx
1. Start Superset with this PR code and ensure at least one extension is registered so
`_get_extension_or_404` in `superset/extensions/storage/api.py:55-58` returns a real
extension for its ID.
2. Authenticate as a user (so `g.user.id` is set) and call `PUT
/api/v1/extensions/storage/ephemeral/<extension_id>/foo` (handled by
`ExtensionStorageRestApi.set_ephemeral` at `superset/extensions/storage/api.py:135-189`)
with body `{"value": "private"}`; this writes to cache key
`superset-ext:<extension_id>:user:<user_id>:foo` via `_build_cache_key(KEY_PREFIX,
extension_id, "user", user_id, key)` at lines `185-187`.
3. Still authenticated, call `PUT
/api/v1/extensions/storage/ephemeral/shared/<extension_id>/user:<user_id>:foo` (handled by
`set_ephemeral_shared` at `superset/extensions/storage/api.py:279-320`) with body
`{"value": "shared"}`; this uses `_build_cache_key(KEY_PREFIX, extension_id, key)` at
lines `272` and `320-312`, producing the *same* physical cache key
`superset-ext:<extension_id>:user:<user_id>:foo` because the shared `key` string is
literally `"user:<user_id>:foo"`, overwriting the user-scoped value.
4. As the same user, call `GET /api/v1/extensions/storage/ephemeral/<extension_id>/foo`
(handled by `get_ephemeral` at `superset/extensions/storage/api.py:89-131`), which again
builds `superset-ext:<extension_id>:user:<user_id>:foo` via `_build_cache_key(KEY_PREFIX,
extension_id, "user", user_id, key)` at lines `127-129`, and observe it now returns the
shared value `"shared"`, demonstrating that the shared namespace
(`SharedEphemeralStateAccessor._build_key` at
`superset/extensions/storage/ephemeral_state.py:82-84`) and user namespace
(`EphemeralStateImpl._build_user_key` at
`superset/extensions/storage/ephemeral_state.py:128-131`) can collide when an extension
chooses keys of the form `"user:<id>:<key>"`.
```
</details>
<details>
<summary><b>Prompt for AI Agent π€ </b></summary>
```mdx
This is a comment left during a code review.
**Path:** superset/extensions/storage/ephemeral_state.py
**Line:** 84:84
**Comment:**
*Security: Shared keys are missing a dedicated scope segment, so a shared key like `user:42:foo` can collide with a private user-scoped key for user 42 (`superset-ext:<ext>:user:42:foo`). This breaks isolation guarantees and can leak or overwrite user-specific data through shared storage. Add an explicit `shared` namespace component to shared keys.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F39171&comment_hash=0f9e644439a572848d9386f310871428cb7e9458d2870bfffea36cd74a4c4108&reaction=like'>π</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F39171&comment_hash=0f9e644439a572848d9386f310871428cb7e9458d2870bfffea36cd74a4c4108&reaction=dislike'>π</a>
##########
superset-frontend/src/extensions/ExtensionsStartup.tsx:
##########
@@ -77,6 +64,7 @@ const ExtensionsStartup: React.FC<{ children?: React.ReactNode }> = ({
extensions,
menus,
sqlLab,
+ storage,
Review Comment:
**Suggestion:** The global assignment exposes the host `storage` object including its internal `forExtension` helper, which allows callers to construct storage clients for arbitrary extension IDs and bypass the intended namespace isolation contract. Expose only the public tier APIs (`localState`, `sessionState`, `ephemeralState`) on `window.superset.storage`. [security]
<details>
<summary><b>Severity Level:</b> Critical π¨</summary>
```mdx
- β Extension storage isolation bypass for tier 1 and 2.
- β οΈ Malicious scripts can read other extensions' cached data.
- β οΈ Violates documented type contract in extensions/types.ts.
```
</details>
```suggestion
storage: {
localState: storage.localState,
sessionState: storage.sessionState,
ephemeralState: storage.ephemeralState,
},
```
<details>
<summary><b>Steps of Reproduction β
</b></summary>
```mdx
1. Start the Superset frontend with this PR applied and log in as any authenticated user
so that React renders the app and `ExtensionsStartup` mounts
(`superset-frontend/src/extensions/ExtensionsStartup.tsx:39-48`).
2. When the `useEffect` in `ExtensionsStartup` runs (same file, lines 48-58), it assigns
`window.superset = { ...supersetCore, authentication, core, ..., sqlLab, storage, views
}`, where `storage` is imported from `'src/core'` (line 31) and is the host storage object
defined in `superset-frontend/src/core/storage/index.ts:52-59`, which explicitly includes
the internal `forExtension` helper.
3. With the page loaded and before any extension code needs to run, open the browser
developer console and execute:
`window.superset.storage.forExtension('some_other_extension').ephemeralState.set('test_key',
'secret');`
This calls the internal `forExtension(extensionId)` exported in
`src/core/storage/index.ts:44-50`, which returns a storage instance bound to the
attacker-controlled `extensionId` without ever calling `getCurrentExtensionId()` from
`src/core/storage/utils.ts:36-47`.
4. Any extension whose real ID is `'some_other_extension'` will later access its ephemeral
state via the public API implementation in `src/core/storage/ephemeralState.ts:89-120`,
which uses `getCurrentExtensionId()` for its own calls; those APIs will read/write data
under the same `/api/v1/extensions/storage/ephemeral/<extensionId>/<key>` paths (see
`buildEphemeralStateUrl` at lines 34-42 and bound implementation at lines 168-193),
allowing the attacker-controlled `forExtension('some_other_extension')` call to inject or
read data in that other extension's namespace, breaking the intended per-extension
isolation.
```
</details>
<details>
<summary><b>Prompt for AI Agent π€ </b></summary>
```mdx
This is a comment left during a code review.
**Path:** superset-frontend/src/extensions/ExtensionsStartup.tsx
**Line:** 67:67
**Comment:**
*Security: The global assignment exposes the host `storage` object including its internal `forExtension` helper, which allows callers to construct storage clients for arbitrary extension IDs and bypass the intended namespace isolation contract. Expose only the public tier APIs (`localState`, `sessionState`, `ephemeralState`) on `window.superset.storage`.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F39171&comment_hash=aa51a7f2615f7a3991f48c5d11242aabe6a8a2f3497372520c4c11425551e3a4&reaction=like'>π</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F39171&comment_hash=aa51a7f2615f7a3991f48c5d11242aabe6a8a2f3497372520c4c11425551e3a4&reaction=dislike'>π</a>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(extensions): add Tier 1 and Tier 2 storage APIs [superset]
Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #39171:
URL: https://github.com/apache/superset/pull/39171#issuecomment-4201636825
## [Codecov](https://app.codecov.io/gh/apache/superset/pull/39171?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:white_check_mark: All modified and coverable lines are covered by tests.
:white_check_mark: Project coverage is 64.64%. Comparing base ([`d796543`](https://app.codecov.io/gh/apache/superset/commit/d796543f5a0c5f6306f3140d48647640b7a28a14?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`bfcc05a`](https://app.codecov.io/gh/apache/superset/commit/bfcc05a56f030b49fccf2ba9102718f19016d285?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 14 commits behind head on master.
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #39171 +/- ##
==========================================
+ Coverage 64.52% 64.64% +0.12%
==========================================
Files 2536 2541 +5
Lines 131208 131827 +619
Branches 30457 30457
==========================================
+ Hits 84661 85224 +563
- Misses 45084 45140 +56
Partials 1463 1463
```
| [Flag](https://app.codecov.io/gh/apache/superset/pull/39171/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Ξ | |
|---|---|---|
| [superset-extensions-cli](https://app.codecov.io/gh/apache/superset/pull/39171/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `90.82% <ΓΈ> (?)` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
</details>
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/superset/pull/39171?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(extensions): add a Storage API for extensions [superset]
Posted by "codeant-ai-for-open-source[bot] (via GitHub)" <gi...@apache.org>.
codeant-ai-for-open-source[bot] commented on code in PR #39171:
URL: https://github.com/apache/superset/pull/39171#discussion_r3624442959
##########
superset/extensions/storage/api.py:
##########
@@ -0,0 +1,740 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""
+REST API for extension storage.
+
+Provides HTTP endpoints for frontend extensions to access server-side
+ephemeral storage without direct backend code.
+
+All operations are user-scoped by default. Use `?shared=true` query param
+to access shared state visible to all users.
+"""
+
+from __future__ import annotations
+
+import base64
+from typing import Any
+
+from flask import g, request
+from flask.wrappers import Response
+from flask_appbuilder.api import BaseApi, expose, protect, safe
+
+from superset.extensions.storage.codecs import DEFAULT_CODEC, get_codec, SAFE_CODECS
+from superset.extensions.storage.ephemeral_dao import (
+ ExtensionEphemeralDAO,
+ ExtensionEphemeralTTLInvalid,
+ ExtensionEphemeralValueTooLarge,
+)
+from superset.extensions.storage.persistent_dao import (
+ ExtensionStorageDAO,
+ ExtensionStorageKeyTooLong,
+ ExtensionStorageListPayloadTooLarge,
+ ExtensionStorageQuotaExceeded,
+ ExtensionStorageValueTooLarge,
+)
+from superset.extensions.storage.utils import get_extension_or_404, parse_ttl
+from superset.key_value.exceptions import KeyValueCodecEncodeException
+from superset.utils.decorators import transaction
+
+
+def _decoded_result_for_wire(decoded: Any) -> tuple[Any, bool]:
+ """Convert a codec-decoded value into its JSON wire representation.
+
+ JSON has no byte type, so a raw `bytes` value is base64-encoded to a
+ string for the response and flagged as such; every other value is
+ already JSON-representable as-is. Checked on the decoded value's
+ actual type, not the codec's name, so this keeps working for a codec
+ this module didn't define.
+
+ :returns: (wire_value, is_binary)
+ """
+ if isinstance(decoded, bytes):
+ return base64.b64encode(decoded).decode("ascii"), True
+ return decoded, False
+
+
+def _wire_value_for_request(value: Any, is_binary: bool) -> Any:
+ """Convert a request body's JSON `value` into a codec's input type.
+
+ `is_binary` is an explicit flag from the caller: JSON has no byte
+ type, so there is no way to tell, from the JSON `value` alone,
+ whether it is a base64 string that must be decoded to bytes before
+ being handed to the codec's `encode`, or a literal value to pass
+ through as-is. Only the caller knows which it sent.
+
+ :raises ValueError: if `is_binary` is set and `value` is not a valid
+ base64 string.
+ """
+ if is_binary:
+ return base64.b64decode(value, validate=True)
+ return value
+
+
+class ExtensionStorageRestApi(BaseApi):
+ """REST API for extension ephemeral (Tier 2) and persistent (Tier 3) storage."""
+
+ allow_browser_login = True
+ route_base = "/api/v1/extensions"
+
+ def response(self, status_code: int, **kwargs: Any) -> Response:
+ """Helper method to create JSON responses."""
+ from flask import jsonify
+
+ return jsonify(kwargs), status_code
+
+ def response_404(self, message: str = "Not found") -> Response:
+ """Helper method to create 404 responses."""
+ from flask import jsonify
+
+ return jsonify({"message": message}), 404
+
+ def response_400(self, message: str) -> Response:
+ """Helper method to create 400 responses."""
+ from flask import jsonify
+
+ return jsonify({"message": message}), 400
+
+ @protect()
+ @safe
+ @expose("/<publisher>/<name>/storage/ephemeral/<key>", methods=("GET",))
+ def get_ephemeral(
+ self, publisher: str, name: str, key: str, **kwargs: Any
+ ) -> Response:
+ """Get a value from ephemeral state.
+ ---
+ get:
+ summary: Get a value from ephemeral state
+ parameters:
+ - in: path
+ name: publisher
+ schema:
+ type: string
+ required: true
+ description: Extension publisher
+ - in: path
+ name: name
+ schema:
+ type: string
+ required: true
+ description: Extension name
+ - in: path
+ name: key
+ schema:
+ type: string
+ required: true
+ description: Storage key
+ - in: query
+ name: shared
+ schema:
+ type: boolean
+ required: false
+ description: If true, read from shared state visible to all users
+ responses:
+ 200:
+ description: Value retrieved successfully
+ content:
+ application/json:
+ schema:
+ type: object
+ properties:
+ result:
+ description: The stored value. When 'isBinary' is
+ true, this is a base64 string that must be
+ decoded to get the actual value.
+ codec:
+ type: string
+ description: Name of the codec 'result' was encoded
+ with, e.g. "json" (default) or "binary"
+ isBinary:
+ type: boolean
+ description: Whether the stored value is binary
+ data
+ 400:
+ description: Value was stored with a codec unavailable over the API
+ 404:
+ description: Extension not found
+ """
+ extension_id = f"{publisher}.{name}"
+ extension = get_extension_or_404(extension_id)
+ if not extension:
+ return self.response_404("Extension not found")
+
+ shared = request.args.get("shared", "false").lower() == "true"
+ raw = ExtensionEphemeralDAO.get_raw(extension_id, key, shared=shared)
+ if raw is None:
+ return self.response(200, result=None)
+ value, codec = raw
+ if codec not in SAFE_CODECS:
+ return self.response_400(
+ f"Value was stored with codec '{codec}', which cannot be "
+ "read over the REST API."
+ )
+
+ result, is_binary = _decoded_result_for_wire(get_codec(codec).decode(value))
+ return self.response(200, result=result, codec=codec, isBinary=is_binary)
+
+ @protect()
+ @safe
+ @expose("/<publisher>/<name>/storage/ephemeral/<key>", methods=("PUT",))
+ def set_ephemeral(
+ self, publisher: str, name: str, key: str, **kwargs: Any
+ ) -> Response:
+ """Set a value in ephemeral state.
+ ---
+ put:
+ summary: Set a value in ephemeral state
+ parameters:
+ - in: path
+ name: publisher
+ schema:
+ type: string
+ required: true
+ description: Extension publisher
+ - in: path
+ name: name
+ schema:
+ type: string
+ required: true
+ description: Extension name
+ - in: path
+ name: key
+ schema:
+ type: string
+ required: true
+ description: Storage key
+ - in: query
+ name: shared
+ schema:
+ type: boolean
+ required: false
+ description: If true, store as shared state visible to all users
+ requestBody:
+ required: true
+ content:
+ application/json:
+ schema:
+ type: object
+ required:
+ - value
+ - ttl
+ properties:
+ value:
+ description: The value to store (must not exceed
+ MAX_VALUE_SIZE bytes once encoded with 'codec').
+ When 'isBinary' is true, this must be a base64
+ string, decoded before being handed to 'codec'.
+ codec:
+ type: string
+ description: Name of the codec used to encode 'value',
+ e.g. "json" (default). Must be one of the codecs
+ allowed over the REST API.
+ isBinary:
+ type: boolean
+ description: Whether 'value' is binary data
+ ttl:
+ type: integer
+ description: Time-to-live in seconds (must be a positive
+ integer not exceeding MAX_TTL)
+ responses:
+ 200:
+ description: Value stored successfully
+ 400:
+ description: Invalid request body, or codec not allowed over the API
+ 404:
+ description: Extension not found
+ """
+ extension_id = f"{publisher}.{name}"
+ extension = get_extension_or_404(extension_id)
+ if not extension:
+ return self.response_404("Extension not found")
+
+ body = request.get_json(silent=True) or {}
+ if "value" not in body:
+ return self.response_400("Request body must contain 'value' field")
+
+ codec = body.get("codec", DEFAULT_CODEC)
Review Comment:
**Suggestion:** The request body is not validated to be a JSON object before calling mapping methods, so a valid non-object JSON payload (for example a list containing "value") can bypass the first check and then crash with an AttributeError on `body.get(...)`. Add an explicit object-type check and return 400 for non-object payloads. [type error]
<details>
<summary><b>Severity Level:</b> Major β οΈ</summary>
```mdx
- β Ephemeral storage writes can return 500 on bad JSON.
- β οΈ Extensions see 5xx instead of clear client error.
```
</details>
<details>
<summary><b>Steps of Reproduction β
</b></summary>
```mdx
1. Start Superset with this PR and authenticate a user.
2. Call `PUT /api/v1/extensions/test.publisher/testext/storage/ephemeral/foo` (handled by
`ExtensionStorageRestApi.set_ephemeral` in `superset/extensions/storage/api.py:194-299`).
3. Send JSON body `["value"]` with `Content-Type: application/json`, so
`request.get_json(silent=True)` at line 266 returns a list.
4. At lines 266-270, `"value" in body` succeeds on the list, then `body.get("codec",
DEFAULT_CODEC)` raises `AttributeError: 'list' object has no attribute 'get'`, producing a
500 instead of a 400.
```
</details>
[](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=ffcaf1c20000486fa6da2cbd6bcea6f7&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=ffcaf1c20000486fa6da2cbd6bcea6f7&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset)
*(Use Cmd/Ctrl + Click for best experience)*
<details>
<summary><b>Prompt for AI Agent π€ </b></summary>
```mdx
This is a comment left during a code review.
**Path:** superset/extensions/storage/api.py
**Line:** 266:270
**Comment:**
*Type Error: The request body is not validated to be a JSON object before calling mapping methods, so a valid non-object JSON payload (for example a list containing "value") can bypass the first check and then crash with an AttributeError on `body.get(...)`. Add an explicit object-type check and return 400 for non-object payloads.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F39171&comment_hash=9b40cff66af0573e8de49408bf4ef580776ac7f607c66ba6d94e742dc5074883&reaction=like'>π</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F39171&comment_hash=9b40cff66af0573e8de49408bf4ef580776ac7f607c66ba6d94e742dc5074883&reaction=dislike'>π</a>
##########
superset/extensions/storage/api.py:
##########
@@ -0,0 +1,740 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""
+REST API for extension storage.
+
+Provides HTTP endpoints for frontend extensions to access server-side
+ephemeral storage without direct backend code.
+
+All operations are user-scoped by default. Use `?shared=true` query param
+to access shared state visible to all users.
+"""
+
+from __future__ import annotations
+
+import base64
+from typing import Any
+
+from flask import g, request
+from flask.wrappers import Response
+from flask_appbuilder.api import BaseApi, expose, protect, safe
+
+from superset.extensions.storage.codecs import DEFAULT_CODEC, get_codec, SAFE_CODECS
+from superset.extensions.storage.ephemeral_dao import (
+ ExtensionEphemeralDAO,
+ ExtensionEphemeralTTLInvalid,
+ ExtensionEphemeralValueTooLarge,
+)
+from superset.extensions.storage.persistent_dao import (
+ ExtensionStorageDAO,
+ ExtensionStorageKeyTooLong,
+ ExtensionStorageListPayloadTooLarge,
+ ExtensionStorageQuotaExceeded,
+ ExtensionStorageValueTooLarge,
+)
+from superset.extensions.storage.utils import get_extension_or_404, parse_ttl
+from superset.key_value.exceptions import KeyValueCodecEncodeException
+from superset.utils.decorators import transaction
+
+
+def _decoded_result_for_wire(decoded: Any) -> tuple[Any, bool]:
+ """Convert a codec-decoded value into its JSON wire representation.
+
+ JSON has no byte type, so a raw `bytes` value is base64-encoded to a
+ string for the response and flagged as such; every other value is
+ already JSON-representable as-is. Checked on the decoded value's
+ actual type, not the codec's name, so this keeps working for a codec
+ this module didn't define.
+
+ :returns: (wire_value, is_binary)
+ """
+ if isinstance(decoded, bytes):
+ return base64.b64encode(decoded).decode("ascii"), True
+ return decoded, False
+
+
+def _wire_value_for_request(value: Any, is_binary: bool) -> Any:
+ """Convert a request body's JSON `value` into a codec's input type.
+
+ `is_binary` is an explicit flag from the caller: JSON has no byte
+ type, so there is no way to tell, from the JSON `value` alone,
+ whether it is a base64 string that must be decoded to bytes before
+ being handed to the codec's `encode`, or a literal value to pass
+ through as-is. Only the caller knows which it sent.
+
+ :raises ValueError: if `is_binary` is set and `value` is not a valid
+ base64 string.
+ """
+ if is_binary:
+ return base64.b64decode(value, validate=True)
+ return value
+
+
+class ExtensionStorageRestApi(BaseApi):
+ """REST API for extension ephemeral (Tier 2) and persistent (Tier 3) storage."""
+
+ allow_browser_login = True
+ route_base = "/api/v1/extensions"
+
+ def response(self, status_code: int, **kwargs: Any) -> Response:
+ """Helper method to create JSON responses."""
+ from flask import jsonify
+
+ return jsonify(kwargs), status_code
+
+ def response_404(self, message: str = "Not found") -> Response:
+ """Helper method to create 404 responses."""
+ from flask import jsonify
+
+ return jsonify({"message": message}), 404
+
+ def response_400(self, message: str) -> Response:
+ """Helper method to create 400 responses."""
+ from flask import jsonify
+
+ return jsonify({"message": message}), 400
+
+ @protect()
+ @safe
+ @expose("/<publisher>/<name>/storage/ephemeral/<key>", methods=("GET",))
+ def get_ephemeral(
+ self, publisher: str, name: str, key: str, **kwargs: Any
+ ) -> Response:
+ """Get a value from ephemeral state.
+ ---
+ get:
+ summary: Get a value from ephemeral state
+ parameters:
+ - in: path
+ name: publisher
+ schema:
+ type: string
+ required: true
+ description: Extension publisher
+ - in: path
+ name: name
+ schema:
+ type: string
+ required: true
+ description: Extension name
+ - in: path
+ name: key
+ schema:
+ type: string
+ required: true
+ description: Storage key
+ - in: query
+ name: shared
+ schema:
+ type: boolean
+ required: false
+ description: If true, read from shared state visible to all users
+ responses:
+ 200:
+ description: Value retrieved successfully
+ content:
+ application/json:
+ schema:
+ type: object
+ properties:
+ result:
+ description: The stored value. When 'isBinary' is
+ true, this is a base64 string that must be
+ decoded to get the actual value.
+ codec:
+ type: string
+ description: Name of the codec 'result' was encoded
+ with, e.g. "json" (default) or "binary"
+ isBinary:
+ type: boolean
+ description: Whether the stored value is binary
+ data
+ 400:
+ description: Value was stored with a codec unavailable over the API
+ 404:
+ description: Extension not found
+ """
+ extension_id = f"{publisher}.{name}"
+ extension = get_extension_or_404(extension_id)
+ if not extension:
+ return self.response_404("Extension not found")
+
+ shared = request.args.get("shared", "false").lower() == "true"
+ raw = ExtensionEphemeralDAO.get_raw(extension_id, key, shared=shared)
+ if raw is None:
+ return self.response(200, result=None)
+ value, codec = raw
+ if codec not in SAFE_CODECS:
+ return self.response_400(
+ f"Value was stored with codec '{codec}', which cannot be "
+ "read over the REST API."
+ )
+
+ result, is_binary = _decoded_result_for_wire(get_codec(codec).decode(value))
+ return self.response(200, result=result, codec=codec, isBinary=is_binary)
+
+ @protect()
+ @safe
+ @expose("/<publisher>/<name>/storage/ephemeral/<key>", methods=("PUT",))
+ def set_ephemeral(
+ self, publisher: str, name: str, key: str, **kwargs: Any
+ ) -> Response:
+ """Set a value in ephemeral state.
+ ---
+ put:
+ summary: Set a value in ephemeral state
+ parameters:
+ - in: path
+ name: publisher
+ schema:
+ type: string
+ required: true
+ description: Extension publisher
+ - in: path
+ name: name
+ schema:
+ type: string
+ required: true
+ description: Extension name
+ - in: path
+ name: key
+ schema:
+ type: string
+ required: true
+ description: Storage key
+ - in: query
+ name: shared
+ schema:
+ type: boolean
+ required: false
+ description: If true, store as shared state visible to all users
+ requestBody:
+ required: true
+ content:
+ application/json:
+ schema:
+ type: object
+ required:
+ - value
+ - ttl
+ properties:
+ value:
+ description: The value to store (must not exceed
+ MAX_VALUE_SIZE bytes once encoded with 'codec').
+ When 'isBinary' is true, this must be a base64
+ string, decoded before being handed to 'codec'.
+ codec:
+ type: string
+ description: Name of the codec used to encode 'value',
+ e.g. "json" (default). Must be one of the codecs
+ allowed over the REST API.
+ isBinary:
+ type: boolean
+ description: Whether 'value' is binary data
+ ttl:
+ type: integer
+ description: Time-to-live in seconds (must be a positive
+ integer not exceeding MAX_TTL)
+ responses:
+ 200:
+ description: Value stored successfully
+ 400:
+ description: Invalid request body, or codec not allowed over the API
+ 404:
+ description: Extension not found
+ """
+ extension_id = f"{publisher}.{name}"
+ extension = get_extension_or_404(extension_id)
+ if not extension:
+ return self.response_404("Extension not found")
+
+ body = request.get_json(silent=True) or {}
+ if "value" not in body:
+ return self.response_400("Request body must contain 'value' field")
+
+ codec = body.get("codec", DEFAULT_CODEC)
+ if not isinstance(codec, str) or codec not in SAFE_CODECS:
+ return self.response_400(
+ f"Codec '{codec}' is not allowed over the REST API."
+ )
+
+ is_binary = bool(body.get("isBinary", False))
+ try:
+ value = _wire_value_for_request(body["value"], is_binary)
+ except (ValueError, TypeError):
+ return self.response_400(
+ "Value must be a valid base64 string when 'isBinary' is true."
+ )
+ ttl, error = parse_ttl(body)
+ if error:
+ return self.response_400(error)
+
+ shared = request.args.get("shared", "false").lower() == "true"
+ try:
+ ExtensionEphemeralDAO.set(
+ extension_id, key, value, ttl, codec=codec, shared=shared
+ )
+ except (ExtensionEphemeralTTLInvalid, ExtensionEphemeralValueTooLarge) as ex:
+ return self.response_400(ex.message)
+ except (KeyValueCodecEncodeException, TypeError, ValueError):
+ return self.response_400(
+ f"Value could not be encoded with codec '{codec}'."
+ )
+
+ return self.response(200, message="Value stored successfully")
+
+ @protect()
+ @safe
+ @expose("/<publisher>/<name>/storage/ephemeral/<key>", methods=("DELETE",))
+ def delete_ephemeral(
+ self, publisher: str, name: str, key: str, **kwargs: Any
+ ) -> Response:
+ """Delete a value from ephemeral state.
+ ---
+ delete:
+ summary: Delete a value from ephemeral state
+ parameters:
+ - in: path
+ name: publisher
+ schema:
+ type: string
+ required: true
+ description: Extension publisher
+ - in: path
+ name: name
+ schema:
+ type: string
+ required: true
+ description: Extension name
+ - in: path
+ name: key
+ schema:
+ type: string
+ required: true
+ description: Storage key
+ - in: query
+ name: shared
+ schema:
+ type: boolean
+ required: false
+ description: If true, delete from shared state
+ responses:
+ 200:
+ description: Value deleted successfully
+ 404:
+ description: Extension not found
+ """
+ extension_id = f"{publisher}.{name}"
+ extension = get_extension_or_404(extension_id)
+ if not extension:
+ return self.response_404("Extension not found")
+
+ shared = request.args.get("shared", "false").lower() == "true"
+ ExtensionEphemeralDAO.delete(extension_id, key, shared=shared)
+
+ return self.response(200, message="Value deleted successfully")
+
+ @protect()
+ @safe
+ @expose("/<publisher>/<name>/storage/persistent", methods=("GET",))
+ def list_persistent(self, publisher: str, name: str, **kwargs: Any) -> Response:
+ """List entries in persistent state.
+ ---
+ get:
+ summary: List entries in persistent state
+ parameters:
+ - in: path
+ name: publisher
+ schema:
+ type: string
+ required: true
+ description: Extension publisher
+ - in: path
+ name: name
+ schema:
+ type: string
+ required: true
+ description: Extension name
+ - in: query
+ name: shared
+ schema:
+ type: boolean
+ required: false
+ description: If true, list shared state visible to all users
+ - in: query
+ name: resource_type
+ schema:
+ type: string
+ required: false
+ description: Filter by resource type
+ - in: query
+ name: resource_uuid
+ schema:
+ type: string
+ required: false
+ description: Filter by resource UUID (requires resource_type)
+ - in: query
+ name: page
+ schema:
+ type: integer
+ required: false
+ description: Zero-indexed page number. Defaults to 0.
+ - in: query
+ name: page_size
+ schema:
+ type: integer
+ required: false
+ description: Number of entries per page. Defaults to 10. There
+ is no fixed ceiling, but a page whose combined value size
+ exceeds MAX_LIST_PAYLOAD_SIZE is rejected β reduce page_size
+ and retry if that happens.
+ responses:
+ 200:
+ description: Entries retrieved successfully
+ content:
+ application/json:
+ schema:
+ type: object
+ properties:
+ result:
+ type: array
+ items:
+ type: object
+ properties:
+ key:
+ type: string
+ value:
+ description: The stored value, or null if its
+ codec cannot be read over the REST API.
+ When 'isBinary' is true, this is a base64
+ string that must be decoded to get the
+ actual value.
+ codec:
+ type: string
+ isBinary:
+ type: boolean
+ description: Whether the stored value is
+ binary data
+ count:
+ type: integer
+ description: Total number of entries matching the
+ given scope/filters, across all pages
+ 400:
+ description: The requested page's combined value size exceeds
+ MAX_LIST_PAYLOAD_SIZE
+ 404:
+ description: Extension not found
+ """
+ extension_id = f"{publisher}.{name}"
+ extension = get_extension_or_404(extension_id)
+ if not extension:
+ return self.response_404("Extension not found")
+
+ shared = request.args.get("shared", "false").lower() == "true"
+ user_fk = None if shared else g.user.id
+ resource_type = request.args.get("resource_type")
+ resource_uuid = request.args.get("resource_uuid")
+ try:
+ page = int(request.args.get("page", 0))
+ page_size = int(request.args.get("page_size", 10))
+ except (TypeError, ValueError):
+ return self.response_400("'page' and 'page_size' must be integers")
+
+ try:
+ entries, count = ExtensionStorageDAO.list_entries(
+ extension_id,
+ user_fk=user_fk,
+ resource_type=resource_type,
+ resource_uuid=resource_uuid,
+ page=page,
+ page_size=page_size,
+ )
+ except ExtensionStorageListPayloadTooLarge as ex:
+ return self.response(ex.status, message=ex.message)
+
+ result = []
+ for entry in entries:
+ value: Any = None
+ is_binary = False
+ if entry.codec in SAFE_CODECS and entry.value is not None:
+ decoded = get_codec(entry.codec).decode(entry.value)
+ value, is_binary = _decoded_result_for_wire(decoded)
+ result.append(
+ {
+ "key": entry.key,
+ "value": value,
+ "codec": entry.codec,
+ "isBinary": is_binary,
+ }
+ )
+
+ return self.response(200, result=result, count=count)
+
+ @protect()
+ @safe
+ @expose("/<publisher>/<name>/storage/persistent/<key>", methods=("GET",))
+ def get_persistent(
+ self, publisher: str, name: str, key: str, **kwargs: Any
+ ) -> Response:
+ """Get a value from persistent state.
+ ---
+ get:
+ summary: Get a value from persistent state
+ parameters:
+ - in: path
+ name: publisher
+ schema:
+ type: string
+ required: true
+ description: Extension publisher
+ - in: path
+ name: name
+ schema:
+ type: string
+ required: true
+ description: Extension name
+ - in: path
+ name: key
+ schema:
+ type: string
+ required: true
+ description: Storage key
+ - in: query
+ name: shared
+ schema:
+ type: boolean
+ required: false
+ description: If true, read from shared state visible to all users
+ responses:
+ 200:
+ description: Value retrieved successfully
+ content:
+ application/json:
+ schema:
+ type: object
+ properties:
+ result:
+ description: The stored value. When 'isBinary' is
+ true, this is a base64 string that must be
+ decoded to get the actual value.
+ codec:
+ type: string
+ description: Name of the codec 'result' was encoded
+ with, e.g. "json" (default) or "binary"
+ isBinary:
+ type: boolean
+ description: Whether the stored value is binary
+ data
+ 400:
+ description: Value was stored with a codec unavailable over the API
+ 404:
+ description: Extension not found
+ """
+ extension_id = f"{publisher}.{name}"
+ extension = get_extension_or_404(extension_id)
+ if not extension:
+ return self.response_404("Extension not found")
+
+ shared = request.args.get("shared", "false").lower() == "true"
+ user_fk = None if shared else g.user.id
+ entry = ExtensionStorageDAO.get(extension_id, key, user_fk=user_fk)
+ if entry is None:
+ return self.response(200, result=None)
+ if entry.codec not in SAFE_CODECS:
+ return self.response_400(
+ f"Value was stored with codec '{entry.codec}', which cannot be "
+ "read over the REST API."
+ )
+
+ decoded = ExtensionStorageDAO.decode_entry(entry)
+ result, is_binary = _decoded_result_for_wire(decoded)
+
+ return self.response(200, result=result, codec=entry.codec, isBinary=is_binary)
+
+ @protect()
+ @safe
+ @expose("/<publisher>/<name>/storage/persistent/<key>", methods=("PUT",))
+ @transaction()
+ def set_persistent(
+ self, publisher: str, name: str, key: str, **kwargs: Any
+ ) -> Response:
+ """Set a value in persistent state.
+ ---
+ put:
+ summary: Set a value in persistent state
+ parameters:
+ - in: path
+ name: publisher
+ schema:
+ type: string
+ required: true
+ description: Extension publisher
+ - in: path
+ name: name
+ schema:
+ type: string
+ required: true
+ description: Extension name
+ - in: path
+ name: key
+ schema:
+ type: string
+ required: true
+ description: Storage key
+ - in: query
+ name: shared
+ schema:
+ type: boolean
+ required: false
+ description: If true, store as shared state visible to all users
+ requestBody:
+ required: true
+ content:
+ application/json:
+ schema:
+ type: object
+ required:
+ - value
+ properties:
+ value:
+ description: The value to store. When 'isBinary' is
+ true, this must be a base64 string, decoded
+ before being handed to 'codec'.
+ codec:
+ type: string
+ description: Name of the codec used to encode 'value',
+ e.g. "json" (default). Must be one of the codecs
+ allowed over the REST API.
+ isBinary:
+ type: boolean
+ description: Whether 'value' is binary data
+ encrypt:
+ type: boolean
+ description: If true, the value is encrypted at rest
+ responses:
+ 200:
+ description: Value stored successfully
+ 400:
+ description: Invalid request body, codec not allowed over the
+ API, or value exceeds MAX_VALUE_SIZE
+ 404:
+ description: Extension not found
+ 413:
+ description: Extension persistent storage quota exceeded
+ """
+ extension_id = f"{publisher}.{name}"
+ extension = get_extension_or_404(extension_id)
+ if not extension:
+ return self.response_404("Extension not found")
+
+ body = request.get_json(silent=True) or {}
+ if "value" not in body:
+ return self.response_400("Request body must contain 'value' field")
+
+ codec = body.get("codec", DEFAULT_CODEC)
Review Comment:
**Suggestion:** The same body-shape bug exists in persistent writes: non-dict JSON can pass the membership check and then fail on `body.get`, causing a 500 instead of a client error. Validate that the parsed JSON is a dictionary before accessing keys. [type error]
<details>
<summary><b>Severity Level:</b> Major β οΈ</summary>
```mdx
- β Persistent storage writes can fail with internal errors.
- β οΈ Extensions receive 5xx instead of actionable validation message.
```
</details>
<details>
<summary><b>Steps of Reproduction β
</b></summary>
```mdx
1. Start Superset with this PR and authenticate a user.
2. Call `PUT /api/v1/extensions/test.publisher/testext/storage/persistent/foo` (handled by
`ExtensionStorageRestApi.set_persistent` in `superset/extensions/storage/api.py:571-687`).
3. Send JSON body `["value"]` with `Content-Type: application/json`, so
`request.get_json(silent=True)` at line 645 returns a list.
4. At lines 645-649, `"value" in body` succeeds, then `body.get("codec", DEFAULT_CODEC)`
raises `AttributeError`, causing a 500 response instead of a 400.
```
</details>
[](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=323b162ca26544849382ef5bd5a2f182&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=323b162ca26544849382ef5bd5a2f182&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset)
*(Use Cmd/Ctrl + Click for best experience)*
<details>
<summary><b>Prompt for AI Agent π€ </b></summary>
```mdx
This is a comment left during a code review.
**Path:** superset/extensions/storage/api.py
**Line:** 645:649
**Comment:**
*Type Error: The same body-shape bug exists in persistent writes: non-dict JSON can pass the membership check and then fail on `body.get`, causing a 500 instead of a client error. Validate that the parsed JSON is a dictionary before accessing keys.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F39171&comment_hash=36286e4a239ce92294774a9d993593ab8f3afed2e3bcec4ef2d3be29a48d4124&reaction=like'>π</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F39171&comment_hash=36286e4a239ce92294774a9d993593ab8f3afed2e3bcec4ef2d3be29a48d4124&reaction=dislike'>π</a>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(extensions): add a Storage API for extensions [superset]
Posted by "michael-s-molina (via GitHub)" <gi...@apache.org>.
michael-s-molina commented on code in PR #39171:
URL: https://github.com/apache/superset/pull/39171#discussion_r3622997140
##########
superset-core/src/superset_core/extensions/context.py:
##########
@@ -0,0 +1,136 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""
+Extension Context API for superset-core extensions.
+
+Provides access to the current extension's context, including metadata
+and scoped resources like storage. Extensions call `get_context()` to
+access their context during execution.
+
+The context is set by the host (Superset) during extension loading and
+is only available within extension code.
+
+Usage:
+ from superset_core.extensions.context import get_context
+
+ def setup():
+ ctx = get_context()
+
+ # Access extension metadata
+ print(f"Running {ctx.extension.displayName} v{ctx.extension.version}")
+
+ # Access extension-scoped storage
+ ctx.storage.ephemeral.set("lastRun", time.time())
+ data = ctx.storage.ephemeral.get("cachedData")
Review Comment:
Fixed β added `EphemeralSetOptions(ttl=3600)` to the example.
##########
superset/extensions/storage/persistent_model.py:
##########
@@ -0,0 +1,133 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import uuid as uuid_module
+
+from flask_appbuilder import Model
+from sqlalchemy import (
+ Boolean,
+ Column,
+ ForeignKey,
+ Index,
+ Integer,
+ LargeBinary,
+ String,
+ UniqueConstraint,
+)
+from sqlalchemy_utils import UUIDType
+from superset_core.extensions.storage.models import (
+ ExtensionStorageEntry as CoreExtensionStorageEntry,
+)
+
+from superset.models.helpers import AuditMixinNullable
+
+
+class ExtensionStorage(CoreExtensionStorageEntry, AuditMixinNullable, Model):
+ """Generic persistent key-value storage for extensions (Tier 3).
+
+ Each row is identified by (extension_id, user_fk, resource_type,
+ resource_uuid, key):
+
+ * Global scope β user_fk IS NULL, resource_type IS NULL
+ * User scope β user_fk set, resource_type IS NULL
+ * Resource scope β resource_type + resource_uuid set (user_fk optional)
+
+ The payload is stored as raw bytes (value) alongside the identifier of
+ the codec used to encode it (codec), so the same bytes can be decoded
+ back into a value on read. When is_encrypted is True the value has been
+ encrypted at the DAO layer using Fernet and must be decrypted before
+ decoding.
+ """
+
+ __tablename__ = "extension_storage"
+
+ id = Column(Integer, primary_key=True, autoincrement=True)
+ uuid = Column(
+ UUIDType(binary=True),
+ default=uuid_module.uuid4,
+ unique=True,
+ nullable=False,
+ )
+
+ # Extension identity
+ extension_id = Column(String(255), nullable=False)
+
+ # Scope discriminators β all nullable; NULLs define the scope (see docstring).
+ # No relationship() is declared for user_fk: extension storage rows are
+ # not deleted when their owning user is (ondelete="SET NULL" demotes them
+ # to global scope instead), and nothing reads through such a relationship,
+ # so there's no ORM object here for a future change to attach a cascade to.
+ user_fk = Column(
+ Integer,
+ ForeignKey(
+ "ab_user.id",
+ ondelete="SET NULL",
+ name="fk_extension_storage_user_fk_ab_user",
+ ),
+ nullable=True,
+ )
+ # No FK here: resource_type is a free-form string naming the referenced
+ # table (a built-in Superset model or a resource type defined by another
+ # extension entirely), so the table a given row's resource_uuid points
+ # into varies per row and can't be fixed at the schema level.
+ resource_type = Column(String(64), nullable=True)
+ resource_uuid = Column(String(36), nullable=True)
+
+ # Storage key within the scope
+ key = Column(String(255), nullable=False)
+
+ # Payload. Size is bounded at the application level by
+ # EXTENSIONS_PERSISTENT_STORAGE["MAX_VALUE_SIZE"], enforced in
+ # ExtensionStorageDAO.set(), rather than by a fixed column length here.
+ value = Column(LargeBinary, nullable=False)
+ # Byte length of `value`, kept in sync at write time so the quota SUM in
+ # _check_quota() can be computed from this fixed-width column instead of
+ # reading (and, once TOASTed, detoasting) every extension's stored blobs.
+ value_size = Column(Integer, nullable=False)
+ codec = Column(String(255), nullable=False, default="json")
+ is_encrypted = Column(Boolean, nullable=False, default=False)
+
+ __table_args__ = (
+ # Unique constraint prevents duplicate rows from concurrent writes
+ UniqueConstraint(
+ "extension_id",
+ "user_fk",
+ "resource_type",
+ "resource_uuid",
+ "key",
+ name="uq_extension_storage_scoped_key",
+ ),
Review Comment:
Not changing the schema β partial/filtered unique indexes aren't portable to MySQL (no support at all), and sentinel values for `user_fk` would mean giving up its FK + `ondelete=SET NULL` auto-demotion-to-global-scope semantics. Concurrent writes to the same key are instead serialized by the `DistributedLock` in `ExtensionStorageDAO.set()`. The one real gap β `ExtensionStorageDAO.create()` bypassing that lock entirely β is closed separately: `create()` now raises `NotImplementedError` directing callers to `.set()`, since it never had upsert/validation/locking semantics to begin with.
##########
superset/extensions/storage/api.py:
##########
@@ -0,0 +1,738 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""
+REST API for extension storage.
+
+Provides HTTP endpoints for frontend extensions to access server-side
+ephemeral storage without direct backend code.
+
+All operations are user-scoped by default. Use `?shared=true` query param
+to access shared state visible to all users.
+"""
+
+from __future__ import annotations
+
+import base64
+from typing import Any
+
+from flask import g, request
+from flask.wrappers import Response
+from flask_appbuilder.api import BaseApi, expose, protect, safe
+
+from superset.extensions.storage.codecs import DEFAULT_CODEC, get_codec, SAFE_CODECS
+from superset.extensions.storage.ephemeral_dao import (
+ ExtensionEphemeralDAO,
+ ExtensionEphemeralTTLInvalid,
+ ExtensionEphemeralValueTooLarge,
+)
+from superset.extensions.storage.persistent_dao import (
+ ExtensionStorageDAO,
+ ExtensionStorageKeyTooLong,
+ ExtensionStorageListPayloadTooLarge,
+ ExtensionStorageQuotaExceeded,
+ ExtensionStorageValueTooLarge,
+)
+from superset.extensions.storage.utils import get_extension_or_404, parse_ttl
+from superset.key_value.exceptions import KeyValueCodecEncodeException
+from superset.utils.decorators import transaction
+
+
+def _decoded_result_for_wire(decoded: Any) -> tuple[Any, bool]:
+ """Convert a codec-decoded value into its JSON wire representation.
+
+ JSON has no byte type, so a raw `bytes` value is base64-encoded to a
+ string for the response and flagged as such; every other value is
+ already JSON-representable as-is. Checked on the decoded value's
+ actual type, not the codec's name, so this keeps working for a codec
+ this module didn't define.
+
+ :returns: (wire_value, is_binary)
+ """
+ if isinstance(decoded, bytes):
+ return base64.b64encode(decoded).decode("ascii"), True
+ return decoded, False
+
+
+def _wire_value_for_request(value: Any, is_binary: bool) -> Any:
+ """Convert a request body's JSON `value` into a codec's input type.
+
+ `is_binary` is an explicit flag from the caller: JSON has no byte
+ type, so there is no way to tell, from the JSON `value` alone,
+ whether it is a base64 string that must be decoded to bytes before
+ being handed to the codec's `encode`, or a literal value to pass
+ through as-is. Only the caller knows which it sent.
+
+ :raises ValueError: if `is_binary` is set and `value` is not a valid
+ base64 string.
+ """
+ if is_binary:
+ return base64.b64decode(value, validate=True)
+ return value
+
+
+class ExtensionStorageRestApi(BaseApi):
+ """REST API for extension ephemeral (Tier 2) and persistent (Tier 3) storage."""
+
+ allow_browser_login = True
+ route_base = "/api/v1/extensions"
+
+ def response(self, status_code: int, **kwargs: Any) -> Response:
+ """Helper method to create JSON responses."""
+ from flask import jsonify
+
+ return jsonify(kwargs), status_code
+
+ def response_404(self, message: str = "Not found") -> Response:
+ """Helper method to create 404 responses."""
+ from flask import jsonify
+
+ return jsonify({"message": message}), 404
+
+ def response_400(self, message: str) -> Response:
+ """Helper method to create 400 responses."""
+ from flask import jsonify
+
+ return jsonify({"message": message}), 400
+
+ @protect()
+ @safe
+ @expose("/<publisher>/<name>/storage/ephemeral/<key>", methods=("GET",))
+ def get_ephemeral(
+ self, publisher: str, name: str, key: str, **kwargs: Any
+ ) -> Response:
+ """Get a value from ephemeral state.
+ ---
+ get:
+ summary: Get a value from ephemeral state
+ parameters:
+ - in: path
+ name: publisher
+ schema:
+ type: string
+ required: true
+ description: Extension publisher
+ - in: path
+ name: name
+ schema:
+ type: string
+ required: true
+ description: Extension name
+ - in: path
+ name: key
+ schema:
+ type: string
+ required: true
+ description: Storage key
+ - in: query
+ name: shared
+ schema:
+ type: boolean
+ required: false
+ description: If true, read from shared state visible to all users
+ responses:
+ 200:
+ description: Value retrieved successfully
+ content:
+ application/json:
+ schema:
+ type: object
+ properties:
+ result:
+ description: The stored value. When 'isBinary' is
+ true, this is a base64 string that must be
+ decoded to get the actual value.
+ codec:
+ type: string
+ description: Name of the codec 'result' was encoded
+ with, e.g. "json" (default) or "binary"
+ isBinary:
+ type: boolean
+ description: Whether the stored value is binary
+ data
+ 400:
+ description: Value was stored with a codec unavailable over the API
+ 404:
+ description: Extension not found
+ """
+ extension_id = f"{publisher}.{name}"
+ extension = get_extension_or_404(extension_id)
+ if not extension:
+ return self.response_404("Extension not found")
+
+ shared = request.args.get("shared", "false").lower() == "true"
+ raw = ExtensionEphemeralDAO.get_raw(extension_id, key, shared=shared)
+ if raw is None:
+ return self.response(200, result=None)
+ value, codec = raw
+ if codec not in SAFE_CODECS:
+ return self.response_400(
+ f"Value was stored with codec '{codec}', which cannot be "
+ "read over the REST API."
+ )
+
+ result, is_binary = _decoded_result_for_wire(get_codec(codec).decode(value))
+ return self.response(200, result=result, codec=codec, isBinary=is_binary)
+
+ @protect()
+ @safe
+ @expose("/<publisher>/<name>/storage/ephemeral/<key>", methods=("PUT",))
+ def set_ephemeral(
+ self, publisher: str, name: str, key: str, **kwargs: Any
+ ) -> Response:
+ """Set a value in ephemeral state.
+ ---
+ put:
+ summary: Set a value in ephemeral state
+ parameters:
+ - in: path
+ name: publisher
+ schema:
+ type: string
+ required: true
+ description: Extension publisher
+ - in: path
+ name: name
+ schema:
+ type: string
+ required: true
+ description: Extension name
+ - in: path
+ name: key
+ schema:
+ type: string
+ required: true
+ description: Storage key
+ - in: query
+ name: shared
+ schema:
+ type: boolean
+ required: false
+ description: If true, store as shared state visible to all users
+ requestBody:
+ required: true
+ content:
+ application/json:
+ schema:
+ type: object
+ required:
+ - value
+ - ttl
+ properties:
+ value:
+ description: The value to store (must not exceed
+ MAX_VALUE_SIZE bytes once encoded with 'codec').
+ When 'isBinary' is true, this must be a base64
+ string, decoded before being handed to 'codec'.
+ codec:
+ type: string
+ description: Name of the codec used to encode 'value',
+ e.g. "json" (default). Must be one of the codecs
+ allowed over the REST API.
+ isBinary:
+ type: boolean
+ description: Whether 'value' is binary data
+ ttl:
+ type: integer
+ description: Time-to-live in seconds (must be a positive
+ integer not exceeding MAX_TTL)
+ responses:
+ 200:
+ description: Value stored successfully
+ 400:
+ description: Invalid request body, or codec not allowed over the API
+ 404:
+ description: Extension not found
+ """
+ extension_id = f"{publisher}.{name}"
+ extension = get_extension_or_404(extension_id)
+ if not extension:
+ return self.response_404("Extension not found")
+
+ body = request.get_json(silent=True) or {}
+ if "value" not in body:
+ return self.response_400("Request body must contain 'value' field")
+
+ codec = body.get("codec", DEFAULT_CODEC)
+ if not isinstance(codec, str) or codec not in SAFE_CODECS:
+ return self.response_400(
+ f"Codec '{codec}' is not allowed over the REST API."
+ )
+
+ is_binary = bool(body.get("isBinary", False))
+ try:
+ value = _wire_value_for_request(body["value"], is_binary)
+ except (ValueError, TypeError):
+ return self.response_400(
+ "Value must be a valid base64 string when 'isBinary' is true."
+ )
+ ttl, error = parse_ttl(body)
+ if error:
+ return self.response_400(error)
+
+ shared = request.args.get("shared", "false").lower() == "true"
+ try:
+ ExtensionEphemeralDAO.set(
+ extension_id, key, value, ttl, codec=codec, shared=shared
+ )
+ except (ExtensionEphemeralTTLInvalid, ExtensionEphemeralValueTooLarge) as ex:
+ return self.response_400(ex.message)
Review Comment:
Fixed β added a `KeyValueCodecEncodeException`/`TypeError`/`ValueError` catch here too, returning 400 instead of letting it 500, matching the persistent path's fix.
##########
superset/extensions/storage/api.py:
##########
@@ -0,0 +1,738 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""
+REST API for extension storage.
+
+Provides HTTP endpoints for frontend extensions to access server-side
+ephemeral storage without direct backend code.
+
+All operations are user-scoped by default. Use `?shared=true` query param
+to access shared state visible to all users.
+"""
+
+from __future__ import annotations
+
+import base64
+from typing import Any
+
+from flask import g, request
+from flask.wrappers import Response
+from flask_appbuilder.api import BaseApi, expose, protect, safe
+
+from superset.extensions.storage.codecs import DEFAULT_CODEC, get_codec, SAFE_CODECS
+from superset.extensions.storage.ephemeral_dao import (
+ ExtensionEphemeralDAO,
+ ExtensionEphemeralTTLInvalid,
+ ExtensionEphemeralValueTooLarge,
+)
+from superset.extensions.storage.persistent_dao import (
+ ExtensionStorageDAO,
+ ExtensionStorageKeyTooLong,
+ ExtensionStorageListPayloadTooLarge,
+ ExtensionStorageQuotaExceeded,
+ ExtensionStorageValueTooLarge,
+)
+from superset.extensions.storage.utils import get_extension_or_404, parse_ttl
+from superset.key_value.exceptions import KeyValueCodecEncodeException
+from superset.utils.decorators import transaction
+
+
+def _decoded_result_for_wire(decoded: Any) -> tuple[Any, bool]:
+ """Convert a codec-decoded value into its JSON wire representation.
+
+ JSON has no byte type, so a raw `bytes` value is base64-encoded to a
+ string for the response and flagged as such; every other value is
+ already JSON-representable as-is. Checked on the decoded value's
+ actual type, not the codec's name, so this keeps working for a codec
+ this module didn't define.
+
+ :returns: (wire_value, is_binary)
+ """
+ if isinstance(decoded, bytes):
+ return base64.b64encode(decoded).decode("ascii"), True
+ return decoded, False
+
+
+def _wire_value_for_request(value: Any, is_binary: bool) -> Any:
+ """Convert a request body's JSON `value` into a codec's input type.
+
+ `is_binary` is an explicit flag from the caller: JSON has no byte
+ type, so there is no way to tell, from the JSON `value` alone,
+ whether it is a base64 string that must be decoded to bytes before
+ being handed to the codec's `encode`, or a literal value to pass
+ through as-is. Only the caller knows which it sent.
+
+ :raises ValueError: if `is_binary` is set and `value` is not a valid
+ base64 string.
+ """
+ if is_binary:
+ return base64.b64decode(value, validate=True)
+ return value
+
+
+class ExtensionStorageRestApi(BaseApi):
+ """REST API for extension ephemeral (Tier 2) and persistent (Tier 3) storage."""
+
+ allow_browser_login = True
+ route_base = "/api/v1/extensions"
+
+ def response(self, status_code: int, **kwargs: Any) -> Response:
+ """Helper method to create JSON responses."""
+ from flask import jsonify
+
+ return jsonify(kwargs), status_code
+
+ def response_404(self, message: str = "Not found") -> Response:
+ """Helper method to create 404 responses."""
+ from flask import jsonify
+
+ return jsonify({"message": message}), 404
+
+ def response_400(self, message: str) -> Response:
+ """Helper method to create 400 responses."""
+ from flask import jsonify
+
+ return jsonify({"message": message}), 400
+
+ @protect()
+ @safe
+ @expose("/<publisher>/<name>/storage/ephemeral/<key>", methods=("GET",))
+ def get_ephemeral(
+ self, publisher: str, name: str, key: str, **kwargs: Any
+ ) -> Response:
+ """Get a value from ephemeral state.
+ ---
+ get:
+ summary: Get a value from ephemeral state
+ parameters:
+ - in: path
+ name: publisher
+ schema:
+ type: string
+ required: true
+ description: Extension publisher
+ - in: path
+ name: name
+ schema:
+ type: string
+ required: true
+ description: Extension name
+ - in: path
+ name: key
+ schema:
+ type: string
+ required: true
+ description: Storage key
+ - in: query
+ name: shared
+ schema:
+ type: boolean
+ required: false
+ description: If true, read from shared state visible to all users
+ responses:
+ 200:
+ description: Value retrieved successfully
+ content:
+ application/json:
+ schema:
+ type: object
+ properties:
+ result:
+ description: The stored value. When 'isBinary' is
+ true, this is a base64 string that must be
+ decoded to get the actual value.
+ codec:
+ type: string
+ description: Name of the codec 'result' was encoded
+ with, e.g. "json" (default) or "binary"
+ isBinary:
+ type: boolean
+ description: Whether the stored value is binary
+ data
+ 400:
+ description: Value was stored with a codec unavailable over the API
+ 404:
+ description: Extension not found
+ """
+ extension_id = f"{publisher}.{name}"
+ extension = get_extension_or_404(extension_id)
+ if not extension:
+ return self.response_404("Extension not found")
+
+ shared = request.args.get("shared", "false").lower() == "true"
+ raw = ExtensionEphemeralDAO.get_raw(extension_id, key, shared=shared)
+ if raw is None:
+ return self.response(200, result=None)
+ value, codec = raw
+ if codec not in SAFE_CODECS:
+ return self.response_400(
+ f"Value was stored with codec '{codec}', which cannot be "
+ "read over the REST API."
+ )
+
+ result, is_binary = _decoded_result_for_wire(get_codec(codec).decode(value))
+ return self.response(200, result=result, codec=codec, isBinary=is_binary)
+
+ @protect()
+ @safe
+ @expose("/<publisher>/<name>/storage/ephemeral/<key>", methods=("PUT",))
+ def set_ephemeral(
+ self, publisher: str, name: str, key: str, **kwargs: Any
+ ) -> Response:
+ """Set a value in ephemeral state.
+ ---
+ put:
+ summary: Set a value in ephemeral state
+ parameters:
+ - in: path
+ name: publisher
+ schema:
+ type: string
+ required: true
+ description: Extension publisher
+ - in: path
+ name: name
+ schema:
+ type: string
+ required: true
+ description: Extension name
+ - in: path
+ name: key
+ schema:
+ type: string
+ required: true
+ description: Storage key
+ - in: query
+ name: shared
+ schema:
+ type: boolean
+ required: false
+ description: If true, store as shared state visible to all users
+ requestBody:
+ required: true
+ content:
+ application/json:
+ schema:
+ type: object
+ required:
+ - value
+ - ttl
+ properties:
+ value:
+ description: The value to store (must not exceed
+ MAX_VALUE_SIZE bytes once encoded with 'codec').
+ When 'isBinary' is true, this must be a base64
+ string, decoded before being handed to 'codec'.
+ codec:
+ type: string
+ description: Name of the codec used to encode 'value',
+ e.g. "json" (default). Must be one of the codecs
+ allowed over the REST API.
+ isBinary:
+ type: boolean
+ description: Whether 'value' is binary data
+ ttl:
+ type: integer
+ description: Time-to-live in seconds (must be a positive
+ integer not exceeding MAX_TTL)
+ responses:
+ 200:
+ description: Value stored successfully
+ 400:
+ description: Invalid request body, or codec not allowed over the API
+ 404:
+ description: Extension not found
+ """
+ extension_id = f"{publisher}.{name}"
+ extension = get_extension_or_404(extension_id)
+ if not extension:
+ return self.response_404("Extension not found")
+
+ body = request.get_json(silent=True) or {}
+ if "value" not in body:
+ return self.response_400("Request body must contain 'value' field")
+
+ codec = body.get("codec", DEFAULT_CODEC)
+ if not isinstance(codec, str) or codec not in SAFE_CODECS:
+ return self.response_400(
+ f"Codec '{codec}' is not allowed over the REST API."
+ )
+
+ is_binary = bool(body.get("isBinary", False))
+ try:
+ value = _wire_value_for_request(body["value"], is_binary)
+ except (ValueError, TypeError):
+ return self.response_400(
+ "Value must be a valid base64 string when 'isBinary' is true."
+ )
+ ttl, error = parse_ttl(body)
+ if error:
+ return self.response_400(error)
+
+ shared = request.args.get("shared", "false").lower() == "true"
+ try:
+ ExtensionEphemeralDAO.set(
+ extension_id, key, value, ttl, codec=codec, shared=shared
+ )
+ except (ExtensionEphemeralTTLInvalid, ExtensionEphemeralValueTooLarge) as ex:
+ return self.response_400(ex.message)
+
+ return self.response(200, message="Value stored successfully")
+
+ @protect()
+ @safe
+ @expose("/<publisher>/<name>/storage/ephemeral/<key>", methods=("DELETE",))
+ def delete_ephemeral(
+ self, publisher: str, name: str, key: str, **kwargs: Any
+ ) -> Response:
+ """Delete a value from ephemeral state.
+ ---
+ delete:
+ summary: Delete a value from ephemeral state
+ parameters:
+ - in: path
+ name: publisher
+ schema:
+ type: string
+ required: true
+ description: Extension publisher
+ - in: path
+ name: name
+ schema:
+ type: string
+ required: true
+ description: Extension name
+ - in: path
+ name: key
+ schema:
+ type: string
+ required: true
+ description: Storage key
+ - in: query
+ name: shared
+ schema:
+ type: boolean
+ required: false
+ description: If true, delete from shared state
+ responses:
+ 200:
+ description: Value deleted successfully
+ 404:
+ description: Extension not found
+ """
+ extension_id = f"{publisher}.{name}"
+ extension = get_extension_or_404(extension_id)
+ if not extension:
+ return self.response_404("Extension not found")
+
+ shared = request.args.get("shared", "false").lower() == "true"
+ ExtensionEphemeralDAO.delete(extension_id, key, shared=shared)
+
+ return self.response(200, message="Value deleted successfully")
+
+ @protect()
+ @safe
+ @expose("/<publisher>/<name>/storage/persistent", methods=("GET",))
+ def list_persistent(self, publisher: str, name: str, **kwargs: Any) -> Response:
+ """List entries in persistent state.
+ ---
+ get:
+ summary: List entries in persistent state
+ parameters:
+ - in: path
+ name: publisher
+ schema:
+ type: string
+ required: true
+ description: Extension publisher
+ - in: path
+ name: name
+ schema:
+ type: string
+ required: true
+ description: Extension name
+ - in: query
+ name: shared
+ schema:
+ type: boolean
+ required: false
+ description: If true, list shared state visible to all users
+ - in: query
+ name: resource_type
+ schema:
+ type: string
+ required: false
+ description: Filter by resource type
+ - in: query
+ name: resource_uuid
+ schema:
+ type: string
+ required: false
+ description: Filter by resource UUID (requires resource_type)
+ - in: query
+ name: page
+ schema:
+ type: integer
+ required: false
+ description: Zero-indexed page number. Defaults to 0.
+ - in: query
+ name: page_size
+ schema:
+ type: integer
+ required: false
+ description: Number of entries per page. Defaults to 10. There
+ is no fixed ceiling, but a page whose combined value size
+ exceeds MAX_LIST_PAYLOAD_SIZE is rejected β reduce page_size
+ and retry if that happens.
+ responses:
+ 200:
+ description: Entries retrieved successfully
+ content:
+ application/json:
+ schema:
+ type: object
+ properties:
+ result:
+ type: array
+ items:
+ type: object
+ properties:
+ key:
+ type: string
+ value:
+ description: The stored value, or null if its
+ codec cannot be read over the REST API.
+ When 'isBinary' is true, this is a base64
+ string that must be decoded to get the
+ actual value.
+ codec:
+ type: string
+ isBinary:
+ type: boolean
+ description: Whether the stored value is
+ binary data
+ count:
+ type: integer
+ description: Total number of entries matching the
+ given scope/filters, across all pages
+ 400:
+ description: The requested page's combined value size exceeds
+ MAX_LIST_PAYLOAD_SIZE
+ 404:
+ description: Extension not found
+ """
+ extension_id = f"{publisher}.{name}"
+ extension = get_extension_or_404(extension_id)
+ if not extension:
+ return self.response_404("Extension not found")
+
+ shared = request.args.get("shared", "false").lower() == "true"
+ user_fk = None if shared else g.user.id
+ resource_type = request.args.get("resource_type")
+ resource_uuid = request.args.get("resource_uuid")
+ try:
+ page = int(request.args.get("page", 0))
+ page_size = int(request.args.get("page_size", 10))
+ except (TypeError, ValueError):
+ return self.response_400("'page' and 'page_size' must be integers")
+
+ try:
+ entries, count = ExtensionStorageDAO.list_entries(
+ extension_id,
+ user_fk=user_fk,
+ resource_type=resource_type,
+ resource_uuid=resource_uuid,
+ page=page,
+ page_size=page_size,
+ )
+ except ExtensionStorageListPayloadTooLarge as ex:
+ return self.response(ex.status, message=ex.message)
+
+ result = []
+ for entry in entries:
+ value: Any = None
+ is_binary = False
+ if entry.codec in SAFE_CODECS and entry.value is not None:
+ decoded = get_codec(entry.codec).decode(entry.value)
+ value, is_binary = _decoded_result_for_wire(decoded)
+ result.append(
+ {
+ "key": entry.key,
+ "value": value,
+ "codec": entry.codec,
+ "isBinary": is_binary,
+ }
+ )
+
+ return self.response(200, result=result, count=count)
+
+ @protect()
+ @safe
+ @expose("/<publisher>/<name>/storage/persistent/<key>", methods=("GET",))
+ def get_persistent(
+ self, publisher: str, name: str, key: str, **kwargs: Any
+ ) -> Response:
+ """Get a value from persistent state.
+ ---
+ get:
+ summary: Get a value from persistent state
+ parameters:
+ - in: path
+ name: publisher
+ schema:
+ type: string
+ required: true
+ description: Extension publisher
+ - in: path
+ name: name
+ schema:
+ type: string
+ required: true
+ description: Extension name
+ - in: path
+ name: key
+ schema:
+ type: string
+ required: true
+ description: Storage key
+ - in: query
+ name: shared
+ schema:
+ type: boolean
+ required: false
+ description: If true, read from shared state visible to all users
+ responses:
+ 200:
+ description: Value retrieved successfully
+ content:
+ application/json:
+ schema:
+ type: object
+ properties:
+ result:
+ description: The stored value. When 'isBinary' is
+ true, this is a base64 string that must be
+ decoded to get the actual value.
+ codec:
+ type: string
+ description: Name of the codec 'result' was encoded
+ with, e.g. "json" (default) or "binary"
+ isBinary:
+ type: boolean
+ description: Whether the stored value is binary
+ data
+ 400:
+ description: Value was stored with a codec unavailable over the API
+ 404:
+ description: Extension not found
+ """
+ extension_id = f"{publisher}.{name}"
+ extension = get_extension_or_404(extension_id)
+ if not extension:
+ return self.response_404("Extension not found")
+
+ shared = request.args.get("shared", "false").lower() == "true"
+ user_fk = None if shared else g.user.id
+ entry = ExtensionStorageDAO.get(extension_id, key, user_fk=user_fk)
+ if entry is None:
+ return self.response(200, result=None)
+ if entry.codec not in SAFE_CODECS:
+ return self.response_400(
+ f"Value was stored with codec '{entry.codec}', which cannot be "
+ "read over the REST API."
+ )
+
+ decoded = ExtensionStorageDAO.get_decoded_value(
+ extension_id, key, user_fk=user_fk
+ )
Review Comment:
Fixed β added `ExtensionStorageDAO.decode_entry()` to decode the already-fetched entry directly instead of re-querying, so the codec safety check and the decode always see the same row.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(extensions): add a Storage API for extensions [superset]
Posted by "matebenyovszky (via GitHub)" <gi...@apache.org>.
matebenyovszky commented on PR #39171:
URL: https://github.com/apache/superset/pull/39171#issuecomment-4834940373
> > our [Vambery AI Agent](https://github.com/integrityauthority/superset-extensions) extension
>
>
>
> @matebenyovszky I'm assuming you are aware of [[SIP-214] Introducing a Chat extension in Superset](https://github.com/apache/superset/issues/41059)? If not, it might be useful for your extension.
Great, haven't seen it but will check tomorrow! Thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(extensions): add a Storage API for extensions [superset]
Posted by "michael-s-molina (via GitHub)" <gi...@apache.org>.
michael-s-molina commented on PR #39171:
URL: https://github.com/apache/superset/pull/39171#issuecomment-4833571667
> our [Vambery AI Agent](https://github.com/integrityauthority/superset-extensions) extension
@matebenyovszky I'm assuming you are aware of [[SIP-214] Introducing a Chat extension in Superset](https://github.com/apache/superset/issues/41059)? If not, it might be useful for your extension.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(extensions): add a Storage API for extensions [superset]
Posted by "michael-s-molina (via GitHub)" <gi...@apache.org>.
michael-s-molina commented on code in PR #39171:
URL: https://github.com/apache/superset/pull/39171#discussion_r3561001323
##########
superset/extensions/storage/persistent.py:
##########
@@ -0,0 +1,193 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""
+Host implementation for Persistent State (Tier 3 Storage).
+
+Provides the concrete database-backed implementation that is injected into
+superset_core.extensions.storage.persistent at startup.
+"""
+
+from __future__ import annotations
+
+from typing import Any, ClassVar
+
+from flask import g
+from superset_core.extensions.storage.persistent import (
+ PersistentSetOptions,
+ PersistentState as CorePersistentState,
+)
+
+from superset.extensions.context import get_current_extension_context
+from superset.extensions.storage.persistent_dao import ExtensionStorageDAO
+from superset.utils import json
+from superset.utils.decorators import transaction
+
+
+def _get_extension_id() -> str:
+ """Get the current extension ID from context."""
+ context = get_current_extension_context()
+ if context is None:
+ raise RuntimeError(
+ "persistent_state can only be used within an extension context. "
+ "Ensure this code is being executed during extension loading or "
+ "within an extension API request handler."
+ )
+ return context.manifest.id
+
+
+def _get_current_user_id() -> int:
+ """Get the current authenticated user's ID."""
+ user = getattr(g, "user", None)
+ if user is None or not hasattr(user, "id"):
+ raise RuntimeError(
+ "persistent_state requires an authenticated user. "
+ "Ensure the request has been authenticated."
+ )
+ return user.id
+
+
+def _decode(raw: bytes | None) -> Any:
+ """Decode stored bytes back to a Python value."""
+ if raw is None:
+ return None
+ return json.loads(raw)
+
+
+def _encode(value: Any) -> bytes:
+ """Encode a Python value for database storage."""
+ return json.dumps(value).encode()
Review Comment:
Added support for codecs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(extensions): add a Storage API for extensions [superset]
Posted by "matebenyovszky (via GitHub)" <gi...@apache.org>.
matebenyovszky commented on PR #39171:
URL: https://github.com/apache/superset/pull/39171#issuecomment-4382444744
Hey @michael-s-molina @villebro @rusackas @geido @kgabryje π
We at [Integrity Authority Hungary](https://github.com/integrityauthority) are actively building a Superset extension β **Vambery AI Assistant** (see our [superset-extensions](https://github.com/integrityauthority/superset-extensions) repo) β and the Storage API from this PR is exactly what we need for persisting conversation state, user preferences, and AI context across sessions.
We're ready to adopt this as soon as it lands. While reviewing the code to prepare for integration, I found a **breaking bug** (see my comment above): the frontend URL builder wasn't updated to match the restructured backend routes from `f08bcd4`. I've prepared a complete patch (bug fix + unit tests for all 3 storage modules) that can be applied in one command:
`ash
curl -sL https://gist.githubusercontent.com/matebenyovszky/f64b21f895a34289171fd18fe381e85b/raw/fix-storage-urls-and-add-tests.patch | git am
`
**What the patch fixes:**
- π Frontend `ephemeralState.ts` and `persistentState.ts` URL paths β aligned with backend `/<publisher>/<name>/storage/...` routes
- β
Added ~490 lines of unit tests for `localState`, `ephemeralState`, and `persistentState` (should help with the codecov check)
**What we think is still needed to unblock this PR:**
1. Apply the patch above (or I can open a PR against your branch)
2. Mark as "Ready for Review" (currently draft)
3. @villebro β your earlier comments have all been addressed by Michael, would be great if you could re-review!
This is a really well-designed module β the tiered storage approach with automatic namespace isolation is exactly the right pattern for extensions. Would love to see it merged soon! π
cc @aminghadersohi @betodealmeida
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(extensions): add a Storage API for extensions [superset]
Posted by "rusackas (via GitHub)" <gi...@apache.org>.
rusackas commented on PR #39171:
URL: https://github.com/apache/superset/pull/39171#issuecomment-4442973534
I targeted [this PR](https://github.com/michael-s-molina/superset/pull/774) to your feature branch, but it's a bit orthogonal... it's stuff needed to build the extension I'm working on which uses the storage mechanisms herein.
I can retarget that to `master` after this merges, though it's weirdly on your fork so I can stack onto this feature branch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(extensions): add a Storage API for extensions [superset]
Posted by "rusackas (via GitHub)" <gi...@apache.org>.
rusackas commented on PR #39171:
URL: https://github.com/apache/superset/pull/39171#issuecomment-4443017717
@michael-s-molina happy to attempt a rebase here if you'd like, which might clear up codecov. Also happy to review @matebenyovszky 's PR with you if/when time allows. Also, @matebenyovszky if you want to join the Extensions sync we have on a weekly basis (time zone permitting) you seem well-qualified! :D
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(extensions): add a Storage API for extensions [superset]
Posted by "matebenyovszky (via GitHub)" <gi...@apache.org>.
matebenyovszky commented on PR #39171:
URL: https://github.com/apache/superset/pull/39171#issuecomment-4378900702
## π Bug: Frontend storage URLs don't match backend API routes
Hi @michael-s-molina, great work on this PR! We're eager to use the Storage API for our extension work at [Integrity Authority Hungary](https://github.com/integrityauthority).
While reviewing, I found a **breaking bug** introduced in commit `f08bcd4` ("Updates the API"):
### Problem
The backend API routes were restructured to use the `/{publisher}/{name}/storage/{tier}/{key}` pattern:
`python
# Backend (api.py line 113)
@expose("/<publisher>/<name>/storage/ephemeral/<key>", methods=("GET",))
# Results in: /api/v1/extensions/{publisher}/{name}/storage/ephemeral/{key}
`
But the frontend `ephemeralState.ts` and `persistentState.ts` still use the **old** URL pattern:
` ypescript
// Frontend (ephemeralState.ts line 30)
const basePath = '/api/v1/extensions/storage/ephemeral';
const url = `//`;
// Results in: /api/v1/extensions/storage/ephemeral/{extensionId}/{key}
`
**All frontend storage API calls will 404** because the URL patterns don't match.
### Fix
Split `extensionId` (format: `{publisher}.{name}`) on the first dot and build URLs using `/{publisher}/{name}/storage/...`:
` ypescript
const dotIndex = extensionId.indexOf('.');
const publisher = encodeURIComponent(extensionId.slice(0, dotIndex));
const name = encodeURIComponent(extensionId.slice(dotIndex + 1));
const encodedKey = encodeURIComponent(key);
const url = `/api/v1/extensions///storage/ephemeral/`;
`
### Additional: Missing unit tests
I also wrote unit tests for all three storage modules (`localState`, `ephemeralState`, `persistentState`) which should help with the `codecov/project/core-packages-ts` check failure.
### Patch
I've prepared a complete patch with the fix + tests (5 files, +495/-6 lines). Happy to provide it as a PR against your branch or as a downloadable patch file β let me know what works best!
**Summary of changes:**
- `ephemeralState.ts` β fix URL pattern to match backend `/{publisher}/{name}/storage/ephemeral/{key}`
- `persistentState.ts` β fix URL pattern to match backend `/{publisher}/{name}/storage/persistent/{key}`
- `localState.test.ts` β tests for createBrowserStorage (user-scoped, shared, isolation)
- `ephemeralState.test.ts` β tests for createEphemeralState (CRUD, shared, URL encoding)
- `persistentState.test.ts` β tests for createPersistentState (CRUD, shared, URL encoding, key length)
Ping me if you'd like me to open a PR against your branch! π
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(extensions): add Tier 1 and Tier 2 storage APIs [superset]
Posted by "michael-s-molina (via GitHub)" <gi...@apache.org>.
michael-s-molina commented on code in PR #39171:
URL: https://github.com/apache/superset/pull/39171#discussion_r3058602085
##########
superset/extensions/storage/api.py:
##########
@@ -0,0 +1,265 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""
+REST API for extension storage.
+
+Provides HTTP endpoints for frontend extensions to access server-side
+ephemeral storage without direct backend code.
+
+All operations are user-scoped by default. Use `?shared=true` query param
+to access shared state visible to all users.
+"""
+
+from __future__ import annotations
+
+from typing import Any
+
+from flask import g, request
+from flask.wrappers import Response
+from flask_appbuilder.api import BaseApi, expose, protect, safe
+
+from superset.extensions import cache_manager
+from superset.extensions.types import LoadedExtension
+from superset.extensions.utils import get_extensions
+
+# Key separator
+SEPARATOR = ":"
Review Comment:
Memcached explicitly forbids whitespace and control characters in keys. Since `CACHE_TYPE` is configurable and Memcached is a common Superset cache backend, a `\n` separator would silently break those deployments.
The actual protection against key injection is already handled at the boundaries β the API endpoint URL-encodes `extension_id` and `key` via `encodeURIComponent` on the frontend, and Flask's routing percent-decodes the path params before they reach `_build_storage_key`. So the components arriving at `_build_cache_key` are already sanitized.
Given the existing boundary encoding, `:` seems fine and is the established convention for namespaced cache keys in Superset and elsewhere.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(extensions): add Tier 1 and Tier 2 storage APIs [superset]
Posted by "michael-s-molina (via GitHub)" <gi...@apache.org>.
michael-s-molina commented on code in PR #39171:
URL: https://github.com/apache/superset/pull/39171#discussion_r3058602085
##########
superset/extensions/storage/api.py:
##########
@@ -0,0 +1,265 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""
+REST API for extension storage.
+
+Provides HTTP endpoints for frontend extensions to access server-side
+ephemeral storage without direct backend code.
+
+All operations are user-scoped by default. Use `?shared=true` query param
+to access shared state visible to all users.
+"""
+
+from __future__ import annotations
+
+from typing import Any
+
+from flask import g, request
+from flask.wrappers import Response
+from flask_appbuilder.api import BaseApi, expose, protect, safe
+
+from superset.extensions import cache_manager
+from superset.extensions.types import LoadedExtension
+from superset.extensions.utils import get_extensions
+
+# Key separator
+SEPARATOR = ":"
Review Comment:
Memcached explicitly forbids whitespace and control characters in keys. Since `CACHE_TYPE` is configurable and Memcached is a common Superset cache backend, a `\n` separator would silently break those deployments.
The actual protection against key injection is already handled at the boundaries β the API endpoint URL-encodes `extension_id` and `key` via `encodeURIComponent` on the frontend, and Flask's routing percent-decodes the path params before they reach `_build_storage_key`. So the components arriving at `_build_cache_key` are already sanitized.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(extensions): add Tier 1 and Tier 2 storage APIs [superset]
Posted by "netlify[bot] (via GitHub)" <gi...@apache.org>.
netlify[bot] commented on PR #39171:
URL: https://github.com/apache/superset/pull/39171#issuecomment-4201634427
### <span aria-hidden="true">β
</span> Deploy Preview for *superset-docs-preview* ready!
| Name | Link |
|:-:|------------------------|
|<span aria-hidden="true">π¨</span> Latest commit | bfcc05a56f030b49fccf2ba9102718f19016d285 |
|<span aria-hidden="true">π</span> Latest deploy log | https://app.netlify.com/projects/superset-docs-preview/deploys/69d55941489a6e000887dd57 |
|<span aria-hidden="true">π</span> Deploy Preview | [https://deploy-preview-39171--superset-docs-preview.netlify.app](https://deploy-preview-39171--superset-docs-preview.netlify.app) |
|<span aria-hidden="true">π±</span> Preview on mobile | <details><summary> Toggle QR Code... </summary><br /><br /><br /><br />_Use your smartphone camera to open QR code link._</details> |
---
<!-- [superset-docs-preview Preview](https://deploy-preview-39171--superset-docs-preview.netlify.app) -->
_To edit notification comments on pull requests, go to your [Netlify project configuration](https://app.netlify.com/projects/superset-docs-preview/configuration/notifications#deploy-notifications)._
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(extensions): add Tier 1 and Tier 2 storage APIs [superset]
Posted by "netlify[bot] (via GitHub)" <gi...@apache.org>.
netlify[bot] commented on PR #39171:
URL: https://github.com/apache/superset/pull/39171#issuecomment-4209493289
### <span aria-hidden="true">β
</span> Deploy Preview for *superset-docs-preview* ready!
| Name | Link |
|:-:|------------------------|
|<span aria-hidden="true">π¨</span> Latest commit | 2c00c8eef9919b027959b3dff810266ff3d0e5c5 |
|<span aria-hidden="true">π</span> Latest deploy log | https://app.netlify.com/projects/superset-docs-preview/deploys/69d6bcad8fb0af00085a6e90 |
|<span aria-hidden="true">π</span> Deploy Preview | [https://deploy-preview-39171--superset-docs-preview.netlify.app](https://deploy-preview-39171--superset-docs-preview.netlify.app) |
|<span aria-hidden="true">π±</span> Preview on mobile | <details><summary> Toggle QR Code... </summary><br /><br /><br /><br />_Use your smartphone camera to open QR code link._</details> |
---
<!-- [superset-docs-preview Preview](https://deploy-preview-39171--superset-docs-preview.netlify.app) -->
_To edit notification comments on pull requests, go to your [Netlify project configuration](https://app.netlify.com/projects/superset-docs-preview/configuration/notifications#deploy-notifications)._
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(extensions): add a Storage API for extensions [superset]
Posted by "matebenyovszky (via GitHub)" <gi...@apache.org>.
matebenyovszky commented on PR #39171:
URL: https://github.com/apache/superset/pull/39171#issuecomment-4833550025
@rusackas β hey! Following up on your offer from May 13:
> *happy to attempt a rebase here if you'd like, which might clear up codecov. Also happy to review @matebenyovszky's PR*
Great news: Michael already rebased and merged our fixes into the branch. **All CI is green except one trivial `prettier` formatting issue** β I've opened a 1-file fix PR (https://github.com/michael-s-molina/superset/pull/811).
Once that tiny fix lands and the PR is marked Ready for Review, it just needs a reviewer approval. Would you be able to review and/or help push this across the finish line?
The Storage API is a foundational piece β our [Vambery AI Agent](https://github.com/integrityauthority/superset-extensions) extension needs it for conversation persistence, and I imagine your SQL Lab extension work (the contribution surfaces PR you filed) will benefit from it too.
Thanks! π
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(extensions): add a Storage API for extensions [superset]
Posted by "michael-s-molina (via GitHub)" <gi...@apache.org>.
michael-s-molina commented on code in PR #39171:
URL: https://github.com/apache/superset/pull/39171#discussion_r3561372837
##########
superset-frontend/src/extensions/ExtensionsLoader.test.ts:
##########
@@ -48,9 +47,53 @@ function createMockExtension(overrides: Partial<Extension> = {}): Extension {
};
}
+// Simulate what real webpack does when __webpack_init_sharing__ runs: it
+// registers the host's own eagerly-provided @apache-superset/core versions
+// into the share scope.
+function mockWebpackSharing(coreVersions: Record<string, unknown> = {}) {
+ (globalThis as any).__webpack_init_sharing__ = jest
Review Comment:
Fixed β replaced the `any` cast with the project's existing typed globals (`WebpackSharedScope`/`WebpackSharedScopeModule` from `webpackFederation.d.ts`).
##########
superset-frontend/src/extensions/ExtensionsLoader.test.ts:
##########
@@ -48,9 +47,53 @@ function createMockExtension(overrides: Partial<Extension> = {}): Extension {
};
}
+// Simulate what real webpack does when __webpack_init_sharing__ runs: it
+// registers the host's own eagerly-provided @apache-superset/core versions
+// into the share scope.
+function mockWebpackSharing(coreVersions: Record<string, unknown> = {}) {
+ (globalThis as any).__webpack_init_sharing__ = jest
+ .fn()
+ .mockImplementation(() => {
+ (globalThis as any).__webpack_share_scopes__.default[
+ '@apache-superset/core'
+ ] = coreVersions;
+ });
+ (globalThis as any).__webpack_share_scopes__ = { default: {} };
Review Comment:
Fixed β replaced the `any` cast with the project's existing typed globals (`WebpackSharedScope`/`WebpackSharedScopeModule` from `webpackFederation.d.ts`).
##########
superset-frontend/src/extensions/ExtensionsLoader.test.ts:
##########
@@ -48,9 +47,53 @@ function createMockExtension(overrides: Partial<Extension> = {}): Extension {
};
}
+// Simulate what real webpack does when __webpack_init_sharing__ runs: it
+// registers the host's own eagerly-provided @apache-superset/core versions
+// into the share scope.
+function mockWebpackSharing(coreVersions: Record<string, unknown> = {}) {
+ (globalThis as any).__webpack_init_sharing__ = jest
+ .fn()
+ .mockImplementation(() => {
+ (globalThis as any).__webpack_share_scopes__.default[
+ '@apache-superset/core'
+ ] = coreVersions;
+ });
+ (globalThis as any).__webpack_share_scopes__ = { default: {} };
+}
+
+function cleanupWebpackSharing() {
+ delete (globalThis as any).__webpack_init_sharing__;
+ delete (globalThis as any).__webpack_share_scopes__;
+}
+
+// Intercept script element creation so onload fires synchronously in jsdom.
+function mockRemoteEntryLoad() {
+ return jest
+ .spyOn(document.head, 'appendChild')
+ .mockImplementation((el: Node) => {
+ if (el instanceof HTMLScriptElement && el.onload) {
+ (el as HTMLScriptElement).onload!(new Event('load'));
+ }
+ return el;
+ });
+}
+
beforeEach(() => {
(ExtensionsLoader as any).instance = undefined;
mockApplicationRoot.mockReturnValue('');
+ // Reset window.superset to a base object before each test, including a
+ // shared singleton (commands) to verify identity is preserved across
+ // per-extension scoped copies.
+ (window as any).superset = {
Review Comment:
Fixed β replaced the `any` cast with the existing `Namespaces` type for `window.superset`.
##########
superset-frontend/src/extensions/ExtensionsLoader.test.ts:
##########
@@ -170,6 +213,212 @@ test('logs success after initializeExtensions completes', async () => {
infoSpy.mockRestore();
});
+test('logs partial failure count when some extensions fail to initialize', async () => {
+ const loader = ExtensionsLoader.getInstance();
+ const infoSpy = jest.spyOn(logging, 'info').mockImplementation();
+ jest.spyOn(SupersetClient, 'get').mockResolvedValue({
+ json: {
+ result: [
+ createMockExtension({ id: 'good-ext', remoteEntry: '' }),
+ createMockExtension({
+ id: 'broken-ext',
+ remoteEntry: 'http://broken-url/remoteEntry.js',
+ }),
+ ],
+ },
+ } as any);
+ const appendChildSpy = jest
+ .spyOn(document.head, 'appendChild')
+ .mockImplementation((element: Node) => {
+ if (element instanceof HTMLScriptElement && element.onerror) {
+ setTimeout(() => {
+ (element.onerror as any)('Script load error');
+ }, 0);
+ }
+ return element;
+ });
+
+ await loader.initializeExtensions();
+
+ expect(infoSpy).toHaveBeenCalledWith(
+ expect.stringContaining('1 of 2 extension(s) failing'),
+ );
+ expect(loader.getExtension('good-ext')).toBeDefined();
+ expect(loader.getExtension('broken-ext')).toBeUndefined();
+
+ infoSpy.mockRestore();
+ appendChildSpy.mockRestore();
+});
+
+test('each extension gets an isolated getContext via module federation custom scope', async () => {
+ const loader = ExtensionsLoader.getInstance();
+
+ // Capture the scoped @apache-superset/core instance each container receives
+ const capturedCoreModules: Array<any> = [];
Review Comment:
Fixed β replaced `Array<any>` with `Array<typeof import('@apache-superset/core')>`, matching the captured core module shape.
##########
superset-frontend/src/extensions/ExtensionsLoader.test.ts:
##########
@@ -170,6 +213,212 @@ test('logs success after initializeExtensions completes', async () => {
infoSpy.mockRestore();
});
+test('logs partial failure count when some extensions fail to initialize', async () => {
+ const loader = ExtensionsLoader.getInstance();
+ const infoSpy = jest.spyOn(logging, 'info').mockImplementation();
+ jest.spyOn(SupersetClient, 'get').mockResolvedValue({
+ json: {
+ result: [
+ createMockExtension({ id: 'good-ext', remoteEntry: '' }),
+ createMockExtension({
+ id: 'broken-ext',
+ remoteEntry: 'http://broken-url/remoteEntry.js',
+ }),
+ ],
+ },
+ } as any);
+ const appendChildSpy = jest
+ .spyOn(document.head, 'appendChild')
+ .mockImplementation((element: Node) => {
+ if (element instanceof HTMLScriptElement && element.onerror) {
+ setTimeout(() => {
+ (element.onerror as any)('Script load error');
+ }, 0);
+ }
+ return element;
+ });
+
+ await loader.initializeExtensions();
+
+ expect(infoSpy).toHaveBeenCalledWith(
+ expect.stringContaining('1 of 2 extension(s) failing'),
+ );
+ expect(loader.getExtension('good-ext')).toBeDefined();
+ expect(loader.getExtension('broken-ext')).toBeUndefined();
+
+ infoSpy.mockRestore();
+ appendChildSpy.mockRestore();
+});
+
+test('each extension gets an isolated getContext via module federation custom scope', async () => {
+ const loader = ExtensionsLoader.getInstance();
+
+ // Capture the scoped @apache-superset/core instance each container receives
+ const capturedCoreModules: Array<any> = [];
+
+ const makeContainer = () => ({
+ init: jest.fn().mockImplementation(async (scope: any) => {
Review Comment:
Fixed β typed `scope` with the existing `WebpackSharedScope` interface instead of `any`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(extensions): add a Storage API for extensions [superset]
Posted by "codeant-ai-for-open-source[bot] (via GitHub)" <gi...@apache.org>.
codeant-ai-for-open-source[bot] commented on code in PR #39171:
URL: https://github.com/apache/superset/pull/39171#discussion_r3622506970
##########
superset/extensions/storage/persistent_dao.py:
##########
@@ -0,0 +1,594 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from __future__ import annotations
+
+import hashlib
+import hmac
+import logging
+from dataclasses import dataclass
+from typing import Any
+
+import backoff
+from flask import current_app
+from flask_babel import gettext as _
+from sqlalchemy import and_, desc, func, LargeBinary
+
+from superset import db
+from superset.daos.base import BaseDAO
+from superset.distributed_lock import DistributedLock
+from superset.exceptions import (
+ AcquireDistributedLockFailedException,
+ SupersetGenericErrorException,
+)
+from superset.extensions.storage.codecs import DEFAULT_CODEC, get_codec
+from superset.extensions.storage.filters import ExtensionStorageFilter
+from superset.extensions.storage.persistent_model import ExtensionStorage
+from superset.utils.encrypt import (
+ DEFAULT_ENCRYPTION_ENGINE_NAME,
+ EncryptedType,
+ resolve_encryption_engine,
+)
+
+logger = logging.getLogger(__name__)
+
+
+class ExtensionStorageQuotaExceeded(SupersetGenericErrorException):
+ """Raised when a write would exceed an extension's persistent storage quota."""
+
+ status = 413
+
+ def __init__(self, extension_id: str, quota: int) -> None:
+ super().__init__(
+ message=_(
+ "Extension '%(extension_id)s' has exceeded its persistent "
+ "storage quota of %(quota)d bytes.",
+ extension_id=extension_id,
+ quota=quota,
+ ),
+ )
+
+
+class ExtensionStorageValueTooLarge(SupersetGenericErrorException):
+ """Raised when a `set` call's value exceeds MAX_VALUE_SIZE."""
+
+ status = 400
+
+ def __init__(self, max_size: int) -> None:
+ super().__init__(
+ message=_(
+ "Value exceeds the maximum allowed size of %(max_size)d bytes.",
+ max_size=max_size,
+ ),
+ )
+
+
+class ExtensionStorageListPayloadTooLarge(SupersetGenericErrorException):
+ """Raised when a `list` page's combined value size exceeds
+ MAX_LIST_PAYLOAD_SIZE."""
+
+ status = 400
+
+ def __init__(self, requested_size: int, max_size: int) -> None:
+ super().__init__(
+ message=_(
+ "Requested page would return %(requested_size)d bytes, "
+ "exceeding the maximum allowed list payload size of "
+ "%(max_size)d bytes. Reduce page_size and try again.",
+ requested_size=requested_size,
+ max_size=max_size,
+ ),
+ )
+
+
+class ExtensionStorageKeyTooLong(SupersetGenericErrorException):
+ """Raised when a `set` call's key exceeds MAX_KEY_LENGTH."""
+
+ status = 400
+
+ def __init__(self, max_length: int) -> None:
+ super().__init__(
+ message=_(
+ "Storage key exceeds the maximum allowed length of "
+ "%(max_length)d characters.",
+ max_length=max_length,
+ ),
+ )
+
+
+#: Matches the `key` column's `String(255)` definition in `ExtensionStorage`
+#: (`persistent_model.py`). Enforced here, at the DAO boundary, so every
+#: caller (REST API, ambient accessors, direct DAO use) gets a consistent
+#: error instead of a database integrity/truncation failure β the frontend
+#: SDK enforces the same limit client-side purely so it can fail fast
+#: without a round trip, not as the source of truth.
+MAX_KEY_LENGTH = 255
+
+#: TTL for the lock guarding `ExtensionStorageDAO.set`'s select-then-write.
+#: Short because the guarded section is a handful of local DB operations, no
+#: network calls β long enough to cover normal execution, short enough that
+#: a crashed holder doesn't block the same key for long.
+SET_LOCK_TTL_SECONDS = 5
+
+
+def _validate_key_length(key: str) -> None:
+ """Validate key length against MAX_KEY_LENGTH. Raises if over."""
+ if len(key) > MAX_KEY_LENGTH:
+ raise ExtensionStorageKeyTooLong(MAX_KEY_LENGTH)
+
+
+def _derive_key(secret: str, user_fk: int) -> str:
+ """Derive a per-user encryption key from the master secret and user ID.
+
+ Uses HMAC-SHA256 so the derived key is cryptographically bound to both.
+ Ciphertext encrypted for one user cannot be decrypted as another.
+ """
+ return hmac.new(
+ secret.encode(),
+ str(user_fk).encode(),
+ hashlib.sha256,
+ ).hexdigest()
+
+
+def _enc_type(user_fk: int | None) -> EncryptedType:
+ """Return an EncryptedType for the given scope.
+
+ User-scoped rows use a key derived from SECRET_KEY and the user ID, so
+ ciphertext written for one user cannot be decrypted as another.
+ Shared rows (user_fk=None) use SECRET_KEY directly.
+ """
+ secret = current_app.config["SECRET_KEY"]
+ key = _derive_key(secret, user_fk) if user_fk is not None else secret
+ engine_name = current_app.config.get(
+ "SQLALCHEMY_ENCRYPTED_FIELD_ENGINE", DEFAULT_ENCRYPTION_ENGINE_NAME
+ )
+ return EncryptedType(
+ LargeBinary,
+ key=key,
+ engine=resolve_encryption_engine(engine_name),
+ )
+
+
+def _get_quota() -> int | None:
+ """Return the configured per-extension persistent storage quota in bytes.
+
+ Returns None when no quota is configured (unlimited).
+ """
+ config = current_app.config["EXTENSIONS_PERSISTENT_STORAGE"]
+ return config.get("QUOTA_PER_EXTENSION")
+
+
+def _get_max_list_payload_size() -> int | None:
+ """Return the configured max combined value size for a `list()` page.
+
+ Returns None when no limit is configured (unlimited).
+ """
+ config = current_app.config["EXTENSIONS_PERSISTENT_STORAGE"]
+ return config.get("MAX_LIST_PAYLOAD_SIZE")
+
+
+def _validate_value_size(encoded: bytes) -> None:
+ """Validate encoded value bytes against MAX_VALUE_SIZE. Raises if over.
+
+ Enforced independently of `QUOTA_PER_EXTENSION`: the quota bounds an
+ extension's total storage, while this bounds a single value. A
+ `list_entries()` page's combined size is bounded separately by
+ MAX_LIST_PAYLOAD_SIZE (see `ExtensionStorageDAO.list_entries`).
+ """
+ config = current_app.config["EXTENSIONS_PERSISTENT_STORAGE"]
+ max_size = config.get("MAX_VALUE_SIZE")
+ if max_size is None:
+ return
+ if len(encoded) > max_size:
+ raise ExtensionStorageValueTooLarge(max_size)
+
+
+def _check_quota(extension_id: str, new_size: int, existing_size: int = 0) -> None:
+ """Raise if writing `new_size` bytes would push the extension over quota.
+
+ `existing_size` is the size of the row being replaced (0 for inserts),
+ subtracted from current usage so overwriting a key doesn't double-count
+ it against the quota it already occupies.
+
+ The usage query sums the `value_size` column via the covering
+ `ix_ext_storage_extension_id` index (on `extension_id, value_size`)
+ instead of `LENGTH(value)`, so it can run as an index-only scan rather
+ than reading every one of the extension's stored (and potentially
+ TOASTed) blobs. It is still O(rows) for the extension, so write latency
+ grows with row count β acceptable at the row counts a single extension
+ is expected to accumulate, but not a fully constant-time check.
+ """
+ quota = _get_quota()
+ if quota is None:
+ return
+ current_usage = (
+ db.session.query(func.coalesce(func.sum(ExtensionStorage.value_size), 0))
+ .filter(ExtensionStorage.extension_id == extension_id)
+ .scalar()
+ )
+ if current_usage - existing_size + new_size > quota:
+ raise ExtensionStorageQuotaExceeded(extension_id, quota)
+
+
+def _decrypt_if_needed(
+ entry: ExtensionStorage, extension_id: str, key: str
+) -> bytes | None:
+ """Return `entry.value`, decrypted if `entry.is_encrypted`.
+
+ Returns None (and logs) if decryption fails, e.g. after key rotation.
+ """
+ if not entry.is_encrypted:
+ return entry.value
+ try:
+ return _enc_type(entry.user_fk).process_result_value(
+ entry.value, db.engine.dialect
+ )
+ except Exception: # noqa: BLE001
+ logger.error(
+ "Failed to decrypt extension storage value for "
+ "extension_id=%s key=%s β possible key rotation issue",
+ extension_id,
+ key,
+ )
+ return None
+
+
+def _scope_filter(
+ extension_id: str,
+ key: str,
+ user_fk: int | None = None,
+ resource_type: str | None = None,
+ resource_uuid: str | None = None,
+) -> list[object]:
+ """Build the SQLAlchemy filter list for a scoped lookup."""
+ filters: list[object] = [
+ ExtensionStorage.extension_id == extension_id,
+ ExtensionStorage.key == key,
+ ]
+ if user_fk is None:
+ filters.append(ExtensionStorage.user_fk.is_(None))
+ else:
+ filters.append(ExtensionStorage.user_fk == user_fk)
+ if resource_type is None:
+ filters.append(ExtensionStorage.resource_type.is_(None))
+ else:
+ filters.append(ExtensionStorage.resource_type == resource_type)
+ if resource_uuid is None:
+ filters.append(ExtensionStorage.resource_uuid.is_(None))
+ else:
+ filters.append(ExtensionStorage.resource_uuid == resource_uuid)
+ return filters
+
+
+def _list_scope_filter(
+ extension_id: str,
+ user_fk: int | None = None,
+ resource_type: str | None = None,
+ resource_uuid: str | None = None,
+) -> list[object]:
+ """Build the SQLAlchemy filter list for a `list()` call.
+
+ Same scope semantics as `_scope_filter` (an explicit None means "match
+ the global/unset scope"), minus the `key` filter.
+ """
+ filters: list[object] = [ExtensionStorage.extension_id == extension_id]
+ if user_fk is None:
+ filters.append(ExtensionStorage.user_fk.is_(None))
+ else:
+ filters.append(ExtensionStorage.user_fk == user_fk)
+ if resource_type is None:
+ filters.append(ExtensionStorage.resource_type.is_(None))
+ else:
+ filters.append(ExtensionStorage.resource_type == resource_type)
+ if resource_uuid is None:
+ filters.append(ExtensionStorage.resource_uuid.is_(None))
+ else:
+ filters.append(ExtensionStorage.resource_uuid == resource_uuid)
+ return filters
+
+
+@dataclass
+class ExtensionStorageListEntry:
+ """A single row returned by `ExtensionStorageDAO.list_entries`.
+
+ `value` is the raw (decrypted) payload, mirroring `get_value()` β it is
+ not decoded with `codec` here, since REST callers must first check
+ `codec` against `SAFE_CODECS` before deciding whether decoding it is
+ safe to expose over the API, while ambient backend callers (no such
+ restriction) decode unconditionally, same as `get_decoded_value()`.
+ """
+
+ key: str
+ value: bytes | None
+ codec: str
+ is_encrypted: bool
+
+
+class ExtensionStorageDAO(BaseDAO[ExtensionStorage]):
+ """Persistent key-value store for extensions (Tier 3).
+
+ Every query and mutation is scoped to the extension currently executing
+ (resolved from ambient extension context by `ExtensionStorageFilter`),
+ so an extension can never read or modify another extension's storage
+ through this DAO.
+
+ Two access shapes are supported:
+
+ * `get`/`set`/`delete_by_key`/`list_entries` β single-key or
+ paginated-listing operations with explicit scope discriminators
+ (`user_fk`, `resource_type`, `resource_uuid`), used by the ambient
+ `persistent_state`/`ephemeral_state` accessors and the REST API (both
+ of which establish the ambient extension context themselves before
+ calling in).
+ * `find_all`/`find_one_or_none`/`filter_by`/`query`/`create`/`update`/
+ `delete` (inherited from `BaseDAO`) β general-purpose DAO access for
+ backend extension code that needs to enumerate or bulk-manage its own
+ rows, exposed to extensions via `superset_core.extensions.storage.dao`.
+
+ Scope is one of:
+
+ * Global scope β user_fk=None, resource_type=None
+ * User scope β user_fk=<id>, resource_type=None
+ * Resource scope β resource_type + resource_uuid set
+ """
+
+ base_filter = ExtensionStorageFilter
+
+ # ββ Read βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
+
+ @staticmethod
+ def get(
+ extension_id: str,
+ key: str,
+ user_fk: int | None = None,
+ resource_type: str | None = None,
+ resource_uuid: str | None = None,
+ ) -> ExtensionStorage | None:
+ """Return the raw storage entry. The value field may be encrypted."""
+ entry = (
+ db.session.query(ExtensionStorage)
+ .filter(
+ and_(
+ *_scope_filter(
+ extension_id, key, user_fk, resource_type, resource_uuid
+ )
+ )
+ )
+ .first()
+ )
+ return entry
+
+ @staticmethod
+ def get_value(
+ extension_id: str,
+ key: str,
+ user_fk: int | None = None,
+ resource_type: str | None = None,
+ resource_uuid: str | None = None,
+ ) -> bytes | None:
+ """Return the raw (decrypted) value bytes, or None if not found."""
+ entry = ExtensionStorageDAO.get(
+ extension_id, key, user_fk, resource_type, resource_uuid
+ )
+ if entry is None:
+ return None
+ return _decrypt_if_needed(entry, extension_id, key)
+
+ @staticmethod
+ def get_decoded_value(
+ extension_id: str,
+ key: str,
+ user_fk: int | None = None,
+ resource_type: str | None = None,
+ resource_uuid: str | None = None,
+ ) -> Any:
+ """Return the value decoded with the codec it was written with.
+
+ :returns: The decoded value, or None if not found.
+ """
+ entry = ExtensionStorageDAO.get(
+ extension_id, key, user_fk, resource_type, resource_uuid
+ )
+ if entry is None:
+ return None
+ raw = _decrypt_if_needed(entry, extension_id, key)
+ if raw is None:
+ return None
+ return get_codec(entry.codec).decode(raw)
+
+ @staticmethod
+ def list_entries(
+ extension_id: str,
+ user_fk: int | None = None,
+ resource_type: str | None = None,
+ resource_uuid: str | None = None,
+ page: int = 0,
+ page_size: int = 10,
+ ) -> tuple[list[ExtensionStorageListEntry], int]:
+ """List entries in the given scope, most recently changed first.
+
+ Named `list_entries` rather than `list` (unlike the single-key
+ `get`/`set`/`delete_by_key` shape) because `BaseDAO` already
+ defines a generic `list()` with a different, incompatible
+ signature (search/sort/column-selection over arbitrary models);
+ overriding it here would break that contract for callers relying
+ on polymorphic DAO access.
+
+ Callers may request any `page_size` β there is no row-count
+ ceiling β but the combined `value_size` of the requested page is
+ checked against MAX_LIST_PAYLOAD_SIZE *before* the (potentially
+ large) `value` column is fetched for any row, so an oversized
+ request fails fast rather than reading data it will then reject.
+
+ :returns: (entries, total_count). `entries[i].value` is the raw
+ (decrypted) payload β see `ExtensionStorageListEntry`.
+ :raises ExtensionStorageListPayloadTooLarge: if the requested page's
+ combined value size exceeds MAX_LIST_PAYLOAD_SIZE.
+ """
+ filters = _list_scope_filter(
+ extension_id, user_fk, resource_type, resource_uuid
+ )
+ base_query = db.session.query(ExtensionStorage).filter(and_(*filters))
+ total_count = base_query.count()
+
+ page = max(page, 0)
+ page_size = max(page_size, 1)
+ page_query = (
+ base_query.order_by(desc(ExtensionStorage.changed_on))
+ .offset(page * page_size)
+ .limit(page_size)
+ )
+
+ if (max_payload_size := _get_max_list_payload_size()) is not None:
+ # Sum value_size for exactly this page's rows, without touching
+ # the `value` column, by reusing the same ordered/offset/limited
+ # query as a subquery.
+ subquery = page_query.with_entities(ExtensionStorage.value_size).subquery()
+ requested_size = db.session.query(
+ func.coalesce(func.sum(subquery.c.value_size), 0)
+ ).scalar()
+ if requested_size > max_payload_size:
+ raise ExtensionStorageListPayloadTooLarge(
+ requested_size, max_payload_size
+ )
+
+ entries = [
+ ExtensionStorageListEntry(
+ key=row.key,
+ value=_decrypt_if_needed(row, extension_id, row.key),
+ codec=row.codec,
+ is_encrypted=row.is_encrypted,
+ )
+ for row in page_query.all()
+ ]
+ return entries, total_count
+
+ # ββ Write (upsert) ββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
+
+ @staticmethod
+ @backoff.on_exception(
+ backoff.expo,
+ AcquireDistributedLockFailedException,
+ factor=0.1,
+ base=2,
+ max_tries=8,
+ )
+ def set(
+ extension_id: str,
+ key: str,
+ value: bytes,
+ codec: str = DEFAULT_CODEC,
+ user_fk: int | None = None,
+ resource_type: str | None = None,
+ resource_uuid: str | None = None,
+ encrypt: bool = False,
+ ) -> ExtensionStorage:
+ """Upsert a storage entry. Encrypts value when encrypt=True.
+
+ The select-then-insert/update below is guarded by a distributed lock
+ scoped to this exact key: the row's unique constraint can't be relied
+ on to catch concurrent writes here, since `user_fk`, `resource_type`,
+ and `resource_uuid` are nullable and standard SQL never treats
+ NULL = NULL, so two concurrent inserts for the same *global* or
+ *resource*-scoped key (where one or more of those columns is NULL)
+ would not violate it and would land as duplicate rows.
+
+ :raises ExtensionStorageKeyTooLong: if `key` exceeds MAX_KEY_LENGTH.
+ :raises ExtensionStorageValueTooLarge: if `value` exceeds
+ MAX_VALUE_SIZE.
+ :raises ExtensionStorageQuotaExceeded: if writing `value` would push
+ the extension over its total storage quota.
+ """
+ _validate_key_length(key)
+ _validate_value_size(value)
+ stored_value = (
+ _enc_type(user_fk).process_bind_param(value, db.engine.dialect)
+ if encrypt
+ else value
+ )
+
+ with DistributedLock(
+ namespace="extension_storage_set",
+ ttl_seconds=SET_LOCK_TTL_SECONDS,
+ extension_id=extension_id,
+ key=key,
+ user_fk=user_fk,
+ resource_type=resource_type,
+ resource_uuid=resource_uuid,
+ ):
+ entry = (
+ db.session.query(ExtensionStorage)
+ .filter(
+ and_(
+ *_scope_filter(
+ extension_id, key, user_fk, resource_type, resource_uuid
+ )
+ )
+ )
+ .first()
+ )
+ new_size = len(stored_value)
+ existing_size = entry.value_size if entry is not None else 0
+ _check_quota(extension_id, new_size, existing_size)
+ if entry is not None:
Review Comment:
**Suggestion:** The quota check is guarded by a lock scoped to a single key, but quota is enforced at extension-wide scope. Concurrent writes to different keys for the same extension can both pass the usage check and commit, allowing total usage to exceed the configured quota. Use an extension-level lock (or transactional/DB-side quota enforcement) around the quota read+write path. [race condition]
<details>
<summary><b>Severity Level:</b> Major β οΈ</summary>
```mdx
- β Per-extension persistent storage quota can be exceeded.
- β οΈ Extensions may store more data than admins expect.
- β οΈ Multi-tenant isolation guarantees for quotas are weakened.
```
</details>
<details>
<summary><b>Steps of Reproduction β
</b></summary>
```mdx
1. Configure a per-extension quota by setting
EXTENSIONS_PERSISTENT_STORAGE["QUOTA_PER_EXTENSION"] in Flask config (read in
superset/extensions/storage/persistent_dao.py:_get_quota at lines 166-173).
2. From two separate workers/threads, call ExtensionStorageDAO.set()
(superset/extensions/storage/persistent_dao.py:490) for the same extension_id but
different keys, so each call acquires a different DistributedLock at lines 524-532 (lock
scoped by key, user_fk, resource_type, resource_uuid).
3. Ensure that before both calls, the sum of value_size for that extension (computed in
_check_quota at lines 215-224) is close to the quota, for example current_usage=900,
quota=1000, and each new stored_value is 100 bytes (new_size=100, existing_size=0).
4. Both calls execute _check_quota(extension_id, new_size, existing_size) at line 547;
each independently queries current_usage via db.session (lines 215-222), sees the same
pre-write usage, passes the quota check, and then both write and flush (lines 548-565),
resulting in total usage exceeding the configured quota without any
ExtensionStorageQuotaExceeded being raised.
```
</details>
[](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=30525dd1746b4069ac29935f523f5c11&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=30525dd1746b4069ac29935f523f5c11&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset)
*(Use Cmd/Ctrl + Click for best experience)*
<details>
<summary><b>Prompt for AI Agent π€ </b></summary>
```mdx
This is a comment left during a code review.
**Path:** superset/extensions/storage/persistent_dao.py
**Line:** 524:547
**Comment:**
*Race Condition: The quota check is guarded by a lock scoped to a single key, but quota is enforced at extension-wide scope. Concurrent writes to different keys for the same extension can both pass the usage check and commit, allowing total usage to exceed the configured quota. Use an extension-level lock (or transactional/DB-side quota enforcement) around the quota read+write path.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F39171&comment_hash=bd02766ce98e3bbf9dd22418236d4406e85899cc7dd4839008fafd7ce1953de0&reaction=like'>π</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F39171&comment_hash=bd02766ce98e3bbf9dd22418236d4406e85899cc7dd4839008fafd7ce1953de0&reaction=dislike'>π</a>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(extensions): add a Storage API for extensions [superset]
Posted by "michael-s-molina (via GitHub)" <gi...@apache.org>.
michael-s-molina merged PR #39171:
URL: https://github.com/apache/superset/pull/39171
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(extensions): add a Storage API for extensions [superset]
Posted by "michael-s-molina (via GitHub)" <gi...@apache.org>.
michael-s-molina commented on code in PR #39171:
URL: https://github.com/apache/superset/pull/39171#discussion_r3624938477
##########
superset/extensions/storage/persistent_dao.py:
##########
@@ -0,0 +1,637 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from __future__ import annotations
+
+import hashlib
+import hmac
+import logging
+from dataclasses import dataclass
+from typing import Any
+
+import backoff
+from flask import current_app
+from flask_babel import gettext as _
+from sqlalchemy import and_, desc, func, LargeBinary
+
+from superset import db
+from superset.daos.base import BaseDAO
+from superset.distributed_lock import DistributedLock
+from superset.exceptions import (
+ AcquireDistributedLockFailedException,
+ SupersetGenericErrorException,
+)
+from superset.extensions.storage.codecs import DEFAULT_CODEC, get_codec
+from superset.extensions.storage.filters import ExtensionStorageFilter
+from superset.extensions.storage.persistent_model import ExtensionStorage
+from superset.utils.encrypt import (
+ DEFAULT_ENCRYPTION_ENGINE_NAME,
+ EncryptedType,
+ resolve_encryption_engine,
+)
+
+logger = logging.getLogger(__name__)
+
+
+class ExtensionStorageQuotaExceeded(SupersetGenericErrorException):
+ """Raised when a write would exceed an extension's persistent storage quota."""
+
+ status = 413
+
+ def __init__(self, extension_id: str, quota: int) -> None:
+ super().__init__(
+ message=_(
+ "Extension '%(extension_id)s' has exceeded its persistent "
+ "storage quota of %(quota)d bytes.",
+ extension_id=extension_id,
+ quota=quota,
+ ),
+ )
+
+
+class ExtensionStorageValueTooLarge(SupersetGenericErrorException):
+ """Raised when a `set` call's value exceeds MAX_VALUE_SIZE."""
+
+ status = 400
+
+ def __init__(self, max_size: int) -> None:
+ super().__init__(
+ message=_(
+ "Value exceeds the maximum allowed size of %(max_size)d bytes.",
+ max_size=max_size,
+ ),
+ )
+
+
+class ExtensionStorageListPayloadTooLarge(SupersetGenericErrorException):
+ """Raised when a `list` page's combined value size exceeds
+ MAX_LIST_PAYLOAD_SIZE."""
+
+ status = 400
+
+ def __init__(self, requested_size: int, max_size: int) -> None:
+ super().__init__(
+ message=_(
+ "Requested page would return %(requested_size)d bytes, "
+ "exceeding the maximum allowed list payload size of "
+ "%(max_size)d bytes. Reduce page_size and try again.",
+ requested_size=requested_size,
+ max_size=max_size,
+ ),
+ )
+
+
+class ExtensionStorageKeyTooLong(SupersetGenericErrorException):
+ """Raised when a `set` call's key exceeds MAX_KEY_LENGTH."""
+
+ status = 400
+
+ def __init__(self, max_length: int) -> None:
+ super().__init__(
+ message=_(
+ "Storage key exceeds the maximum allowed length of "
+ "%(max_length)d characters.",
+ max_length=max_length,
+ ),
+ )
+
+
+#: Matches the `key` column's `String(255)` definition in `ExtensionStorage`
+#: (`persistent_model.py`). Enforced here, at the DAO boundary, so every
+#: caller (REST API, ambient accessors, direct DAO use) gets a consistent
+#: error instead of a database integrity/truncation failure β the frontend
+#: SDK enforces the same limit client-side purely so it can fail fast
+#: without a round trip, not as the source of truth.
+MAX_KEY_LENGTH = 255
+
+#: TTL for the lock guarding `ExtensionStorageDAO.set`'s select-then-write.
+#: Short because the guarded section is a handful of local DB operations, no
+#: network calls β long enough to cover normal execution, short enough that
+#: a crashed holder doesn't block the same key for long.
+SET_LOCK_TTL_SECONDS = 5
+
+
+def _validate_key_length(key: str) -> None:
+ """Validate key length against MAX_KEY_LENGTH. Raises if over."""
+ if len(key) > MAX_KEY_LENGTH:
+ raise ExtensionStorageKeyTooLong(MAX_KEY_LENGTH)
+
+
+def _derive_key(secret: str, user_fk: int) -> str:
+ """Derive a per-user encryption key from the master secret and user ID.
+
+ Uses HMAC-SHA256 so the derived key is cryptographically bound to both.
+ Ciphertext encrypted for one user cannot be decrypted as another.
+ """
+ return hmac.new(
+ secret.encode(),
+ str(user_fk).encode(),
+ hashlib.sha256,
+ ).hexdigest()
+
+
+def _enc_type(user_fk: int | None) -> EncryptedType:
+ """Return an EncryptedType for the given scope.
+
+ User-scoped rows use a key derived from SECRET_KEY and the user ID, so
+ ciphertext written for one user cannot be decrypted as another.
+ Shared rows (user_fk=None) use SECRET_KEY directly.
+ """
+ secret = current_app.config["SECRET_KEY"]
+ key = _derive_key(secret, user_fk) if user_fk is not None else secret
+ engine_name = current_app.config.get(
+ "SQLALCHEMY_ENCRYPTED_FIELD_ENGINE", DEFAULT_ENCRYPTION_ENGINE_NAME
+ )
+ return EncryptedType(
+ LargeBinary,
+ key=key,
+ engine=resolve_encryption_engine(engine_name),
+ )
+
+
+def _get_quota() -> int | None:
+ """Return the configured per-extension persistent storage quota in bytes.
+
+ Returns None when no quota is configured (unlimited).
+ """
+ config = current_app.config["EXTENSIONS_PERSISTENT_STORAGE"]
+ return config.get("QUOTA_PER_EXTENSION")
+
+
+def _get_max_list_payload_size() -> int | None:
+ """Return the configured max combined value size for a `list()` page.
+
+ Returns None when no limit is configured (unlimited).
+ """
+ config = current_app.config["EXTENSIONS_PERSISTENT_STORAGE"]
+ return config.get("MAX_LIST_PAYLOAD_SIZE")
+
+
+def _validate_value_size(encoded: bytes) -> None:
+ """Validate encoded value bytes against MAX_VALUE_SIZE. Raises if over.
+
+ Enforced independently of `QUOTA_PER_EXTENSION`: the quota bounds an
+ extension's total storage, while this bounds a single value. A
+ `list_entries()` page's combined size is bounded separately by
+ MAX_LIST_PAYLOAD_SIZE (see `ExtensionStorageDAO.list_entries`).
+ """
+ config = current_app.config["EXTENSIONS_PERSISTENT_STORAGE"]
+ max_size = config.get("MAX_VALUE_SIZE")
+ if max_size is None:
+ return
+ if len(encoded) > max_size:
+ raise ExtensionStorageValueTooLarge(max_size)
+
+
+def _check_quota(extension_id: str, new_size: int, existing_size: int = 0) -> None:
+ """Raise if writing `new_size` bytes would push the extension over quota.
+
+ `existing_size` is the size of the row being replaced (0 for inserts),
+ subtracted from current usage so overwriting a key doesn't double-count
+ it against the quota it already occupies.
+
+ The usage query sums the `value_size` column via the covering
+ `ix_ext_storage_extension_id` index (on `extension_id, value_size`)
+ instead of `LENGTH(value)`, so it can run as an index-only scan rather
+ than reading every one of the extension's stored (and potentially
+ TOASTed) blobs. It is still O(rows) for the extension, so write latency
+ grows with row count β acceptable at the row counts a single extension
+ is expected to accumulate, but not a fully constant-time check.
+ """
+ quota = _get_quota()
+ if quota is None:
+ return
+ current_usage = (
+ db.session.query(func.coalesce(func.sum(ExtensionStorage.value_size), 0))
+ .filter(ExtensionStorage.extension_id == extension_id)
+ .scalar()
+ )
+ if current_usage - existing_size + new_size > quota:
+ raise ExtensionStorageQuotaExceeded(extension_id, quota)
+
+
+def _decrypt_if_needed(
+ entry: ExtensionStorage, extension_id: str, key: str
+) -> bytes | None:
+ """Return `entry.value`, decrypted if `entry.is_encrypted`.
+
+ Returns None (and logs) if decryption fails, e.g. after key rotation.
+ """
+ if not entry.is_encrypted:
+ return entry.value
+ try:
+ return _enc_type(entry.user_fk).process_result_value(
+ entry.value, db.engine.dialect
+ )
+ except Exception: # noqa: BLE001
+ logger.error(
+ "Failed to decrypt extension storage value for "
+ "extension_id=%s key=%s β possible key rotation issue",
+ extension_id,
+ key,
+ )
+ return None
+
+
+def _scope_filter(
+ extension_id: str,
+ key: str,
+ user_fk: int | None = None,
+ resource_type: str | None = None,
+ resource_uuid: str | None = None,
+) -> list[object]:
+ """Build the SQLAlchemy filter list for a scoped lookup."""
+ filters: list[object] = [
+ ExtensionStorage.extension_id == extension_id,
+ ExtensionStorage.key == key,
+ ]
+ if user_fk is None:
+ filters.append(ExtensionStorage.user_fk.is_(None))
+ else:
+ filters.append(ExtensionStorage.user_fk == user_fk)
+ if resource_type is None:
+ filters.append(ExtensionStorage.resource_type.is_(None))
+ else:
+ filters.append(ExtensionStorage.resource_type == resource_type)
+ if resource_uuid is None:
+ filters.append(ExtensionStorage.resource_uuid.is_(None))
+ else:
+ filters.append(ExtensionStorage.resource_uuid == resource_uuid)
+ return filters
+
+
+def _list_scope_filter(
+ extension_id: str,
+ user_fk: int | None = None,
+ resource_type: str | None = None,
+ resource_uuid: str | None = None,
+) -> list[object]:
+ """Build the SQLAlchemy filter list for a `list()` call.
+
+ Same scope semantics as `_scope_filter` (an explicit None means "match
+ the global/unset scope"), minus the `key` filter.
+ """
+ filters: list[object] = [ExtensionStorage.extension_id == extension_id]
+ if user_fk is None:
+ filters.append(ExtensionStorage.user_fk.is_(None))
+ else:
+ filters.append(ExtensionStorage.user_fk == user_fk)
+ if resource_type is None:
+ filters.append(ExtensionStorage.resource_type.is_(None))
+ else:
+ filters.append(ExtensionStorage.resource_type == resource_type)
+ if resource_uuid is None:
+ filters.append(ExtensionStorage.resource_uuid.is_(None))
+ else:
+ filters.append(ExtensionStorage.resource_uuid == resource_uuid)
+ return filters
+
+
+@dataclass
+class ExtensionStorageListEntry:
+ """A single row returned by `ExtensionStorageDAO.list_entries`.
+
+ `value` is the raw (decrypted) payload, mirroring `get_value()` β it is
+ not decoded with `codec` here, since REST callers must first check
+ `codec` against `SAFE_CODECS` before deciding whether decoding it is
+ safe to expose over the API, while ambient backend callers (no such
+ restriction) decode unconditionally, same as `get_decoded_value()`.
+ """
+
+ key: str
+ value: bytes | None
+ codec: str
+ is_encrypted: bool
+
+
+class ExtensionStorageDAO(BaseDAO[ExtensionStorage]):
+ """Persistent key-value store for extensions (Tier 3).
+
+ Every query and mutation is scoped to the extension currently executing
+ (resolved from ambient extension context by `ExtensionStorageFilter`),
+ so an extension can never read or modify another extension's storage
+ through this DAO.
+
+ Two access shapes are supported:
+
+ * `get`/`set`/`delete_by_key`/`list_entries` β single-key or
+ paginated-listing operations with explicit scope discriminators
+ (`user_fk`, `resource_type`, `resource_uuid`), used by the ambient
+ `persistent_state`/`ephemeral_state` accessors and the REST API (both
+ of which establish the ambient extension context themselves before
+ calling in).
+ * `find_all`/`find_one_or_none`/`filter_by`/`query`/`create`/`update`/
+ `delete` (inherited from `BaseDAO`) β general-purpose DAO access for
+ backend extension code that needs to enumerate or bulk-manage its own
+ rows, exposed to extensions via `superset_core.extensions.storage.dao`.
+
+ Scope is one of:
+
+ * Global scope β user_fk=None, resource_type=None
+ * User scope β user_fk=<id>, resource_type=None
+ * Resource scope β resource_type + resource_uuid set
+ """
+
+ base_filter = ExtensionStorageFilter
+
+ # ββ Read βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
+
+ @staticmethod
+ def get(
+ extension_id: str,
+ key: str,
+ user_fk: int | None = None,
+ resource_type: str | None = None,
+ resource_uuid: str | None = None,
+ ) -> ExtensionStorage | None:
+ """Return the raw storage entry. The value field may be encrypted."""
+ entry = (
+ db.session.query(ExtensionStorage)
+ .filter(
+ and_(
+ *_scope_filter(
+ extension_id, key, user_fk, resource_type, resource_uuid
+ )
+ )
+ )
+ .first()
+ )
+ return entry
+
+ @staticmethod
+ def get_value(
+ extension_id: str,
+ key: str,
+ user_fk: int | None = None,
+ resource_type: str | None = None,
+ resource_uuid: str | None = None,
+ ) -> bytes | None:
+ """Return the raw (decrypted) value bytes, or None if not found."""
+ entry = ExtensionStorageDAO.get(
+ extension_id, key, user_fk, resource_type, resource_uuid
+ )
+ if entry is None:
+ return None
+ return _decrypt_if_needed(entry, extension_id, key)
+
+ @staticmethod
+ def get_decoded_value(
+ extension_id: str,
+ key: str,
+ user_fk: int | None = None,
+ resource_type: str | None = None,
+ resource_uuid: str | None = None,
+ ) -> Any:
+ """Return the value decoded with the codec it was written with.
+
+ :returns: The decoded value, or None if not found.
+ """
+ entry = ExtensionStorageDAO.get(
+ extension_id, key, user_fk, resource_type, resource_uuid
+ )
+ if entry is None:
+ return None
+ return ExtensionStorageDAO.decode_entry(entry)
+
+ @staticmethod
+ def decode_entry(entry: ExtensionStorage) -> Any:
+ """Decode an already-fetched entry's value with the codec it was
+ written with.
+
+ Unlike `get_decoded_value`, this takes the entry directly instead
+ of re-querying for it, so a caller that already fetched (and, e.g.,
+ validated the codec of) an entry decodes that exact same row
+ instead of racing a concurrent write between two separate reads.
+
+ :returns: The decoded value, or None if the value fails to decrypt.
+ """
+ raw = _decrypt_if_needed(entry, entry.extension_id, entry.key)
+ if raw is None:
+ return None
+ return get_codec(entry.codec).decode(raw)
+
+ @staticmethod
+ def list_entries(
+ extension_id: str,
+ user_fk: int | None = None,
+ resource_type: str | None = None,
+ resource_uuid: str | None = None,
+ page: int = 0,
+ page_size: int = 10,
+ ) -> tuple[list[ExtensionStorageListEntry], int]:
+ """List entries in the given scope, most recently changed first.
+
+ Named `list_entries` rather than `list` (unlike the single-key
+ `get`/`set`/`delete_by_key` shape) because `BaseDAO` already
+ defines a generic `list()` with a different, incompatible
+ signature (search/sort/column-selection over arbitrary models);
+ overriding it here would break that contract for callers relying
+ on polymorphic DAO access.
+
+ Callers may request any `page_size` β there is no row-count
+ ceiling β but the combined `value_size` of the requested page is
+ checked against MAX_LIST_PAYLOAD_SIZE *before* the (potentially
+ large) `value` column is fetched for any row, so an oversized
+ request fails fast rather than reading data it will then reject.
+
+ :returns: (entries, total_count). `entries[i].value` is the raw
+ (decrypted) payload β see `ExtensionStorageListEntry`.
+ :raises ExtensionStorageListPayloadTooLarge: if the requested page's
+ combined value size exceeds MAX_LIST_PAYLOAD_SIZE.
+ """
+ filters = _list_scope_filter(
+ extension_id, user_fk, resource_type, resource_uuid
+ )
+ base_query = db.session.query(ExtensionStorage).filter(and_(*filters))
+ total_count = base_query.count()
+
+ page = max(page, 0)
+ page_size = max(page_size, 1)
+ page_query = (
+ base_query.order_by(desc(ExtensionStorage.changed_on))
+ .offset(page * page_size)
+ .limit(page_size)
+ )
+
+ if (max_payload_size := _get_max_list_payload_size()) is not None:
+ # Sum value_size for exactly this page's rows, without touching
+ # the `value` column, by reusing the same ordered/offset/limited
+ # query as a subquery.
+ subquery = page_query.with_entities(ExtensionStorage.value_size).subquery()
+ requested_size = db.session.query(
+ func.coalesce(func.sum(subquery.c.value_size), 0)
+ ).scalar()
+ if requested_size > max_payload_size:
+ raise ExtensionStorageListPayloadTooLarge(
+ requested_size, max_payload_size
+ )
+
+ entries = [
+ ExtensionStorageListEntry(
+ key=row.key,
+ value=_decrypt_if_needed(row, extension_id, row.key),
+ codec=row.codec,
+ is_encrypted=row.is_encrypted,
+ )
+ for row in page_query.all()
+ ]
+ return entries, total_count
+
+ # ββ Write (upsert) ββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
+
+ @staticmethod
+ @backoff.on_exception(
+ backoff.expo,
+ AcquireDistributedLockFailedException,
+ factor=0.1,
+ base=2,
+ max_tries=8,
+ )
+ def set(
+ extension_id: str,
+ key: str,
+ value: bytes,
+ codec: str = DEFAULT_CODEC,
+ user_fk: int | None = None,
+ resource_type: str | None = None,
+ resource_uuid: str | None = None,
+ encrypt: bool = False,
+ ) -> ExtensionStorage:
+ """Upsert a storage entry. Encrypts value when encrypt=True.
+
+ The select-then-insert/update below is guarded by a distributed lock
+ scoped to the whole extension, for two independent reasons:
+
+ * The row's unique constraint can't be relied on to catch concurrent
+ writes to the same key, since `user_fk`, `resource_type`, and
+ `resource_uuid` are nullable and standard SQL never treats
+ NULL = NULL, so two concurrent inserts for the same *global* or
+ *resource*-scoped key (where one or more of those columns is NULL)
+ would not violate it and would land as duplicate rows.
+ * `_check_quota` below enforces usage extension-wide, summed across
+ every key. A lock scoped to only this key would let concurrent
+ writes to *different* keys of the same extension each read usage
+ before the other commits, both pass the check, and land a combined
+ write that exceeds the extension's quota.
+
+ :raises ExtensionStorageKeyTooLong: if `key` exceeds MAX_KEY_LENGTH.
+ :raises ExtensionStorageValueTooLarge: if `value` exceeds
+ MAX_VALUE_SIZE.
+ :raises ExtensionStorageQuotaExceeded: if writing `value` would push
+ the extension over its total storage quota.
+ """
+ _validate_key_length(key)
+ _validate_value_size(value)
+ stored_value = (
+ _enc_type(user_fk).process_bind_param(value, db.engine.dialect)
+ if encrypt
+ else value
+ )
+
+ with DistributedLock(
+ namespace="extension_storage_set",
+ ttl_seconds=SET_LOCK_TTL_SECONDS,
+ extension_id=extension_id,
+ ):
+ entry = (
+ db.session.query(ExtensionStorage)
+ .filter(
+ and_(
+ *_scope_filter(
+ extension_id, key, user_fk, resource_type, resource_uuid
+ )
+ )
+ )
+ .first()
+ )
+ new_size = len(stored_value)
+ existing_size = entry.value_size if entry is not None else 0
+ _check_quota(extension_id, new_size, existing_size)
+ if entry is not None:
+ entry.value = stored_value
+ entry.value_size = new_size
+ entry.codec = codec
+ entry.is_encrypted = encrypt
+ else:
+ entry = ExtensionStorage(
+ extension_id=extension_id,
+ key=key,
+ value=stored_value,
+ value_size=new_size,
+ codec=codec,
+ user_fk=user_fk,
+ resource_type=resource_type,
+ resource_uuid=resource_uuid,
+ is_encrypted=encrypt,
+ )
+ db.session.add(entry)
+ db.session.flush()
+ return entry
+
+ # ββ Create (disallowed) βββββββββββββββββββββββββββββββββββββββββββββββββββββ
+
+ @classmethod
+ def create(
+ cls,
+ item: ExtensionStorage | None = None,
+ attributes: dict[str, Any] | None = None,
+ ) -> ExtensionStorage:
+ """Not supported for this DAO.
+
+ `create()` is `BaseDAO`'s raw insert: no upsert dedup against an
+ existing row for the same scoped key, no quota/key-length/value-size
+ validation, and no locking β calling it directly could construct a
+ duplicate row for a key `set()` already wrote (see `set()`'s
+ docstring) without any of the checks that make that safe.
+
+ :raises NotImplementedError: always; use `ExtensionStorageDAO.set()`
+ (or the ambient `ctx.storage.persistent.set()` accessor) instead.
+ """
+ raise NotImplementedError(
+ "ExtensionStorageDAO.create() is not supported for extension "
+ "storage: it bypasses the locking, quota, and validation that "
+ "make writing a value safe. Use ctx.storage.persistent.set() "
+ "(or ExtensionStorageDAO.set() from backend code) instead."
+ )
+
+ # ββ Delete ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
+
+ @staticmethod
+ def delete_by_key(
+ extension_id: str,
+ key: str,
+ user_fk: int | None = None,
+ resource_type: str | None = None,
+ resource_uuid: str | None = None,
+ ) -> bool:
+ """Delete a single entry by key. Returns True if a row was removed."""
+ entry = (
+ db.session.query(ExtensionStorage)
+ .filter(
+ and_(
+ *_scope_filter(
+ extension_id, key, user_fk, resource_type, resource_uuid
+ )
+ )
+ )
+ .first()
+ )
+ if entry is None:
+ return False
+ db.session.delete(entry)
+ db.session.flush()
+ return True
Review Comment:
Confirmed the underlying race, but fixed it at the root instead of in `delete_by_key`. Traced why duplicates could actually occur: `ExtensionStorageDAO.set()`'s `DistributedLock` released as soon as its own `with` block exited, but the wrapping `@transaction()` (on `PersistentState.set`/`set_persistent`) only commits *after* the whole call returns β so the lock was freed before the row was durable, letting a second writer's post-lock `SELECT` still miss the first writer's uncommitted insert and also insert.
Fix: `set()` now commits while still holding the lock (with a `pylint: disable=consider-using-transaction`, matching existing precedent elsewhere for this exact situation), so the write is guaranteed durable before the next lock holder reads. Combined with Superset forcing `READ COMMITTED` isolation by default for both MySQL/Postgres, that closes the race β duplicate rows for the same scoped key can no longer be created, so `delete_by_key` stays as a single-row delete (reverted the `.delete()`-all-matching-rows change from my earlier pass). Added `test_dao_set_commits_before_releasing_the_lock`, which asserts `commit()` is called before the lock's `__exit__`, so a regression here would be caught.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(extensions): add a Storage API for extensions [superset]
Posted by "michael-s-molina (via GitHub)" <gi...@apache.org>.
michael-s-molina commented on code in PR #39171:
URL: https://github.com/apache/superset/pull/39171#discussion_r3622754350
##########
superset/extensions/storage/api.py:
##########
@@ -0,0 +1,732 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""
+REST API for extension storage.
+
+Provides HTTP endpoints for frontend extensions to access server-side
+ephemeral storage without direct backend code.
+
+All operations are user-scoped by default. Use `?shared=true` query param
+to access shared state visible to all users.
+"""
+
+from __future__ import annotations
+
+import base64
+from typing import Any
+
+from flask import g, request
+from flask.wrappers import Response
+from flask_appbuilder.api import BaseApi, expose, protect, safe
+
+from superset.extensions.storage.codecs import DEFAULT_CODEC, get_codec, SAFE_CODECS
+from superset.extensions.storage.ephemeral_dao import (
+ ExtensionEphemeralDAO,
+ ExtensionEphemeralTTLInvalid,
+ ExtensionEphemeralValueTooLarge,
+)
+from superset.extensions.storage.persistent_dao import (
+ ExtensionStorageDAO,
+ ExtensionStorageKeyTooLong,
+ ExtensionStorageListPayloadTooLarge,
+ ExtensionStorageQuotaExceeded,
+ ExtensionStorageValueTooLarge,
+)
+from superset.extensions.storage.utils import get_extension_or_404, parse_ttl
+from superset.utils.decorators import transaction
+
+
+def _decoded_result_for_wire(decoded: Any) -> tuple[Any, bool]:
+ """Convert a codec-decoded value into its JSON wire representation.
+
+ JSON has no byte type, so a raw `bytes` value is base64-encoded to a
+ string for the response and flagged as such; every other value is
+ already JSON-representable as-is. Checked on the decoded value's
+ actual type, not the codec's name, so this keeps working for a codec
+ this module didn't define.
+
+ :returns: (wire_value, is_binary)
+ """
+ if isinstance(decoded, bytes):
+ return base64.b64encode(decoded).decode("ascii"), True
+ return decoded, False
+
+
+def _wire_value_for_request(value: Any, is_binary: bool) -> Any:
+ """Convert a request body's JSON `value` into a codec's input type.
+
+ `is_binary` is an explicit flag from the caller: JSON has no byte
+ type, so there is no way to tell, from the JSON `value` alone,
+ whether it is a base64 string that must be decoded to bytes before
+ being handed to the codec's `encode`, or a literal value to pass
+ through as-is. Only the caller knows which it sent.
+
+ :raises ValueError: if `is_binary` is set and `value` is not a valid
+ base64 string.
+ """
+ if is_binary:
+ return base64.b64decode(value, validate=True)
+ return value
+
+
+class ExtensionStorageRestApi(BaseApi):
+ """REST API for extension ephemeral state storage."""
+
Review Comment:
Fixed β updated the class docstring to "REST API for extension ephemeral (Tier 2) and persistent (Tier 3) storage."
##########
superset/extensions/storage/api.py:
##########
@@ -0,0 +1,732 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""
+REST API for extension storage.
+
+Provides HTTP endpoints for frontend extensions to access server-side
+ephemeral storage without direct backend code.
+
+All operations are user-scoped by default. Use `?shared=true` query param
+to access shared state visible to all users.
+"""
+
+from __future__ import annotations
+
+import base64
+from typing import Any
+
+from flask import g, request
+from flask.wrappers import Response
+from flask_appbuilder.api import BaseApi, expose, protect, safe
+
+from superset.extensions.storage.codecs import DEFAULT_CODEC, get_codec, SAFE_CODECS
+from superset.extensions.storage.ephemeral_dao import (
+ ExtensionEphemeralDAO,
+ ExtensionEphemeralTTLInvalid,
+ ExtensionEphemeralValueTooLarge,
+)
+from superset.extensions.storage.persistent_dao import (
+ ExtensionStorageDAO,
+ ExtensionStorageKeyTooLong,
+ ExtensionStorageListPayloadTooLarge,
+ ExtensionStorageQuotaExceeded,
+ ExtensionStorageValueTooLarge,
+)
+from superset.extensions.storage.utils import get_extension_or_404, parse_ttl
+from superset.utils.decorators import transaction
+
+
+def _decoded_result_for_wire(decoded: Any) -> tuple[Any, bool]:
+ """Convert a codec-decoded value into its JSON wire representation.
+
+ JSON has no byte type, so a raw `bytes` value is base64-encoded to a
+ string for the response and flagged as such; every other value is
+ already JSON-representable as-is. Checked on the decoded value's
+ actual type, not the codec's name, so this keeps working for a codec
+ this module didn't define.
+
+ :returns: (wire_value, is_binary)
+ """
+ if isinstance(decoded, bytes):
+ return base64.b64encode(decoded).decode("ascii"), True
+ return decoded, False
+
+
+def _wire_value_for_request(value: Any, is_binary: bool) -> Any:
+ """Convert a request body's JSON `value` into a codec's input type.
+
+ `is_binary` is an explicit flag from the caller: JSON has no byte
+ type, so there is no way to tell, from the JSON `value` alone,
+ whether it is a base64 string that must be decoded to bytes before
+ being handed to the codec's `encode`, or a literal value to pass
+ through as-is. Only the caller knows which it sent.
+
+ :raises ValueError: if `is_binary` is set and `value` is not a valid
+ base64 string.
+ """
+ if is_binary:
+ return base64.b64decode(value, validate=True)
+ return value
+
+
+class ExtensionStorageRestApi(BaseApi):
+ """REST API for extension ephemeral state storage."""
+
+ allow_browser_login = True
+ route_base = "/api/v1/extensions"
+
+ def response(self, status_code: int, **kwargs: Any) -> Response:
+ """Helper method to create JSON responses."""
+ from flask import jsonify
+
+ return jsonify(kwargs), status_code
+
+ def response_404(self, message: str = "Not found") -> Response:
+ """Helper method to create 404 responses."""
+ from flask import jsonify
+
+ return jsonify({"message": message}), 404
+
+ def response_400(self, message: str) -> Response:
+ """Helper method to create 400 responses."""
+ from flask import jsonify
+
+ return jsonify({"message": message}), 400
+
+ @protect()
+ @safe
+ @expose("/<publisher>/<name>/storage/ephemeral/<key>", methods=("GET",))
+ def get_ephemeral(
+ self, publisher: str, name: str, key: str, **kwargs: Any
+ ) -> Response:
+ """Get a value from ephemeral state.
+ ---
+ get:
+ summary: Get a value from ephemeral state
+ parameters:
+ - in: path
+ name: publisher
+ schema:
+ type: string
+ required: true
+ description: Extension publisher
+ - in: path
+ name: name
+ schema:
+ type: string
+ required: true
+ description: Extension name
+ - in: path
+ name: key
+ schema:
+ type: string
+ required: true
+ description: Storage key
+ - in: query
+ name: shared
+ schema:
+ type: boolean
+ required: false
+ description: If true, read from shared state visible to all users
+ responses:
+ 200:
+ description: Value retrieved successfully
+ content:
+ application/json:
+ schema:
+ type: object
+ properties:
+ result:
+ description: The stored value. When 'isBinary' is
+ true, this is a base64 string that must be
+ decoded to get the actual value.
+ codec:
+ type: string
+ description: Name of the codec 'result' was encoded
+ with, e.g. "json" (default) or "binary"
+ isBinary:
+ type: boolean
+ description: Whether the stored value is binary
+ data
+ 400:
+ description: Value was stored with a codec unavailable over the API
+ 404:
+ description: Extension not found
+ """
+ extension_id = f"{publisher}.{name}"
+ extension = get_extension_or_404(extension_id)
+ if not extension:
+ return self.response_404("Extension not found")
+
+ shared = request.args.get("shared", "false").lower() == "true"
+ raw = ExtensionEphemeralDAO.get_raw(extension_id, key, shared=shared)
+ if raw is None:
+ return self.response(200, result=None)
+ value, codec = raw
+ if codec not in SAFE_CODECS:
+ return self.response_400(
+ f"Value was stored with codec '{codec}', which cannot be "
+ "read over the REST API."
+ )
+
+ result, is_binary = _decoded_result_for_wire(get_codec(codec).decode(value))
+ return self.response(200, result=result, codec=codec, isBinary=is_binary)
+
+ @protect()
+ @safe
+ @expose("/<publisher>/<name>/storage/ephemeral/<key>", methods=("PUT",))
+ def set_ephemeral(
+ self, publisher: str, name: str, key: str, **kwargs: Any
+ ) -> Response:
+ """Set a value in ephemeral state.
+ ---
+ put:
+ summary: Set a value in ephemeral state
+ parameters:
+ - in: path
+ name: publisher
+ schema:
+ type: string
+ required: true
+ description: Extension publisher
+ - in: path
+ name: name
+ schema:
+ type: string
+ required: true
+ description: Extension name
+ - in: path
+ name: key
+ schema:
+ type: string
+ required: true
+ description: Storage key
+ - in: query
+ name: shared
+ schema:
+ type: boolean
+ required: false
+ description: If true, store as shared state visible to all users
+ requestBody:
+ required: true
+ content:
+ application/json:
+ schema:
+ type: object
+ required:
+ - value
+ - ttl
+ properties:
+ value:
+ description: The value to store (must not exceed
+ MAX_VALUE_SIZE bytes once encoded with 'codec').
+ When 'isBinary' is true, this must be a base64
+ string, decoded before being handed to 'codec'.
+ codec:
+ type: string
+ description: Name of the codec used to encode 'value',
+ e.g. "json" (default). Must be one of the codecs
+ allowed over the REST API.
+ isBinary:
+ type: boolean
+ description: Whether 'value' is binary data
+ ttl:
+ type: integer
+ description: Time-to-live in seconds (must be a positive
+ integer not exceeding MAX_TTL)
+ responses:
+ 200:
+ description: Value stored successfully
+ 400:
+ description: Invalid request body, or codec not allowed over the API
+ 404:
+ description: Extension not found
+ """
+ extension_id = f"{publisher}.{name}"
+ extension = get_extension_or_404(extension_id)
+ if not extension:
+ return self.response_404("Extension not found")
+
+ body = request.get_json(silent=True) or {}
+ if "value" not in body:
+ return self.response_400("Request body must contain 'value' field")
+
+ codec = body.get("codec", DEFAULT_CODEC)
+ if codec not in SAFE_CODECS:
+ return self.response_400(
+ f"Codec '{codec}' is not allowed over the REST API."
+ )
Review Comment:
Fixed β added a `not isinstance(codec, str)` check to the guard in `set_ephemeral`, so a non-string `codec` now returns a clean 400 instead of raising `TypeError` on the `SAFE_CODECS` membership check.
##########
superset/extensions/storage/api.py:
##########
@@ -0,0 +1,732 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""
+REST API for extension storage.
+
+Provides HTTP endpoints for frontend extensions to access server-side
+ephemeral storage without direct backend code.
+
+All operations are user-scoped by default. Use `?shared=true` query param
+to access shared state visible to all users.
+"""
+
+from __future__ import annotations
+
+import base64
+from typing import Any
+
+from flask import g, request
+from flask.wrappers import Response
+from flask_appbuilder.api import BaseApi, expose, protect, safe
+
+from superset.extensions.storage.codecs import DEFAULT_CODEC, get_codec, SAFE_CODECS
+from superset.extensions.storage.ephemeral_dao import (
+ ExtensionEphemeralDAO,
+ ExtensionEphemeralTTLInvalid,
+ ExtensionEphemeralValueTooLarge,
+)
+from superset.extensions.storage.persistent_dao import (
+ ExtensionStorageDAO,
+ ExtensionStorageKeyTooLong,
+ ExtensionStorageListPayloadTooLarge,
+ ExtensionStorageQuotaExceeded,
+ ExtensionStorageValueTooLarge,
+)
+from superset.extensions.storage.utils import get_extension_or_404, parse_ttl
+from superset.utils.decorators import transaction
+
+
+def _decoded_result_for_wire(decoded: Any) -> tuple[Any, bool]:
+ """Convert a codec-decoded value into its JSON wire representation.
+
+ JSON has no byte type, so a raw `bytes` value is base64-encoded to a
+ string for the response and flagged as such; every other value is
+ already JSON-representable as-is. Checked on the decoded value's
+ actual type, not the codec's name, so this keeps working for a codec
+ this module didn't define.
+
+ :returns: (wire_value, is_binary)
+ """
+ if isinstance(decoded, bytes):
+ return base64.b64encode(decoded).decode("ascii"), True
+ return decoded, False
+
+
+def _wire_value_for_request(value: Any, is_binary: bool) -> Any:
+ """Convert a request body's JSON `value` into a codec's input type.
+
+ `is_binary` is an explicit flag from the caller: JSON has no byte
+ type, so there is no way to tell, from the JSON `value` alone,
+ whether it is a base64 string that must be decoded to bytes before
+ being handed to the codec's `encode`, or a literal value to pass
+ through as-is. Only the caller knows which it sent.
+
+ :raises ValueError: if `is_binary` is set and `value` is not a valid
+ base64 string.
+ """
+ if is_binary:
+ return base64.b64decode(value, validate=True)
+ return value
+
+
+class ExtensionStorageRestApi(BaseApi):
+ """REST API for extension ephemeral state storage."""
+
+ allow_browser_login = True
+ route_base = "/api/v1/extensions"
+
+ def response(self, status_code: int, **kwargs: Any) -> Response:
+ """Helper method to create JSON responses."""
+ from flask import jsonify
+
+ return jsonify(kwargs), status_code
+
+ def response_404(self, message: str = "Not found") -> Response:
+ """Helper method to create 404 responses."""
+ from flask import jsonify
+
+ return jsonify({"message": message}), 404
+
+ def response_400(self, message: str) -> Response:
+ """Helper method to create 400 responses."""
+ from flask import jsonify
+
+ return jsonify({"message": message}), 400
+
+ @protect()
+ @safe
+ @expose("/<publisher>/<name>/storage/ephemeral/<key>", methods=("GET",))
+ def get_ephemeral(
+ self, publisher: str, name: str, key: str, **kwargs: Any
+ ) -> Response:
+ """Get a value from ephemeral state.
+ ---
+ get:
+ summary: Get a value from ephemeral state
+ parameters:
+ - in: path
+ name: publisher
+ schema:
+ type: string
+ required: true
+ description: Extension publisher
+ - in: path
+ name: name
+ schema:
+ type: string
+ required: true
+ description: Extension name
+ - in: path
+ name: key
+ schema:
+ type: string
+ required: true
+ description: Storage key
+ - in: query
+ name: shared
+ schema:
+ type: boolean
+ required: false
+ description: If true, read from shared state visible to all users
+ responses:
+ 200:
+ description: Value retrieved successfully
+ content:
+ application/json:
+ schema:
+ type: object
+ properties:
+ result:
+ description: The stored value. When 'isBinary' is
+ true, this is a base64 string that must be
+ decoded to get the actual value.
+ codec:
+ type: string
+ description: Name of the codec 'result' was encoded
+ with, e.g. "json" (default) or "binary"
+ isBinary:
+ type: boolean
+ description: Whether the stored value is binary
+ data
+ 400:
+ description: Value was stored with a codec unavailable over the API
+ 404:
+ description: Extension not found
+ """
+ extension_id = f"{publisher}.{name}"
+ extension = get_extension_or_404(extension_id)
+ if not extension:
+ return self.response_404("Extension not found")
+
+ shared = request.args.get("shared", "false").lower() == "true"
+ raw = ExtensionEphemeralDAO.get_raw(extension_id, key, shared=shared)
+ if raw is None:
+ return self.response(200, result=None)
+ value, codec = raw
+ if codec not in SAFE_CODECS:
+ return self.response_400(
+ f"Value was stored with codec '{codec}', which cannot be "
+ "read over the REST API."
+ )
+
+ result, is_binary = _decoded_result_for_wire(get_codec(codec).decode(value))
+ return self.response(200, result=result, codec=codec, isBinary=is_binary)
+
+ @protect()
+ @safe
+ @expose("/<publisher>/<name>/storage/ephemeral/<key>", methods=("PUT",))
+ def set_ephemeral(
+ self, publisher: str, name: str, key: str, **kwargs: Any
+ ) -> Response:
+ """Set a value in ephemeral state.
+ ---
+ put:
+ summary: Set a value in ephemeral state
+ parameters:
+ - in: path
+ name: publisher
+ schema:
+ type: string
+ required: true
+ description: Extension publisher
+ - in: path
+ name: name
+ schema:
+ type: string
+ required: true
+ description: Extension name
+ - in: path
+ name: key
+ schema:
+ type: string
+ required: true
+ description: Storage key
+ - in: query
+ name: shared
+ schema:
+ type: boolean
+ required: false
+ description: If true, store as shared state visible to all users
+ requestBody:
+ required: true
+ content:
+ application/json:
+ schema:
+ type: object
+ required:
+ - value
+ - ttl
+ properties:
+ value:
+ description: The value to store (must not exceed
+ MAX_VALUE_SIZE bytes once encoded with 'codec').
+ When 'isBinary' is true, this must be a base64
+ string, decoded before being handed to 'codec'.
+ codec:
+ type: string
+ description: Name of the codec used to encode 'value',
+ e.g. "json" (default). Must be one of the codecs
+ allowed over the REST API.
+ isBinary:
+ type: boolean
+ description: Whether 'value' is binary data
+ ttl:
+ type: integer
+ description: Time-to-live in seconds (must be a positive
+ integer not exceeding MAX_TTL)
+ responses:
+ 200:
+ description: Value stored successfully
+ 400:
+ description: Invalid request body, or codec not allowed over the API
+ 404:
+ description: Extension not found
+ """
+ extension_id = f"{publisher}.{name}"
+ extension = get_extension_or_404(extension_id)
+ if not extension:
+ return self.response_404("Extension not found")
+
+ body = request.get_json(silent=True) or {}
+ if "value" not in body:
+ return self.response_400("Request body must contain 'value' field")
+
+ codec = body.get("codec", DEFAULT_CODEC)
+ if codec not in SAFE_CODECS:
+ return self.response_400(
+ f"Codec '{codec}' is not allowed over the REST API."
+ )
+
+ is_binary = bool(body.get("isBinary", False))
+ try:
+ value = _wire_value_for_request(body["value"], is_binary)
+ except (ValueError, TypeError):
+ return self.response_400(
+ "Value must be a valid base64 string when 'isBinary' is true."
+ )
+ ttl, error = parse_ttl(body)
+ if error:
+ return self.response_400(error)
+
+ shared = request.args.get("shared", "false").lower() == "true"
+ try:
+ ExtensionEphemeralDAO.set(
+ extension_id, key, value, ttl, codec=codec, shared=shared
+ )
+ except (ExtensionEphemeralTTLInvalid, ExtensionEphemeralValueTooLarge) as ex:
+ return self.response_400(ex.message)
+
+ return self.response(200, message="Value stored successfully")
+
+ @protect()
+ @safe
+ @expose("/<publisher>/<name>/storage/ephemeral/<key>", methods=("DELETE",))
+ def delete_ephemeral(
+ self, publisher: str, name: str, key: str, **kwargs: Any
+ ) -> Response:
+ """Delete a value from ephemeral state.
+ ---
+ delete:
+ summary: Delete a value from ephemeral state
+ parameters:
+ - in: path
+ name: publisher
+ schema:
+ type: string
+ required: true
+ description: Extension publisher
+ - in: path
+ name: name
+ schema:
+ type: string
+ required: true
+ description: Extension name
+ - in: path
+ name: key
+ schema:
+ type: string
+ required: true
+ description: Storage key
+ - in: query
+ name: shared
+ schema:
+ type: boolean
+ required: false
+ description: If true, delete from shared state
+ responses:
+ 200:
+ description: Value deleted successfully
+ 404:
+ description: Extension not found
+ """
+ extension_id = f"{publisher}.{name}"
+ extension = get_extension_or_404(extension_id)
+ if not extension:
+ return self.response_404("Extension not found")
+
+ shared = request.args.get("shared", "false").lower() == "true"
+ ExtensionEphemeralDAO.delete(extension_id, key, shared=shared)
+
+ return self.response(200, message="Value deleted successfully")
+
+ @protect()
+ @safe
+ @expose("/<publisher>/<name>/storage/persistent", methods=("GET",))
+ def list_persistent(self, publisher: str, name: str, **kwargs: Any) -> Response:
+ """List entries in persistent state.
+ ---
+ get:
+ summary: List entries in persistent state
+ parameters:
+ - in: path
+ name: publisher
+ schema:
+ type: string
+ required: true
+ description: Extension publisher
+ - in: path
+ name: name
+ schema:
+ type: string
+ required: true
+ description: Extension name
+ - in: query
+ name: shared
+ schema:
+ type: boolean
+ required: false
+ description: If true, list shared state visible to all users
+ - in: query
+ name: resource_type
+ schema:
+ type: string
+ required: false
+ description: Filter by resource type
+ - in: query
+ name: resource_uuid
+ schema:
+ type: string
+ required: false
+ description: Filter by resource UUID (requires resource_type)
+ - in: query
+ name: page
+ schema:
+ type: integer
+ required: false
+ description: Zero-indexed page number. Defaults to 0.
+ - in: query
+ name: page_size
+ schema:
+ type: integer
+ required: false
+ description: Number of entries per page. Defaults to 10. There
+ is no fixed ceiling, but a page whose combined value size
+ exceeds MAX_LIST_PAYLOAD_SIZE is rejected β reduce page_size
+ and retry if that happens.
+ responses:
+ 200:
+ description: Entries retrieved successfully
+ content:
+ application/json:
+ schema:
+ type: object
+ properties:
+ result:
+ type: array
+ items:
+ type: object
+ properties:
+ key:
+ type: string
+ value:
+ description: The stored value, or null if its
+ codec cannot be read over the REST API.
+ When 'isBinary' is true, this is a base64
+ string that must be decoded to get the
+ actual value.
+ codec:
+ type: string
+ isBinary:
+ type: boolean
+ description: Whether the stored value is
+ binary data
+ count:
+ type: integer
+ description: Total number of entries matching the
+ given scope/filters, across all pages
+ 400:
+ description: The requested page's combined value size exceeds
+ MAX_LIST_PAYLOAD_SIZE
+ 404:
+ description: Extension not found
+ """
+ extension_id = f"{publisher}.{name}"
+ extension = get_extension_or_404(extension_id)
+ if not extension:
+ return self.response_404("Extension not found")
+
+ shared = request.args.get("shared", "false").lower() == "true"
+ user_fk = None if shared else g.user.id
+ resource_type = request.args.get("resource_type")
+ resource_uuid = request.args.get("resource_uuid")
+ try:
+ page = int(request.args.get("page", 0))
+ page_size = int(request.args.get("page_size", 10))
+ except (TypeError, ValueError):
+ return self.response_400("'page' and 'page_size' must be integers")
Review Comment:
Not fixing β this is already handled: `ExtensionStorageDAO.list_entries()` clamps both (`page = max(page, 0)`, `page_size = max(page_size, 1)`) before they're used for pagination, so negative values can't reach the DAO's offset/limit. Confirmed by reading the DAO.
##########
superset/extensions/storage/api.py:
##########
@@ -0,0 +1,732 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""
+REST API for extension storage.
+
+Provides HTTP endpoints for frontend extensions to access server-side
+ephemeral storage without direct backend code.
+
+All operations are user-scoped by default. Use `?shared=true` query param
+to access shared state visible to all users.
+"""
+
+from __future__ import annotations
+
+import base64
+from typing import Any
+
+from flask import g, request
+from flask.wrappers import Response
+from flask_appbuilder.api import BaseApi, expose, protect, safe
+
+from superset.extensions.storage.codecs import DEFAULT_CODEC, get_codec, SAFE_CODECS
+from superset.extensions.storage.ephemeral_dao import (
+ ExtensionEphemeralDAO,
+ ExtensionEphemeralTTLInvalid,
+ ExtensionEphemeralValueTooLarge,
+)
+from superset.extensions.storage.persistent_dao import (
+ ExtensionStorageDAO,
+ ExtensionStorageKeyTooLong,
+ ExtensionStorageListPayloadTooLarge,
+ ExtensionStorageQuotaExceeded,
+ ExtensionStorageValueTooLarge,
+)
+from superset.extensions.storage.utils import get_extension_or_404, parse_ttl
+from superset.utils.decorators import transaction
+
+
+def _decoded_result_for_wire(decoded: Any) -> tuple[Any, bool]:
+ """Convert a codec-decoded value into its JSON wire representation.
+
+ JSON has no byte type, so a raw `bytes` value is base64-encoded to a
+ string for the response and flagged as such; every other value is
+ already JSON-representable as-is. Checked on the decoded value's
+ actual type, not the codec's name, so this keeps working for a codec
+ this module didn't define.
+
+ :returns: (wire_value, is_binary)
+ """
+ if isinstance(decoded, bytes):
+ return base64.b64encode(decoded).decode("ascii"), True
+ return decoded, False
+
+
+def _wire_value_for_request(value: Any, is_binary: bool) -> Any:
+ """Convert a request body's JSON `value` into a codec's input type.
+
+ `is_binary` is an explicit flag from the caller: JSON has no byte
+ type, so there is no way to tell, from the JSON `value` alone,
+ whether it is a base64 string that must be decoded to bytes before
+ being handed to the codec's `encode`, or a literal value to pass
+ through as-is. Only the caller knows which it sent.
+
+ :raises ValueError: if `is_binary` is set and `value` is not a valid
+ base64 string.
+ """
+ if is_binary:
+ return base64.b64decode(value, validate=True)
+ return value
+
+
+class ExtensionStorageRestApi(BaseApi):
+ """REST API for extension ephemeral state storage."""
+
+ allow_browser_login = True
+ route_base = "/api/v1/extensions"
+
+ def response(self, status_code: int, **kwargs: Any) -> Response:
+ """Helper method to create JSON responses."""
+ from flask import jsonify
+
+ return jsonify(kwargs), status_code
+
+ def response_404(self, message: str = "Not found") -> Response:
+ """Helper method to create 404 responses."""
+ from flask import jsonify
+
+ return jsonify({"message": message}), 404
+
+ def response_400(self, message: str) -> Response:
+ """Helper method to create 400 responses."""
+ from flask import jsonify
+
+ return jsonify({"message": message}), 400
+
+ @protect()
+ @safe
+ @expose("/<publisher>/<name>/storage/ephemeral/<key>", methods=("GET",))
+ def get_ephemeral(
+ self, publisher: str, name: str, key: str, **kwargs: Any
+ ) -> Response:
+ """Get a value from ephemeral state.
+ ---
+ get:
+ summary: Get a value from ephemeral state
+ parameters:
+ - in: path
+ name: publisher
+ schema:
+ type: string
+ required: true
+ description: Extension publisher
+ - in: path
+ name: name
+ schema:
+ type: string
+ required: true
+ description: Extension name
+ - in: path
+ name: key
+ schema:
+ type: string
+ required: true
+ description: Storage key
+ - in: query
+ name: shared
+ schema:
+ type: boolean
+ required: false
+ description: If true, read from shared state visible to all users
+ responses:
+ 200:
+ description: Value retrieved successfully
+ content:
+ application/json:
+ schema:
+ type: object
+ properties:
+ result:
+ description: The stored value. When 'isBinary' is
+ true, this is a base64 string that must be
+ decoded to get the actual value.
+ codec:
+ type: string
+ description: Name of the codec 'result' was encoded
+ with, e.g. "json" (default) or "binary"
+ isBinary:
+ type: boolean
+ description: Whether the stored value is binary
+ data
+ 400:
+ description: Value was stored with a codec unavailable over the API
+ 404:
+ description: Extension not found
+ """
+ extension_id = f"{publisher}.{name}"
+ extension = get_extension_or_404(extension_id)
+ if not extension:
+ return self.response_404("Extension not found")
+
+ shared = request.args.get("shared", "false").lower() == "true"
+ raw = ExtensionEphemeralDAO.get_raw(extension_id, key, shared=shared)
+ if raw is None:
+ return self.response(200, result=None)
+ value, codec = raw
+ if codec not in SAFE_CODECS:
+ return self.response_400(
+ f"Value was stored with codec '{codec}', which cannot be "
+ "read over the REST API."
+ )
+
+ result, is_binary = _decoded_result_for_wire(get_codec(codec).decode(value))
+ return self.response(200, result=result, codec=codec, isBinary=is_binary)
+
+ @protect()
+ @safe
+ @expose("/<publisher>/<name>/storage/ephemeral/<key>", methods=("PUT",))
+ def set_ephemeral(
+ self, publisher: str, name: str, key: str, **kwargs: Any
+ ) -> Response:
+ """Set a value in ephemeral state.
+ ---
+ put:
+ summary: Set a value in ephemeral state
+ parameters:
+ - in: path
+ name: publisher
+ schema:
+ type: string
+ required: true
+ description: Extension publisher
+ - in: path
+ name: name
+ schema:
+ type: string
+ required: true
+ description: Extension name
+ - in: path
+ name: key
+ schema:
+ type: string
+ required: true
+ description: Storage key
+ - in: query
+ name: shared
+ schema:
+ type: boolean
+ required: false
+ description: If true, store as shared state visible to all users
+ requestBody:
+ required: true
+ content:
+ application/json:
+ schema:
+ type: object
+ required:
+ - value
+ - ttl
+ properties:
+ value:
+ description: The value to store (must not exceed
+ MAX_VALUE_SIZE bytes once encoded with 'codec').
+ When 'isBinary' is true, this must be a base64
+ string, decoded before being handed to 'codec'.
+ codec:
+ type: string
+ description: Name of the codec used to encode 'value',
+ e.g. "json" (default). Must be one of the codecs
+ allowed over the REST API.
+ isBinary:
+ type: boolean
+ description: Whether 'value' is binary data
+ ttl:
+ type: integer
+ description: Time-to-live in seconds (must be a positive
+ integer not exceeding MAX_TTL)
+ responses:
+ 200:
+ description: Value stored successfully
+ 400:
+ description: Invalid request body, or codec not allowed over the API
+ 404:
+ description: Extension not found
+ """
+ extension_id = f"{publisher}.{name}"
+ extension = get_extension_or_404(extension_id)
+ if not extension:
+ return self.response_404("Extension not found")
+
+ body = request.get_json(silent=True) or {}
+ if "value" not in body:
+ return self.response_400("Request body must contain 'value' field")
+
+ codec = body.get("codec", DEFAULT_CODEC)
+ if codec not in SAFE_CODECS:
+ return self.response_400(
+ f"Codec '{codec}' is not allowed over the REST API."
+ )
+
+ is_binary = bool(body.get("isBinary", False))
+ try:
+ value = _wire_value_for_request(body["value"], is_binary)
+ except (ValueError, TypeError):
+ return self.response_400(
+ "Value must be a valid base64 string when 'isBinary' is true."
+ )
+ ttl, error = parse_ttl(body)
+ if error:
+ return self.response_400(error)
+
+ shared = request.args.get("shared", "false").lower() == "true"
+ try:
+ ExtensionEphemeralDAO.set(
+ extension_id, key, value, ttl, codec=codec, shared=shared
+ )
+ except (ExtensionEphemeralTTLInvalid, ExtensionEphemeralValueTooLarge) as ex:
+ return self.response_400(ex.message)
+
+ return self.response(200, message="Value stored successfully")
+
+ @protect()
+ @safe
+ @expose("/<publisher>/<name>/storage/ephemeral/<key>", methods=("DELETE",))
+ def delete_ephemeral(
+ self, publisher: str, name: str, key: str, **kwargs: Any
+ ) -> Response:
+ """Delete a value from ephemeral state.
+ ---
+ delete:
+ summary: Delete a value from ephemeral state
+ parameters:
+ - in: path
+ name: publisher
+ schema:
+ type: string
+ required: true
+ description: Extension publisher
+ - in: path
+ name: name
+ schema:
+ type: string
+ required: true
+ description: Extension name
+ - in: path
+ name: key
+ schema:
+ type: string
+ required: true
+ description: Storage key
+ - in: query
+ name: shared
+ schema:
+ type: boolean
+ required: false
+ description: If true, delete from shared state
+ responses:
+ 200:
+ description: Value deleted successfully
+ 404:
+ description: Extension not found
+ """
+ extension_id = f"{publisher}.{name}"
+ extension = get_extension_or_404(extension_id)
+ if not extension:
+ return self.response_404("Extension not found")
+
+ shared = request.args.get("shared", "false").lower() == "true"
+ ExtensionEphemeralDAO.delete(extension_id, key, shared=shared)
+
+ return self.response(200, message="Value deleted successfully")
+
+ @protect()
+ @safe
+ @expose("/<publisher>/<name>/storage/persistent", methods=("GET",))
+ def list_persistent(self, publisher: str, name: str, **kwargs: Any) -> Response:
+ """List entries in persistent state.
+ ---
+ get:
+ summary: List entries in persistent state
+ parameters:
+ - in: path
+ name: publisher
+ schema:
+ type: string
+ required: true
+ description: Extension publisher
+ - in: path
+ name: name
+ schema:
+ type: string
+ required: true
+ description: Extension name
+ - in: query
+ name: shared
+ schema:
+ type: boolean
+ required: false
+ description: If true, list shared state visible to all users
+ - in: query
+ name: resource_type
+ schema:
+ type: string
+ required: false
+ description: Filter by resource type
+ - in: query
+ name: resource_uuid
+ schema:
+ type: string
+ required: false
+ description: Filter by resource UUID (requires resource_type)
+ - in: query
+ name: page
+ schema:
+ type: integer
+ required: false
+ description: Zero-indexed page number. Defaults to 0.
+ - in: query
+ name: page_size
+ schema:
+ type: integer
+ required: false
+ description: Number of entries per page. Defaults to 10. There
+ is no fixed ceiling, but a page whose combined value size
+ exceeds MAX_LIST_PAYLOAD_SIZE is rejected β reduce page_size
+ and retry if that happens.
+ responses:
+ 200:
+ description: Entries retrieved successfully
+ content:
+ application/json:
+ schema:
+ type: object
+ properties:
+ result:
+ type: array
+ items:
+ type: object
+ properties:
+ key:
+ type: string
+ value:
+ description: The stored value, or null if its
+ codec cannot be read over the REST API.
+ When 'isBinary' is true, this is a base64
+ string that must be decoded to get the
+ actual value.
+ codec:
+ type: string
+ isBinary:
+ type: boolean
+ description: Whether the stored value is
+ binary data
+ count:
+ type: integer
+ description: Total number of entries matching the
+ given scope/filters, across all pages
+ 400:
+ description: The requested page's combined value size exceeds
+ MAX_LIST_PAYLOAD_SIZE
+ 404:
+ description: Extension not found
+ """
+ extension_id = f"{publisher}.{name}"
+ extension = get_extension_or_404(extension_id)
+ if not extension:
+ return self.response_404("Extension not found")
+
+ shared = request.args.get("shared", "false").lower() == "true"
+ user_fk = None if shared else g.user.id
+ resource_type = request.args.get("resource_type")
+ resource_uuid = request.args.get("resource_uuid")
+ try:
+ page = int(request.args.get("page", 0))
+ page_size = int(request.args.get("page_size", 10))
+ except (TypeError, ValueError):
+ return self.response_400("'page' and 'page_size' must be integers")
+
+ try:
+ entries, count = ExtensionStorageDAO.list_entries(
+ extension_id,
+ user_fk=user_fk,
+ resource_type=resource_type,
+ resource_uuid=resource_uuid,
+ page=page,
+ page_size=page_size,
+ )
+ except ExtensionStorageListPayloadTooLarge as ex:
+ return self.response(ex.status, message=ex.message)
+
+ result = []
+ for entry in entries:
+ value: Any = None
+ is_binary = False
+ if entry.codec in SAFE_CODECS and entry.value is not None:
+ decoded = get_codec(entry.codec).decode(entry.value)
+ value, is_binary = _decoded_result_for_wire(decoded)
+ result.append(
+ {
+ "key": entry.key,
+ "value": value,
+ "codec": entry.codec,
+ "isBinary": is_binary,
+ }
+ )
+
+ return self.response(200, result=result, count=count)
+
+ @protect()
+ @safe
+ @expose("/<publisher>/<name>/storage/persistent/<key>", methods=("GET",))
+ def get_persistent(
+ self, publisher: str, name: str, key: str, **kwargs: Any
+ ) -> Response:
+ """Get a value from persistent state.
+ ---
+ get:
+ summary: Get a value from persistent state
+ parameters:
+ - in: path
+ name: publisher
+ schema:
+ type: string
+ required: true
+ description: Extension publisher
+ - in: path
+ name: name
+ schema:
+ type: string
+ required: true
+ description: Extension name
+ - in: path
+ name: key
+ schema:
+ type: string
+ required: true
+ description: Storage key
+ - in: query
+ name: shared
+ schema:
+ type: boolean
+ required: false
+ description: If true, read from shared state visible to all users
+ responses:
+ 200:
+ description: Value retrieved successfully
+ content:
+ application/json:
+ schema:
+ type: object
+ properties:
+ result:
+ description: The stored value. When 'isBinary' is
+ true, this is a base64 string that must be
+ decoded to get the actual value.
+ codec:
+ type: string
+ description: Name of the codec 'result' was encoded
+ with, e.g. "json" (default) or "binary"
+ isBinary:
+ type: boolean
+ description: Whether the stored value is binary
+ data
+ 400:
+ description: Value was stored with a codec unavailable over the API
+ 404:
+ description: Extension not found
+ """
+ extension_id = f"{publisher}.{name}"
+ extension = get_extension_or_404(extension_id)
+ if not extension:
+ return self.response_404("Extension not found")
+
+ shared = request.args.get("shared", "false").lower() == "true"
+ user_fk = None if shared else g.user.id
+ entry = ExtensionStorageDAO.get(extension_id, key, user_fk=user_fk)
+ if entry is None:
+ return self.response(200, result=None)
+ if entry.codec not in SAFE_CODECS:
+ return self.response_400(
+ f"Value was stored with codec '{entry.codec}', which cannot be "
+ "read over the REST API."
+ )
+
+ decoded = ExtensionStorageDAO.get_decoded_value(
+ extension_id, key, user_fk=user_fk
+ )
+ result, is_binary = _decoded_result_for_wire(decoded)
+
+ return self.response(200, result=result, codec=entry.codec, isBinary=is_binary)
+
+ @protect()
+ @safe
+ @expose("/<publisher>/<name>/storage/persistent/<key>", methods=("PUT",))
+ @transaction()
+ def set_persistent(
+ self, publisher: str, name: str, key: str, **kwargs: Any
+ ) -> Response:
+ """Set a value in persistent state.
+ ---
+ put:
+ summary: Set a value in persistent state
+ parameters:
+ - in: path
+ name: publisher
+ schema:
+ type: string
+ required: true
+ description: Extension publisher
+ - in: path
+ name: name
+ schema:
+ type: string
+ required: true
+ description: Extension name
+ - in: path
+ name: key
+ schema:
+ type: string
+ required: true
+ description: Storage key
+ - in: query
+ name: shared
+ schema:
+ type: boolean
+ required: false
+ description: If true, store as shared state visible to all users
+ requestBody:
+ required: true
+ content:
+ application/json:
+ schema:
+ type: object
+ required:
+ - value
+ properties:
+ value:
+ description: The value to store. When 'isBinary' is
+ true, this must be a base64 string, decoded
+ before being handed to 'codec'.
+ codec:
+ type: string
+ description: Name of the codec used to encode 'value',
+ e.g. "json" (default). Must be one of the codecs
+ allowed over the REST API.
+ isBinary:
+ type: boolean
+ description: Whether 'value' is binary data
+ encrypt:
+ type: boolean
+ description: If true, the value is encrypted at rest
+ responses:
+ 200:
+ description: Value stored successfully
+ 400:
+ description: Invalid request body, codec not allowed over the
+ API, or value exceeds MAX_VALUE_SIZE
+ 404:
+ description: Extension not found
+ 413:
+ description: Extension persistent storage quota exceeded
+ """
+ extension_id = f"{publisher}.{name}"
+ extension = get_extension_or_404(extension_id)
+ if not extension:
+ return self.response_404("Extension not found")
+
+ body = request.get_json(silent=True) or {}
+ if "value" not in body:
+ return self.response_400("Request body must contain 'value' field")
+
+ codec = body.get("codec", DEFAULT_CODEC)
+ if codec not in SAFE_CODECS:
+ return self.response_400(
+ f"Codec '{codec}' is not allowed over the REST API."
+ )
Review Comment:
Fixed β same fix as the `set_ephemeral` case (non-string `codec` check), applied here to `set_persistent`.
##########
superset/extensions/storage/api.py:
##########
@@ -0,0 +1,732 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""
+REST API for extension storage.
+
+Provides HTTP endpoints for frontend extensions to access server-side
+ephemeral storage without direct backend code.
+
+All operations are user-scoped by default. Use `?shared=true` query param
+to access shared state visible to all users.
+"""
+
+from __future__ import annotations
+
+import base64
+from typing import Any
+
+from flask import g, request
+from flask.wrappers import Response
+from flask_appbuilder.api import BaseApi, expose, protect, safe
+
+from superset.extensions.storage.codecs import DEFAULT_CODEC, get_codec, SAFE_CODECS
+from superset.extensions.storage.ephemeral_dao import (
+ ExtensionEphemeralDAO,
+ ExtensionEphemeralTTLInvalid,
+ ExtensionEphemeralValueTooLarge,
+)
+from superset.extensions.storage.persistent_dao import (
+ ExtensionStorageDAO,
+ ExtensionStorageKeyTooLong,
+ ExtensionStorageListPayloadTooLarge,
+ ExtensionStorageQuotaExceeded,
+ ExtensionStorageValueTooLarge,
+)
+from superset.extensions.storage.utils import get_extension_or_404, parse_ttl
+from superset.utils.decorators import transaction
+
+
+def _decoded_result_for_wire(decoded: Any) -> tuple[Any, bool]:
+ """Convert a codec-decoded value into its JSON wire representation.
+
+ JSON has no byte type, so a raw `bytes` value is base64-encoded to a
+ string for the response and flagged as such; every other value is
+ already JSON-representable as-is. Checked on the decoded value's
+ actual type, not the codec's name, so this keeps working for a codec
+ this module didn't define.
+
+ :returns: (wire_value, is_binary)
+ """
+ if isinstance(decoded, bytes):
+ return base64.b64encode(decoded).decode("ascii"), True
+ return decoded, False
+
+
+def _wire_value_for_request(value: Any, is_binary: bool) -> Any:
+ """Convert a request body's JSON `value` into a codec's input type.
+
+ `is_binary` is an explicit flag from the caller: JSON has no byte
+ type, so there is no way to tell, from the JSON `value` alone,
+ whether it is a base64 string that must be decoded to bytes before
+ being handed to the codec's `encode`, or a literal value to pass
+ through as-is. Only the caller knows which it sent.
+
+ :raises ValueError: if `is_binary` is set and `value` is not a valid
+ base64 string.
+ """
+ if is_binary:
+ return base64.b64decode(value, validate=True)
+ return value
+
+
+class ExtensionStorageRestApi(BaseApi):
+ """REST API for extension ephemeral state storage."""
+
+ allow_browser_login = True
+ route_base = "/api/v1/extensions"
+
+ def response(self, status_code: int, **kwargs: Any) -> Response:
+ """Helper method to create JSON responses."""
+ from flask import jsonify
+
+ return jsonify(kwargs), status_code
+
+ def response_404(self, message: str = "Not found") -> Response:
+ """Helper method to create 404 responses."""
+ from flask import jsonify
+
+ return jsonify({"message": message}), 404
+
+ def response_400(self, message: str) -> Response:
+ """Helper method to create 400 responses."""
+ from flask import jsonify
+
+ return jsonify({"message": message}), 400
+
+ @protect()
+ @safe
+ @expose("/<publisher>/<name>/storage/ephemeral/<key>", methods=("GET",))
+ def get_ephemeral(
+ self, publisher: str, name: str, key: str, **kwargs: Any
+ ) -> Response:
+ """Get a value from ephemeral state.
+ ---
+ get:
+ summary: Get a value from ephemeral state
+ parameters:
+ - in: path
+ name: publisher
+ schema:
+ type: string
+ required: true
+ description: Extension publisher
+ - in: path
+ name: name
+ schema:
+ type: string
+ required: true
+ description: Extension name
+ - in: path
+ name: key
+ schema:
+ type: string
+ required: true
+ description: Storage key
+ - in: query
+ name: shared
+ schema:
+ type: boolean
+ required: false
+ description: If true, read from shared state visible to all users
+ responses:
+ 200:
+ description: Value retrieved successfully
+ content:
+ application/json:
+ schema:
+ type: object
+ properties:
+ result:
+ description: The stored value. When 'isBinary' is
+ true, this is a base64 string that must be
+ decoded to get the actual value.
+ codec:
+ type: string
+ description: Name of the codec 'result' was encoded
+ with, e.g. "json" (default) or "binary"
+ isBinary:
+ type: boolean
+ description: Whether the stored value is binary
+ data
+ 400:
+ description: Value was stored with a codec unavailable over the API
+ 404:
+ description: Extension not found
+ """
+ extension_id = f"{publisher}.{name}"
+ extension = get_extension_or_404(extension_id)
+ if not extension:
+ return self.response_404("Extension not found")
+
+ shared = request.args.get("shared", "false").lower() == "true"
+ raw = ExtensionEphemeralDAO.get_raw(extension_id, key, shared=shared)
+ if raw is None:
+ return self.response(200, result=None)
+ value, codec = raw
+ if codec not in SAFE_CODECS:
+ return self.response_400(
+ f"Value was stored with codec '{codec}', which cannot be "
+ "read over the REST API."
+ )
+
+ result, is_binary = _decoded_result_for_wire(get_codec(codec).decode(value))
+ return self.response(200, result=result, codec=codec, isBinary=is_binary)
+
+ @protect()
+ @safe
+ @expose("/<publisher>/<name>/storage/ephemeral/<key>", methods=("PUT",))
+ def set_ephemeral(
+ self, publisher: str, name: str, key: str, **kwargs: Any
+ ) -> Response:
+ """Set a value in ephemeral state.
+ ---
+ put:
+ summary: Set a value in ephemeral state
+ parameters:
+ - in: path
+ name: publisher
+ schema:
+ type: string
+ required: true
+ description: Extension publisher
+ - in: path
+ name: name
+ schema:
+ type: string
+ required: true
+ description: Extension name
+ - in: path
+ name: key
+ schema:
+ type: string
+ required: true
+ description: Storage key
+ - in: query
+ name: shared
+ schema:
+ type: boolean
+ required: false
+ description: If true, store as shared state visible to all users
+ requestBody:
+ required: true
+ content:
+ application/json:
+ schema:
+ type: object
+ required:
+ - value
+ - ttl
+ properties:
+ value:
+ description: The value to store (must not exceed
+ MAX_VALUE_SIZE bytes once encoded with 'codec').
+ When 'isBinary' is true, this must be a base64
+ string, decoded before being handed to 'codec'.
+ codec:
+ type: string
+ description: Name of the codec used to encode 'value',
+ e.g. "json" (default). Must be one of the codecs
+ allowed over the REST API.
+ isBinary:
+ type: boolean
+ description: Whether 'value' is binary data
+ ttl:
+ type: integer
+ description: Time-to-live in seconds (must be a positive
+ integer not exceeding MAX_TTL)
+ responses:
+ 200:
+ description: Value stored successfully
+ 400:
+ description: Invalid request body, or codec not allowed over the API
+ 404:
+ description: Extension not found
+ """
+ extension_id = f"{publisher}.{name}"
+ extension = get_extension_or_404(extension_id)
+ if not extension:
+ return self.response_404("Extension not found")
+
+ body = request.get_json(silent=True) or {}
+ if "value" not in body:
+ return self.response_400("Request body must contain 'value' field")
+
+ codec = body.get("codec", DEFAULT_CODEC)
+ if codec not in SAFE_CODECS:
+ return self.response_400(
+ f"Codec '{codec}' is not allowed over the REST API."
+ )
+
+ is_binary = bool(body.get("isBinary", False))
+ try:
+ value = _wire_value_for_request(body["value"], is_binary)
+ except (ValueError, TypeError):
+ return self.response_400(
+ "Value must be a valid base64 string when 'isBinary' is true."
+ )
+ ttl, error = parse_ttl(body)
+ if error:
+ return self.response_400(error)
+
+ shared = request.args.get("shared", "false").lower() == "true"
+ try:
+ ExtensionEphemeralDAO.set(
+ extension_id, key, value, ttl, codec=codec, shared=shared
+ )
+ except (ExtensionEphemeralTTLInvalid, ExtensionEphemeralValueTooLarge) as ex:
+ return self.response_400(ex.message)
+
+ return self.response(200, message="Value stored successfully")
+
+ @protect()
+ @safe
+ @expose("/<publisher>/<name>/storage/ephemeral/<key>", methods=("DELETE",))
+ def delete_ephemeral(
+ self, publisher: str, name: str, key: str, **kwargs: Any
+ ) -> Response:
+ """Delete a value from ephemeral state.
+ ---
+ delete:
+ summary: Delete a value from ephemeral state
+ parameters:
+ - in: path
+ name: publisher
+ schema:
+ type: string
+ required: true
+ description: Extension publisher
+ - in: path
+ name: name
+ schema:
+ type: string
+ required: true
+ description: Extension name
+ - in: path
+ name: key
+ schema:
+ type: string
+ required: true
+ description: Storage key
+ - in: query
+ name: shared
+ schema:
+ type: boolean
+ required: false
+ description: If true, delete from shared state
+ responses:
+ 200:
+ description: Value deleted successfully
+ 404:
+ description: Extension not found
+ """
+ extension_id = f"{publisher}.{name}"
+ extension = get_extension_or_404(extension_id)
+ if not extension:
+ return self.response_404("Extension not found")
+
+ shared = request.args.get("shared", "false").lower() == "true"
+ ExtensionEphemeralDAO.delete(extension_id, key, shared=shared)
+
+ return self.response(200, message="Value deleted successfully")
+
+ @protect()
+ @safe
+ @expose("/<publisher>/<name>/storage/persistent", methods=("GET",))
+ def list_persistent(self, publisher: str, name: str, **kwargs: Any) -> Response:
+ """List entries in persistent state.
+ ---
+ get:
+ summary: List entries in persistent state
+ parameters:
+ - in: path
+ name: publisher
+ schema:
+ type: string
+ required: true
+ description: Extension publisher
+ - in: path
+ name: name
+ schema:
+ type: string
+ required: true
+ description: Extension name
+ - in: query
+ name: shared
+ schema:
+ type: boolean
+ required: false
+ description: If true, list shared state visible to all users
+ - in: query
+ name: resource_type
+ schema:
+ type: string
+ required: false
+ description: Filter by resource type
+ - in: query
+ name: resource_uuid
+ schema:
+ type: string
+ required: false
+ description: Filter by resource UUID (requires resource_type)
+ - in: query
+ name: page
+ schema:
+ type: integer
+ required: false
+ description: Zero-indexed page number. Defaults to 0.
+ - in: query
+ name: page_size
+ schema:
+ type: integer
+ required: false
+ description: Number of entries per page. Defaults to 10. There
+ is no fixed ceiling, but a page whose combined value size
+ exceeds MAX_LIST_PAYLOAD_SIZE is rejected β reduce page_size
+ and retry if that happens.
+ responses:
+ 200:
+ description: Entries retrieved successfully
+ content:
+ application/json:
+ schema:
+ type: object
+ properties:
+ result:
+ type: array
+ items:
+ type: object
+ properties:
+ key:
+ type: string
+ value:
+ description: The stored value, or null if its
+ codec cannot be read over the REST API.
+ When 'isBinary' is true, this is a base64
+ string that must be decoded to get the
+ actual value.
+ codec:
+ type: string
+ isBinary:
+ type: boolean
+ description: Whether the stored value is
+ binary data
+ count:
+ type: integer
+ description: Total number of entries matching the
+ given scope/filters, across all pages
+ 400:
+ description: The requested page's combined value size exceeds
+ MAX_LIST_PAYLOAD_SIZE
+ 404:
+ description: Extension not found
+ """
+ extension_id = f"{publisher}.{name}"
+ extension = get_extension_or_404(extension_id)
+ if not extension:
+ return self.response_404("Extension not found")
+
+ shared = request.args.get("shared", "false").lower() == "true"
+ user_fk = None if shared else g.user.id
+ resource_type = request.args.get("resource_type")
+ resource_uuid = request.args.get("resource_uuid")
+ try:
+ page = int(request.args.get("page", 0))
+ page_size = int(request.args.get("page_size", 10))
+ except (TypeError, ValueError):
+ return self.response_400("'page' and 'page_size' must be integers")
+
+ try:
+ entries, count = ExtensionStorageDAO.list_entries(
+ extension_id,
+ user_fk=user_fk,
+ resource_type=resource_type,
+ resource_uuid=resource_uuid,
+ page=page,
+ page_size=page_size,
+ )
+ except ExtensionStorageListPayloadTooLarge as ex:
+ return self.response(ex.status, message=ex.message)
+
+ result = []
+ for entry in entries:
+ value: Any = None
+ is_binary = False
+ if entry.codec in SAFE_CODECS and entry.value is not None:
+ decoded = get_codec(entry.codec).decode(entry.value)
+ value, is_binary = _decoded_result_for_wire(decoded)
+ result.append(
+ {
+ "key": entry.key,
+ "value": value,
+ "codec": entry.codec,
+ "isBinary": is_binary,
+ }
+ )
+
+ return self.response(200, result=result, count=count)
+
+ @protect()
+ @safe
+ @expose("/<publisher>/<name>/storage/persistent/<key>", methods=("GET",))
+ def get_persistent(
+ self, publisher: str, name: str, key: str, **kwargs: Any
+ ) -> Response:
+ """Get a value from persistent state.
+ ---
+ get:
+ summary: Get a value from persistent state
+ parameters:
+ - in: path
+ name: publisher
+ schema:
+ type: string
+ required: true
+ description: Extension publisher
+ - in: path
+ name: name
+ schema:
+ type: string
+ required: true
+ description: Extension name
+ - in: path
+ name: key
+ schema:
+ type: string
+ required: true
+ description: Storage key
+ - in: query
+ name: shared
+ schema:
+ type: boolean
+ required: false
+ description: If true, read from shared state visible to all users
+ responses:
+ 200:
+ description: Value retrieved successfully
+ content:
+ application/json:
+ schema:
+ type: object
+ properties:
+ result:
+ description: The stored value. When 'isBinary' is
+ true, this is a base64 string that must be
+ decoded to get the actual value.
+ codec:
+ type: string
+ description: Name of the codec 'result' was encoded
+ with, e.g. "json" (default) or "binary"
+ isBinary:
+ type: boolean
+ description: Whether the stored value is binary
+ data
+ 400:
+ description: Value was stored with a codec unavailable over the API
+ 404:
+ description: Extension not found
+ """
+ extension_id = f"{publisher}.{name}"
+ extension = get_extension_or_404(extension_id)
+ if not extension:
+ return self.response_404("Extension not found")
+
+ shared = request.args.get("shared", "false").lower() == "true"
+ user_fk = None if shared else g.user.id
+ entry = ExtensionStorageDAO.get(extension_id, key, user_fk=user_fk)
+ if entry is None:
+ return self.response(200, result=None)
+ if entry.codec not in SAFE_CODECS:
+ return self.response_400(
+ f"Value was stored with codec '{entry.codec}', which cannot be "
+ "read over the REST API."
+ )
+
+ decoded = ExtensionStorageDAO.get_decoded_value(
+ extension_id, key, user_fk=user_fk
+ )
+ result, is_binary = _decoded_result_for_wire(decoded)
+
+ return self.response(200, result=result, codec=entry.codec, isBinary=is_binary)
+
+ @protect()
+ @safe
+ @expose("/<publisher>/<name>/storage/persistent/<key>", methods=("PUT",))
+ @transaction()
+ def set_persistent(
+ self, publisher: str, name: str, key: str, **kwargs: Any
+ ) -> Response:
+ """Set a value in persistent state.
+ ---
+ put:
+ summary: Set a value in persistent state
+ parameters:
+ - in: path
+ name: publisher
+ schema:
+ type: string
+ required: true
+ description: Extension publisher
+ - in: path
+ name: name
+ schema:
+ type: string
+ required: true
+ description: Extension name
+ - in: path
+ name: key
+ schema:
+ type: string
+ required: true
+ description: Storage key
+ - in: query
+ name: shared
+ schema:
+ type: boolean
+ required: false
+ description: If true, store as shared state visible to all users
+ requestBody:
+ required: true
+ content:
+ application/json:
+ schema:
+ type: object
+ required:
+ - value
+ properties:
+ value:
+ description: The value to store. When 'isBinary' is
+ true, this must be a base64 string, decoded
+ before being handed to 'codec'.
+ codec:
+ type: string
+ description: Name of the codec used to encode 'value',
+ e.g. "json" (default). Must be one of the codecs
+ allowed over the REST API.
+ isBinary:
+ type: boolean
+ description: Whether 'value' is binary data
+ encrypt:
+ type: boolean
+ description: If true, the value is encrypted at rest
+ responses:
+ 200:
+ description: Value stored successfully
+ 400:
+ description: Invalid request body, codec not allowed over the
+ API, or value exceeds MAX_VALUE_SIZE
+ 404:
+ description: Extension not found
+ 413:
+ description: Extension persistent storage quota exceeded
+ """
+ extension_id = f"{publisher}.{name}"
+ extension = get_extension_or_404(extension_id)
+ if not extension:
+ return self.response_404("Extension not found")
+
+ body = request.get_json(silent=True) or {}
+ if "value" not in body:
+ return self.response_400("Request body must contain 'value' field")
+
+ codec = body.get("codec", DEFAULT_CODEC)
+ if codec not in SAFE_CODECS:
+ return self.response_400(
+ f"Codec '{codec}' is not allowed over the REST API."
+ )
+
+ is_binary = bool(body.get("isBinary", False))
+ encrypt = bool(body.get("encrypt", False))
+ shared = request.args.get("shared", "false").lower() == "true"
+ user_fk = None if shared else g.user.id
+ try:
+ wire_value = _wire_value_for_request(body["value"], is_binary)
+ except (ValueError, TypeError):
+ return self.response_400(
+ "Value must be a valid base64 string when 'isBinary' is true."
+ )
+ value_bytes = get_codec(codec).encode(wire_value)
Review Comment:
Fixed β wrapped `get_codec(codec).encode(wire_value)` in a try/except catching `KeyValueCodecEncodeException`/`TypeError`/`ValueError`, returning a 400 instead of letting it 500.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(extensions): add a Storage API for extensions [superset]
Posted by "matebenyovszky (via GitHub)" <gi...@apache.org>.
matebenyovszky commented on PR #39171:
URL: https://github.com/apache/superset/pull/39171#issuecomment-4396480403
I have added these fixes, if someone with appropiate permission could add - or should I create a separate PR?
curl -sL https://gist.githubusercontent.com/matebenyovszky/f64b21f895a34289171fd18fe381e85b/raw/fix-storage-urls-and-add-tests.patch | git am
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(extensions): add a Storage API for extensions [superset]
Posted by "matebenyovszky (via GitHub)" <gi...@apache.org>.
matebenyovszky commented on PR #39171:
URL: https://github.com/apache/superset/pull/39171#issuecomment-4378904263
### Patch available
Here's the complete patch as a gist β you can apply it directly:
`ash
curl -sL https://gist.githubusercontent.com/matebenyovszky/f64b21f895a34289171fd18fe381e85b/raw/fix-storage-urls-and-add-tests.patch | git am
`
Gist: https://gist.github.com/matebenyovszky/f64b21f895a34289171fd18fe381e85b
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(extensions): add a Storage API for extensions [superset]
Posted by "matebenyovszky (via GitHub)" <gi...@apache.org>.
matebenyovszky commented on PR #39171:
URL: https://github.com/apache/superset/pull/39171#issuecomment-4439896837
Hi @michael-s-molina β I've opened a PR directly against your `extensions-storage-tiers-1-2` branch with the bug fix and tests:
**PR: https://github.com/michael-s-molina/superset/pull/775**
### What's in it:
**Bug fix** β Frontend `ephemeralState.ts` and `persistentState.ts` URL builders don't match the backend's `/{publisher}/{name}/storage/{tier}/{key}` routes (introduced in f08bcd4). All frontend storage API calls currently 404.
**Tests (472 lines)** β Unit tests for all three storage modules (`localState`, `ephemeralState`, `persistentState`) covering CRUD, shared access, URL encoding, and validation. Should resolve the `codecov/project/core-packages-ts` CI failure.
### What I think is still needed to merge #39171:
1. β
**Frontend URL bug** β fixed in my PR
2. β
**Frontend tests** β included in my PR
3. β¬ **Backend Python tests** β `api.py` at 0%, `persistent_state_dao.py` at 32%, `ephemeral_state.py` at 45% (happy to write these too)
4. β¬ **Mark as Ready for Review** β still Draft
5. β¬ **Re-review from @villebro** β earlier comments were all addressed
I noticed @rusackas just referenced this PR from `michael-s-molina/superset#774` β the extensions ecosystem is moving fast and Storage API is a foundational piece. Let me know if there's anything else I can do to help! π
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(extensions): add a Storage API for extensions [superset]
Posted by "matebenyovszky (via GitHub)" <gi...@apache.org>.
matebenyovszky commented on PR #39171:
URL: https://github.com/apache/superset/pull/39171#issuecomment-4833719300
@michael-s-molina β quick note: the **only CI failure left is a trivial `prettier` formatting issue** in `ephemeralState.ts` (two `set` method signatures need line-breaking).
I've opened a 1-line PR against your branch: https://github.com/michael-s-molina/superset/pull/811
Alternatively, from `superset-frontend/`:
```bash
npx prettier --write src/core/storage/ephemeralState.ts
git add -u && git commit --trailer "Co-authored-by: Cursor <cu...@cursor.com>" -m 'Fix prettier formatting' && git push
```
Once that's in, **every CI check is green** and this can be marked **Ready for Review**. Would be awesome to get this over the finish line β it's so close! π
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(extensions): add a Storage API for extensions [superset]
Posted by "bito-code-review[bot] (via GitHub)" <gi...@apache.org>.
bito-code-review[bot] commented on PR #39171:
URL: https://github.com/apache/superset/pull/39171#issuecomment-5038611313
**Bito Automatic Review Skipped β PR Already Merged**<br/><br/>Bito scheduled an automatic review for this pull request, but the review was skipped because this PR was merged before the review could be run.<br/>No action is needed if you didn't intend to review it. To get a review, you can type `/review` in a comment and save it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(extensions): add a Storage API for extensions [superset]
Posted by "codeant-ai-for-open-source[bot] (via GitHub)" <gi...@apache.org>.
codeant-ai-for-open-source[bot] commented on code in PR #39171:
URL: https://github.com/apache/superset/pull/39171#discussion_r3622833625
##########
superset-core/src/superset_core/extensions/context.py:
##########
@@ -0,0 +1,136 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""
+Extension Context API for superset-core extensions.
+
+Provides access to the current extension's context, including metadata
+and scoped resources like storage. Extensions call `get_context()` to
+access their context during execution.
+
+The context is set by the host (Superset) during extension loading and
+is only available within extension code.
+
+Usage:
+ from superset_core.extensions.context import get_context
+
+ def setup():
+ ctx = get_context()
+
+ # Access extension metadata
+ print(f"Running {ctx.extension.displayName} v{ctx.extension.version}")
+
+ # Access extension-scoped storage
+ ctx.storage.ephemeral.set("lastRun", time.time())
+ data = ctx.storage.ephemeral.get("cachedData")
Review Comment:
**Suggestion:** The usage examples call ephemeral `set` without required options, which contradicts the implemented Tier 2 API and will cause runtime `TypeError` when copied by users. Update the examples to pass `EphemeralSetOptions(ttl=...)` so documentation matches executable behavior. [docstring mismatch]
<details>
<summary><b>Severity Level:</b> Minor π§Ή</summary>
```mdx
- β οΈ Docstring example matches visible Python protocol signature.
- β οΈ No concrete ephemeral storage implementation in this PR.
- β οΈ Example not runnable; get_context raises immediately.
- β οΈ Suggestion targets mismatch with unseen external API.
```
</details>
<details>
<summary><b>Steps of Reproduction β
</b></summary>
```mdx
1. Inspect the module docstring example in
`superset-core/src/superset_core/extensions/context.py:31-40`; it shows
`ctx.storage.ephemeral.set("lastRun", time.time())`.
2. Compare this example with the `StorageAccessor.set` protocol at lines 57-59, which
accepts `(key: str, value: Any, ttl: int = 3600)` and makes the `ttl` argument optional
via a default.
3. Inspect `get_context()` at lines 104-133; it raises `NotImplementedError`, so the
example is not executable against any real storage implementation within this module.
4. No Python definition of `EphemeralSetOptions` or a different ephemeral `set` signature
exists in the provided code, so the example call shape matches the visible protocol; the
claimed runtime `TypeError` cannot be reproduced from actual code paths here.
```
</details>
[](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=a84006645daa47758cb9d4c21a506e98&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=a84006645daa47758cb9d4c21a506e98&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset)
*(Use Cmd/Ctrl + Click for best experience)*
<details>
<summary><b>Prompt for AI Agent π€ </b></summary>
```mdx
This is a comment left during a code review.
**Path:** superset-core/src/superset_core/extensions/context.py
**Line:** 37:39
**Comment:**
*Docstring Mismatch: The usage examples call ephemeral `set` without required options, which contradicts the implemented Tier 2 API and will cause runtime `TypeError` when copied by users. Update the examples to pass `EphemeralSetOptions(ttl=...)` so documentation matches executable behavior.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F39171&comment_hash=7b3785e7103641df87aba1976b05318c63e2bc8121d22d6b518de6967410eca0&reaction=like'>π</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F39171&comment_hash=7b3785e7103641df87aba1976b05318c63e2bc8121d22d6b518de6967410eca0&reaction=dislike'>π</a>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(extensions): add a Storage API for extensions [superset]
Posted by "codeant-ai-for-open-source[bot] (via GitHub)" <gi...@apache.org>.
codeant-ai-for-open-source[bot] commented on code in PR #39171:
URL: https://github.com/apache/superset/pull/39171#discussion_r3624430154
##########
superset-core/src/superset_core/extensions/context.py:
##########
@@ -0,0 +1,117 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""
+Extension Context API for superset-core extensions.
+
+Provides access to the current extension's context, including metadata
+and scoped resources like storage. Extensions call `get_context()` to
+access their context during execution.
+
+The context is set by the host (Superset) during extension loading and
+is only available within extension code.
+
+Usage:
+ from superset_core.extensions.context import get_context
+
+ def setup():
+ ctx = get_context()
+
+ # Access extension metadata
+ print(f"Running {ctx.extension.displayName} v{ctx.extension.version}")
+
+ # Access extension-scoped storage
+ ctx.storage.ephemeral.set("lastRun", time.time(), EphemeralSetOptions(ttl=3600))
+ data = ctx.storage.ephemeral.get("cachedData")
+"""
+
+from __future__ import annotations
+
+from typing import Protocol, TYPE_CHECKING
+
+if TYPE_CHECKING:
+ from superset_core.extensions.storage.ephemeral import EphemeralStateAccessor
+ from superset_core.extensions.storage.persistent import PersistentStateAccessor
+ from superset_core.extensions.types import BaseExtension
+
+
+class ExtensionStorage(Protocol):
+ """Extension-scoped storage accessor for all available tiers."""
+
+ @property
+ def ephemeral(self) -> "EphemeralStateAccessor":
+ """Server-side cache (Redis/Memcached) with TTL."""
+ ...
+
+ @property
+ def persistent(self) -> "PersistentStateAccessor":
+ """Database-backed persistent storage."""
+ ...
+
+
+class ExtensionContext(Protocol):
+ """
+ Context object providing extension-specific resources.
+
+ This context is only available during extension execution.
+ Calling `get_context()` outside of an extension will raise an error.
+ """
+
+ @property
+ def extension(self) -> "BaseExtension":
+ """Metadata about the current extension."""
+ ...
+
+ @property
+ def storage(self) -> ExtensionStorage:
+ """Extension-scoped storage across all available tiers."""
+ ...
+
+
+def get_context() -> ExtensionContext:
+ """
+ Get the current extension's context.
+
+ This function returns the context for the currently executing extension,
+ providing access to extension metadata and scoped resources like storage.
+
+ Host implementations will replace this function during initialization
+ with a concrete implementation providing actual functionality.
+
+ :returns: The current extension's context.
+ :raises RuntimeError: If called outside of an extension context.
+
+ Example:
+ from superset_core.extensions.context import get_context
+
+ ctx = get_context()
+
+ # Access extension metadata
+ print(f"Extension: {ctx.extension.id}")
+ print(f"Version: {ctx.extension.version}")
+
+ # Access extension-scoped storage
+ ctx.storage.ephemeral.set("tempData", data, ttl=3600)
Review Comment:
**Suggestion:** The usage example passes `ttl` as a direct keyword argument, but the storage API uses an options object for write configuration; copying this example will raise a runtime argument error in extension code. Update the example to use the correct `set` call shape. [docstring mismatch]
<details>
<summary><b>Severity Level:</b> Major β οΈ</summary>
```mdx
- β οΈ Backend extensions using docstring example get TypeError.
- β οΈ Tier 2 ephemeral storage writes fail for copied pattern.
- β οΈ Developer onboarding hindered by misleading storage documentation.
```
</details>
<details>
<summary><b>Steps of Reproduction β
</b></summary>
```mdx
1. A backend extension author imports `get_context()` from
`superset_core.extensions.context` and follows the example in the docstring of
`get_context()` at `superset-core/src/superset_core/extensions/context.py:85-113`,
specifically the line `ctx.storage.ephemeral.set("tempData", data, ttl=3600)` at line 108.
2. In their extension backend module (for example
`superset/extensions/my_extension/backend.py`), they write:
`ctx = get_context()`
`ctx.storage.ephemeral.set("tempData", data, ttl=3600)`
mirroring the docstring example instead of the earlier correct example using
`EphemeralSetOptions` at `superset-core/src/superset_core/extensions/context.py:38-39`.
3. At runtime, the host implementation wires `ctx.storage.ephemeral` to an
`EphemeralStateAccessor` instance defined in
`superset-core/src/superset_core/extensions/storage/ephemeral.py:68`, whose `set()` method
expects a third argument of type `EphemeralSetOptions` (options object) and does not
declare a `ttl` keyword parameter.
4. When the extension code invokes `ctx.storage.ephemeral.set("tempData", data, ttl=3600)`
the Python runtime attempts to pass `ttl` as a keyword argument to
`EphemeralStateAccessor.set()`, which lacks such a parameter, causing a `TypeError`
(unexpected keyword argument) and failing the extensionβs storage write.
```
</details>
[](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=e347bb9d335e4fd6ab019b02a0fa11c8&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=e347bb9d335e4fd6ab019b02a0fa11c8&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset)
*(Use Cmd/Ctrl + Click for best experience)*
<details>
<summary><b>Prompt for AI Agent π€ </b></summary>
```mdx
This is a comment left during a code review.
**Path:** superset-core/src/superset_core/extensions/context.py
**Line:** 108:108
**Comment:**
*Docstring Mismatch: The usage example passes `ttl` as a direct keyword argument, but the storage API uses an options object for write configuration; copying this example will raise a runtime argument error in extension code. Update the example to use the correct `set` call shape.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F39171&comment_hash=d5f6a4b5bdc5f1a3e28a8aeaec8aa74299a7a5114d9ce80607eee009b0230484&reaction=like'>π</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F39171&comment_hash=d5f6a4b5bdc5f1a3e28a8aeaec8aa74299a7a5114d9ce80607eee009b0230484&reaction=dislike'>π</a>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(extensions): add a Storage API for extensions [superset]
Posted by "michael-s-molina (via GitHub)" <gi...@apache.org>.
michael-s-molina commented on code in PR #39171:
URL: https://github.com/apache/superset/pull/39171#discussion_r3560959772
##########
superset/extensions/storage/persistent.py:
##########
@@ -0,0 +1,193 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""
+Host implementation for Persistent State (Tier 3 Storage).
+
+Provides the concrete database-backed implementation that is injected into
+superset_core.extensions.storage.persistent at startup.
+"""
+
+from __future__ import annotations
+
+from typing import Any, ClassVar
+
+from flask import g
+from superset_core.extensions.storage.persistent import (
+ PersistentSetOptions,
+ PersistentState as CorePersistentState,
+)
+
+from superset.extensions.context import get_current_extension_context
+from superset.extensions.storage.persistent_dao import ExtensionStorageDAO
+from superset.utils import json
+from superset.utils.decorators import transaction
+
+
+def _get_extension_id() -> str:
+ """Get the current extension ID from context."""
+ context = get_current_extension_context()
+ if context is None:
+ raise RuntimeError(
+ "persistent_state can only be used within an extension context. "
+ "Ensure this code is being executed during extension loading or "
+ "within an extension API request handler."
+ )
+ return context.manifest.id
+
+
+def _get_current_user_id() -> int:
+ """Get the current authenticated user's ID."""
+ user = getattr(g, "user", None)
Review Comment:
Done
##########
superset/extensions/storage/ephemeral.py:
##########
@@ -0,0 +1,181 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""
+Host implementation for Ephemeral State (Tier 2 Storage).
+
+Provides the concrete cache-backed implementation that is injected into
+superset_core.extensions.storage.ephemeral at startup.
+"""
+
+from __future__ import annotations
+
+from typing import Any, ClassVar
+
+from flask import g
+from superset_core.extensions.storage.ephemeral import (
+ EphemeralSetOptions,
+ EphemeralState as CoreEphemeralState,
+)
+
+from superset.extensions import cache_manager
+from superset.extensions.context import get_current_extension_context
+
+# Key separator
+SEPARATOR = ":"
+
+# Key prefix for extension ephemeral state
+KEY_PREFIX = "superset-ext"
+
+
+def _get_extension_id() -> str:
+ """Get the current extension ID from context."""
+ context = get_current_extension_context()
+ if context is None:
+ raise RuntimeError(
+ "ephemeral_state can only be used within an extension context. "
+ "Ensure this code is being executed during extension loading or "
+ "within an extension API request handler."
+ )
+ return context.manifest.id
+
+
+def _get_current_user_id() -> int:
+ """Get the current authenticated user's ID."""
+ user = getattr(g, "user", None)
+ if user is None or not hasattr(user, "id"):
+ raise RuntimeError(
+ "ephemeral_state requires an authenticated user. "
+ "Ensure the request has been authenticated."
+ )
+ return user.id
+
+
+def _build_cache_key(*parts: Any) -> str:
+ """Build a namespaced cache key from parts."""
+ return SEPARATOR.join(str(part) for part in parts)
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(extensions): add a Storage API for extensions [superset]
Posted by "codeant-ai-for-open-source[bot] (via GitHub)" <gi...@apache.org>.
codeant-ai-for-open-source[bot] commented on code in PR #39171:
URL: https://github.com/apache/superset/pull/39171#discussion_r3561223191
##########
superset-frontend/src/core/storage/ephemeralState.test.ts:
##########
@@ -0,0 +1,144 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import { SupersetClient } from '@superset-ui/core';
+import { createEphemeralState } from './ephemeralState';
+
+jest.mock('@superset-ui/core', () => ({
+ SupersetClient: {
+ get: jest.fn(),
+ put: jest.fn(),
+ delete: jest.fn(),
+ },
+}));
+
+const mockGet = SupersetClient.get as jest.Mock;
+const mockPut = SupersetClient.put as jest.Mock;
+const mockDelete = SupersetClient.delete as jest.Mock;
+
+beforeEach(() => {
+ jest.clearAllMocks();
+ mockGet.mockResolvedValue({ json: { result: null } });
+ mockPut.mockResolvedValue({});
+ mockDelete.mockResolvedValue({});
+});
+
+test('get calls correct URL with publisher/name pattern', async () => {
+ const store = createEphemeralState('myorg.myext');
+ await store.get('job_progress');
+ expect(mockGet).toHaveBeenCalledWith({
+ endpoint: '/api/v1/extensions/myorg/myext/storage/ephemeral/job_progress',
+ });
+});
+
+test('get returns result from response', async () => {
+ mockGet.mockResolvedValue({ json: { result: { pct: 42 } } });
+ const store = createEphemeralState('myorg.myext');
+ const result = await store.get('job_progress');
+ expect(result).toEqual({ pct: 42 });
+});
+
+test('get returns null when result is absent', async () => {
+ mockGet.mockResolvedValue({ json: {} });
+ const store = createEphemeralState('myorg.myext');
+ expect(await store.get('key')).toBeNull();
+});
+
+test('set calls correct URL and includes value and ttl in body', async () => {
+ const store = createEphemeralState('myorg.myext');
+ await store.set('job_progress', { pct: 42 }, { ttl: 300 });
+ expect(mockPut).toHaveBeenCalledWith({
+ endpoint: '/api/v1/extensions/myorg/myext/storage/ephemeral/job_progress',
+ body: JSON.stringify({ value: { pct: 42 }, ttl: 300, codec: 'json' }),
+ headers: { 'Content-Type': 'application/json' },
+ });
+});
+
+test('set passes codec from options', async () => {
+ const store = createEphemeralState('myorg.myext');
+ await store.set('job_progress', 'sk-...', { ttl: 300, codec: 'pickle' });
+ expect(mockPut).toHaveBeenCalledWith({
+ endpoint: '/api/v1/extensions/myorg/myext/storage/ephemeral/job_progress',
+ body: JSON.stringify({ value: 'sk-...', ttl: 300, codec: 'pickle' }),
+ headers: { 'Content-Type': 'application/json' },
+ });
Review Comment:
β
**Customized review instruction saved!**
**Instruction:**
> Do not flag client-side SDK/storage tests that verify request-payload passthrough of an explicitly provided codec; server-side SAFE_CODECS validation is a separate concern.
**Applied to:**
- `superset-frontend/src/core/storage/**`
---
π‘ *To manage or update this instruction, visit: [CodeAnt AI Settings](https://app.codeant.ai/org/settings/learnings)*
##########
superset-frontend/src/core/storage/persistentState.test.ts:
##########
@@ -0,0 +1,249 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import { SupersetClient } from '@superset-ui/core';
+import { createPersistentState } from './persistentState';
+
+jest.mock('@superset-ui/core', () => ({
+ SupersetClient: {
+ get: jest.fn(),
+ put: jest.fn(),
+ delete: jest.fn(),
+ },
+}));
+
+const mockGet = SupersetClient.get as jest.Mock;
+const mockPut = SupersetClient.put as jest.Mock;
+const mockDelete = SupersetClient.delete as jest.Mock;
+
+beforeEach(() => {
+ jest.clearAllMocks();
+ mockGet.mockResolvedValue({ json: { result: null } });
+ mockPut.mockResolvedValue({});
+ mockDelete.mockResolvedValue({});
+});
+
+test('get calls correct URL with publisher/name pattern', async () => {
+ const store = createPersistentState('myorg.myext');
+ await store.get('prefs');
+ expect(mockGet).toHaveBeenCalledWith({
+ endpoint: '/api/v1/extensions/myorg/myext/storage/persistent/prefs',
+ });
+});
+
+test('get returns result from response', async () => {
+ mockGet.mockResolvedValue({ json: { result: { theme: 'dark' } } });
+ const store = createPersistentState('myorg.myext');
+ expect(await store.get('prefs')).toEqual({ theme: 'dark' });
+});
+
+test('get returns null when result is absent', async () => {
+ mockGet.mockResolvedValue({ json: {} });
+ const store = createPersistentState('myorg.myext');
+ expect(await store.get('key')).toBeNull();
+});
+
+test('set calls correct URL with value in body', async () => {
+ const store = createPersistentState('myorg.myext');
+ await store.set('prefs', { theme: 'dark' });
+ expect(mockPut).toHaveBeenCalledWith({
+ endpoint: '/api/v1/extensions/myorg/myext/storage/persistent/prefs',
+ body: JSON.stringify({
+ value: { theme: 'dark' },
+ encrypt: false,
+ codec: 'json',
+ }),
+ headers: { 'Content-Type': 'application/json' },
+ });
+});
+
+test('set passes codec from options', async () => {
+ const store = createPersistentState('myorg.myext');
+ await store.set('prefs', 'sk-...', { codec: 'pickle' });
+ expect(mockPut).toHaveBeenCalledWith({
+ endpoint: '/api/v1/extensions/myorg/myext/storage/persistent/prefs',
+ body: JSON.stringify({
+ value: 'sk-...',
+ encrypt: false,
+ codec: 'pickle',
+ }),
+ headers: { 'Content-Type': 'application/json' },
+ });
Review Comment:
β
**Customized review instruction saved!**
**Instruction:**
> Do not flag client-side codec passthrough behavior in SDK-layer tests; these tests validate the frontend contract with mocked backend calls, even if the real backend rejects the codec.
**Applied to:**
- `superset-frontend/src/core/storage/**`
---
π‘ *To manage or update this instruction, visit: [CodeAnt AI Settings](https://app.codeant.ai/org/settings/learnings)*
##########
superset-frontend/src/core/storage/persistentState.ts:
##########
@@ -0,0 +1,147 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import type {
+ JsonValue,
+ PersistentListOptions,
+ PersistentListResult,
+ PersistentSetOptions,
+ PersistentStorageAccessor,
+ PersistentStorageTier,
+} from '@apache-superset/core/storage';
+import { SupersetClient } from '@superset-ui/core';
+import { resolveSetPayload } from './binaryCodec';
+
+/**
+ * Create persistent state (database-backed) bound to an extension ID.
+ */
+export function createPersistentState(
+ extensionId: string,
+): PersistentStorageTier {
+ const MAX_KEY_LENGTH = 255;
+ const [publisher, name] = extensionId.split('.');
+
+ const buildUrl = (key: string, shared?: boolean): string => {
+ if (key.length > MAX_KEY_LENGTH) {
+ throw new Error(
+ `Persistent storage key must be ${MAX_KEY_LENGTH} characters or less.`,
+ );
+ }
+ const encodedPublisher = encodeURIComponent(publisher);
+ const encodedName = encodeURIComponent(name);
+ const encodedKey = encodeURIComponent(key);
+ const url = `/api/v1/extensions/${encodedPublisher}/${encodedName}/storage/persistent/${encodedKey}`;
+ return shared ? `${url}?shared=true` : url;
+ };
+
+ const buildListUrl = (
+ options: PersistentListOptions,
+ isShared?: boolean,
+ ): string => {
+ const encodedPublisher = encodeURIComponent(publisher);
+ const encodedName = encodeURIComponent(name);
+ const url = `/api/v1/extensions/${encodedPublisher}/${encodedName}/storage/persistent`;
+ const params = new URLSearchParams();
+ if (isShared) params.set('shared', 'true');
+ if (options.resourceType) params.set('resource_type', options.resourceType);
+ if (options.resourceUuid) params.set('resource_uuid', options.resourceUuid);
+ params.set('page', String(options.page));
+ params.set('page_size', String(options.pageSize));
+ const query = params.toString();
+ return query ? `${url}?${query}` : url;
+ };
+
+ const list = async <T = JsonValue>(
+ options: PersistentListOptions,
+ isShared: boolean,
+ ): Promise<PersistentListResult<T>> => {
+ const response = await SupersetClient.get({
+ endpoint: buildListUrl(options, isShared),
+ });
+ return {
+ entries: response.json?.result ?? [],
+ count: response.json?.count ?? 0,
+ };
+ };
+
+ const shared: PersistentStorageAccessor = {
+ async get<T = JsonValue>(key: string): Promise<T | null> {
+ const response = await SupersetClient.get({
+ endpoint: buildUrl(key, true),
+ });
+ return (response.json?.result ?? null) as T | null;
+ },
+ async set<T = JsonValue>(
+ key: string,
+ value: T,
+ options?: PersistentSetOptions,
+ ): Promise<void> {
+ const payload = resolveSetPayload(value, options?.codec);
+ await SupersetClient.put({
+ endpoint: buildUrl(key, true),
+ body: JSON.stringify({
+ value: payload.value,
+ encrypt: options?.encrypt ?? false,
+ codec: payload.codec,
+ }),
+ headers: { 'Content-Type': 'application/json' },
+ });
+ },
+ async list<T = JsonValue>(
+ options: PersistentListOptions,
+ ): Promise<PersistentListResult<T>> {
+ return list<T>(options, true);
+ },
+ async remove(key: string): Promise<void> {
+ await SupersetClient.delete({ endpoint: buildUrl(key, true) });
+ },
+ };
+
+ return {
+ async get<T = JsonValue>(key: string): Promise<T | null> {
+ const response = await SupersetClient.get({ endpoint: buildUrl(key) });
+ return (response.json?.result ?? null) as T | null;
+ },
+ async set<T = JsonValue>(
+ key: string,
+ value: T,
+ options?: PersistentSetOptions,
+ ): Promise<void> {
+ const payload = resolveSetPayload(value, options?.codec);
+ await SupersetClient.put({
+ endpoint: buildUrl(key),
+ body: JSON.stringify({
+ value: payload.value,
+ encrypt: options?.encrypt ?? false,
+ codec: payload.codec,
+ }),
+ headers: { 'Content-Type': 'application/json' },
+ });
Review Comment:
β
**Customized review instruction saved!**
**Instruction:**
> Do not duplicate backend-safe codec allowlists in frontend persistent storage APIs; let the server remain the source of truth and handle invalid codecs via its 400 response.
**Applied to:**
- `**/*.ts`
---
π‘ *To manage or update this instruction, visit: [CodeAnt AI Settings](https://app.codeant.ai/org/settings/learnings)*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(extensions): add a Storage API for extensions [superset]
Posted by "michael-s-molina (via GitHub)" <gi...@apache.org>.
michael-s-molina commented on code in PR #39171:
URL: https://github.com/apache/superset/pull/39171#discussion_r3560999616
##########
superset/extensions/context.py:
##########
@@ -16,75 +16,124 @@
# under the License.
"""
-Extension Context Management - provides ambient context during extension loading.
+Extension Context Management - provides ambient context for extensions.
-This module provides a thread-local context system that allows decorators to
-automatically detect whether they are being called in host or extension code
-during extension loading.
+This module provides a context system using Python's contextvars that allows
+extensions to access their context (metadata and scoped resources) via get_context().
+
+The context is set during extension loading and when extension callbacks are invoked.
+Uses ContextVar for thread-safe and async-safe context management with automatic
+save/restore for nested contexts.
"""
from __future__ import annotations
-import contextlib
-from threading import local
-from typing import Any, Generator
+from contextlib import contextmanager
+from contextvars import ContextVar
+from typing import Any, Iterator
from superset_core.extensions.types import Manifest
-# Thread-local storage for extension context
-_extension_context: local = local()
-
-class ExtensionContext:
- """Manages ambient extension context during loading."""
+class ExtensionStorage:
+ """Extension storage with all available tiers."""
- def __init__(self, manifest: Manifest):
- self.manifest = manifest
+ @property
+ def ephemeral(self) -> Any:
+ from superset.extensions.storage.ephemeral import EphemeralState
- def __enter__(self) -> "ExtensionContext":
- if getattr(_extension_context, "current", None) is not None:
- current_extension = _extension_context.current.manifest.id
- raise RuntimeError(
- f"Cannot initialize extension {self.manifest.id} while extension "
- f"{current_extension} is already being initialized. "
- f"Nested extension initialization is not supported."
- )
+ return EphemeralState
- _extension_context.current = self
- return self
+ @property
+ def persistent(self) -> Any:
+ from superset.extensions.storage.persistent import PersistentState
- def __exit__(self, exc_type: Any, exc_val: Any, exc_tb: Any) -> None:
- # Clear the current context
- _extension_context.current = None
+ return PersistentState
-class ExtensionContextWrapper:
- """Wrapper for extension context with extensible properties."""
+class ConcreteExtensionContext:
+ """Concrete implementation of ExtensionContext for the host."""
def __init__(self, manifest: Manifest):
self._manifest = manifest
+ self._storage = ExtensionStorage()
+
+ @property
+ def extension(self) -> Manifest:
+ """Extension metadata (new API)."""
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(extensions): add a Storage API for extensions [superset]
Posted by "codeant-ai-for-open-source[bot] (via GitHub)" <gi...@apache.org>.
codeant-ai-for-open-source[bot] commented on code in PR #39171:
URL: https://github.com/apache/superset/pull/39171#discussion_r3561095641
##########
superset-frontend/packages/superset-core/src/storage/index.ts:
##########
@@ -0,0 +1,208 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/**
+ * @fileoverview Shared types for extension storage APIs.
+ *
+ * These types are shared across all storage tiers (local, session, ephemeral,
+ * persistent) to ensure a consistent API pattern.
+ */
+
+/**
+ * JSON-compatible value type.
+ * These are the only values that can be safely serialized/deserialized via JSON.
+ */
+export type JsonValue =
+ string | number | boolean | null | JsonValue[] | { [key: string]: JsonValue };
+
+/**
+ * Base interface for a storage accessor.
+ * All storage tiers implement this interface for both user-scoped and shared access.
+ */
+export interface StorageAccessor {
+ /**
+ * Get a value from storage.
+ *
+ * @param key The key to retrieve.
+ * @returns The stored value, or null if not found.
+ */
+ get<T = JsonValue>(key: string): Promise<T | null>;
+
+ /**
+ * Set a value in storage.
+ *
+ * @param key The key to store.
+ * @param value The value to store (must be JSON-serializable).
+ */
+ set<T = JsonValue>(key: string, value: T): Promise<void>;
Review Comment:
**Suggestion:** The API contract says values must be JSON-serializable, but the unconstrained generic type accepts any value at compile time, so callers can pass unserializable inputs that fail at runtime in storage implementations. Constrain the generic to JSON-compatible (and explicitly supported binary types where applicable) to enforce the documented contract and prevent invalid writes. [api mismatch]
<details>
<summary><b>Severity Level:</b> Major β οΈ</summary>
```mdx
- β Ephemeral storage writes fail for unserializable values.
- β οΈ Extensions see confusing 400s from storage API.
```
</details>
<details>
<summary><b>Steps of Reproduction β
</b></summary>
```mdx
1. Extension authors use the public storage API exposed via `extensions.getContext()` in
`superset-frontend/packages/superset-core/src/extensions/index.ts:80-98`, calling
`ctx.storage.ephemeral.set('tempData', value, { ttl: 3600 })` as shown in the documented
example.
2. The `ctx.storage.ephemeral` accessor is typed as `EphemeralStorageTier`, whose `set`
method is declared generically as `set<T = JsonValue>(key: string, value: T, options:
EphemeralSetOptions): Promise<void>` in
`superset-frontend/packages/superset-core/src/storage/index.ts:132-138`, without
constraining `T` to `JsonValue`.
3. Because `T` is unconstrained, TypeScript happily infers `T = undefined` (or another
non-JSON type) when an extension calls `ctx.storage.ephemeral.set('tempData', undefined, {
ttl: 3600 })`; this compiles even though the JSDoc comment at
`superset-frontend/packages/superset-core/src/storage/index.ts:51-52` states the value
βmust be JSON-serializableβ.
4. At runtime, the host implementation `createEphemeralState` in
`superset-frontend/src/core/storage/ephemeralState.ts:72-82` passes the value into
`resolveSetPayload` and then into `SupersetClient.put`, building the JSON body with
`JSON.stringify({ value: payload.value, ttl: options.ttl, codec: payload.codec })`; when
`payload.value` is `undefined`, `JSON.stringify` drops the `value` field entirely, and the
backend endpoint `set_ephemeral` in `superset/extensions/storage/api.py:144-152` rejects
the request with `400` βRequest body must contain 'value' fieldβ, causing a runtime
failure despite the call type-checking.
```
</details>
[](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=f828cc10402740ad8df17a1a09a30937&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=f828cc10402740ad8df17a1a09a30937&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset)
*(Use Cmd/Ctrl + Click for best experience)*
<details>
<summary><b>Prompt for AI Agent π€ </b></summary>
```mdx
This is a comment left during a code review.
**Path:** superset-frontend/packages/superset-core/src/storage/index.ts
**Line:** 53:53
**Comment:**
*Api Mismatch: The API contract says values must be JSON-serializable, but the unconstrained generic type accepts any value at compile time, so callers can pass unserializable inputs that fail at runtime in storage implementations. Constrain the generic to JSON-compatible (and explicitly supported binary types where applicable) to enforce the documented contract and prevent invalid writes.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F39171&comment_hash=dec0468de2ada36ee0ef57803fb2d1d085a48e2fa73f90b9c8e4f429c31ba505&reaction=like'>π</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F39171&comment_hash=dec0468de2ada36ee0ef57803fb2d1d085a48e2fa73f90b9c8e4f429c31ba505&reaction=dislike'>π</a>
##########
superset-frontend/src/core/storage/localState.ts:
##########
@@ -0,0 +1,63 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import type { JsonValue, StorageAccessor } from '@apache-superset/core/storage';
+import getBootstrapData from 'src/utils/getBootstrapData';
+
+const KEY_PREFIX = 'superset-ext';
+
+function getCurrentUserId(): number {
+ const bootstrapData = getBootstrapData();
+ const userId = bootstrapData?.user?.userId;
+ if (userId === undefined) {
+ throw new Error('Storage APIs require an authenticated user.');
+ }
+ return userId;
+}
+
+function buildKey(...parts: (string | number)[]): string {
+ return [KEY_PREFIX, ...parts].join(':');
+}
+
+/**
+ * Create browser storage (localStorage/sessionStorage) bound to an extension ID.
+ */
+export function createBrowserStorage(
+ storage: Storage,
+ extensionId: string,
+): StorageAccessor {
+ return {
+ async get<T = JsonValue>(key: string): Promise<T | null> {
+ const userId = getCurrentUserId();
+ const storageKey = buildKey(extensionId, 'user', userId, key);
+ const value = storage.getItem(storageKey);
+ return value ? (JSON.parse(value) as T) : null;
+ },
+ async set<T = JsonValue>(key: string, value: T): Promise<void> {
+ const userId = getCurrentUserId();
+ const storageKey = buildKey(extensionId, 'user', userId, key);
+ storage.setItem(storageKey, JSON.stringify(value));
Review Comment:
**Suggestion:** `set` serializes with `JSON.stringify` and writes the result directly, but `JSON.stringify(undefined)` (and some unsupported values) returns non-JSON output that later makes `get` crash at `JSON.parse`. Reject non-serializable values before storing (or coerce to a valid JSON sentinel) so reads cannot fail on values written by the same API. [logic error]
<details>
<summary><b>Severity Level:</b> Major β οΈ</summary>
```mdx
- β Local/session storage reads crash on unparsable stored values.
- β οΈ Extensions may store undefined leading later runtime errors.
```
</details>
<details>
<summary><b>Steps of Reproduction β
</b></summary>
```mdx
1. Extensions obtain per-extension browser storage via `getContext().storage.local` and
`getContext().storage.session` as described in
`superset-frontend/packages/superset-core/src/extensions/index.ts:80-98`, using these
accessors to persist UI state and preferences.
2. The concrete implementation for these accessors is `createBrowserStorage` in
`superset-frontend/src/core/storage/localState.ts:41-63`, which returns a
`StorageAccessor` whose `set` method is declared generically as `async set<T =
JsonValue>(key: string, value: T): Promise<void>` but does not enforce at runtime that
`value` is actually JSON-serializable.
3. Because the generic `T` is unconstrained (definition in
`superset-frontend/packages/superset-core/src/storage/index.ts:38-53`), extension code can
successfully compile and call `await store.set('key', undefined)` or pass other
unsupported values; `createBrowserStorage.set` then executes `storage.setItem(storageKey,
JSON.stringify(value));` at `superset-frontend/src/core/storage/localState.ts:55`, which
for `value === undefined` stores the literal string `"undefined"` under the namespaced
key.
4. On a subsequent read, `createBrowserStorage.get` at
`superset-frontend/src/core/storage/localState.ts:46-50` retrieves the stored string and
calls `JSON.parse(value)`; when `value` is `"undefined"` (or any other non-JSON output
resulting from unsupported inputs), `JSON.parse` throws a `SyntaxError`, causing the
extensionβs code path that uses `ctx.storage.local.get` or `session.get` to crash even
though the values were written through the same API.
```
</details>
[](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=0820f1361b654c10ac8645a95a191bee&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=0820f1361b654c10ac8645a95a191bee&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset)
*(Use Cmd/Ctrl + Click for best experience)*
<details>
<summary><b>Prompt for AI Agent π€ </b></summary>
```mdx
This is a comment left during a code review.
**Path:** superset-frontend/src/core/storage/localState.ts
**Line:** 55:55
**Comment:**
*Logic Error: `set` serializes with `JSON.stringify` and writes the result directly, but `JSON.stringify(undefined)` (and some unsupported values) returns non-JSON output that later makes `get` crash at `JSON.parse`. Reject non-serializable values before storing (or coerce to a valid JSON sentinel) so reads cannot fail on values written by the same API.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F39171&comment_hash=65638f726d8667ea2ea233f3df822117f0b06b45251b49c3d2c0b2bb030c452f&reaction=like'>π</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F39171&comment_hash=65638f726d8667ea2ea233f3df822117f0b06b45251b49c3d2c0b2bb030c452f&reaction=dislike'>π</a>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(extensions): add Tier 1 and Tier 2 storage APIs [superset]
Posted by "michael-s-molina (via GitHub)" <gi...@apache.org>.
michael-s-molina commented on code in PR #39171:
URL: https://github.com/apache/superset/pull/39171#discussion_r3058624924
##########
docs/developer_docs/extensions/storage.md:
##########
@@ -0,0 +1,231 @@
+---
+title: Storage
+sidebar_position: 8
+---
+
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Storage
+
+Superset Extensions have access to a managed storage API for persisting data. The storage system provides multiple tiers with different persistence characteristics, allowing extensions to choose the right storage for their needs.
+
+Each extension receives its own isolated storage namespace. When Superset loads your extension, it binds storage to your extension's unique identifier, ensuring data privacyβtwo extensions using the same key will never collide, and extensions cannot access each other's data.
+
+## Storage Tiers
+
+| Tier | Storage Type | Context Property | Use Case |
+| ---- | ----------------- | ------------------------------------------ | ------------------------------------------------ |
+| 1 | Browser storage | `ctx.storage.local`, `ctx.storage.session` | UI state, wizard progress, draft forms |
+| 2 | Server-side cache | `ctx.storage.ephemeral` | Job progress, temporary results |
+| 3 | Database | `ctx.storage.persistent` | User preferences, extension config (coming soon) |
Review Comment:
Makes sense. I left some doc strings in the code though to guide the follow-up. Addressed in [0328e2d](https://github.com/apache/superset/pull/39171/commits/0328e2d19fe68d29501c9092d30aca1e04a640c1)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(extensions): add Tier 1 and Tier 2 storage APIs [superset]
Posted by "codeant-ai-for-open-source[bot] (via GitHub)" <gi...@apache.org>.
codeant-ai-for-open-source[bot] commented on code in PR #39171:
URL: https://github.com/apache/superset/pull/39171#discussion_r3047377024
##########
superset-frontend/packages/superset-core/src/storage/localState.ts:
##########
@@ -0,0 +1,150 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/**
+ * @fileoverview Local State API for Superset extensions (Tier 1 Storage).
+ *
+ * Provides client-side KV storage backed by the browser's localStorage.
+ * Data persists across browser sessions but is per-device (not shared across
+ * devices or synced to the server).
+ *
+ * By default, all operations are user-scoped (private to the current user).
+ * Use shared() to access state visible to all users on the same browser.
+ *
+ * Key patterns:
+ * - User-scoped (default): superset-ext:{extension_id}:user:{user_id}:{key}
+ * - Shared: superset-ext:{extension_id}:{key}
+ *
+ * The API is async to maintain compatibility with a future sandboxed execution
+ * model where storage calls would go through a postMessage bridge.
+ *
+ * @example
+ * ```typescript
+ * import { localState } from '@apache-superset/core/storage';
+ *
+ * // User-scoped state (default - private to current user)
+ * const isCollapsed = await localState.get('sidebar_collapsed');
+ * await localState.set('sidebar_collapsed', true);
+ * await localState.remove('sidebar_collapsed');
+ *
+ * // Shared state (visible to all users on same browser)
+ * const deviceId = await localState.shared().get('device_id');
+ * await localState.shared().set('device_id', 'abc-123');
+ * ```
+ */
+
+/**
+ * Interface for scoped local state access.
+ * Returned by `shared()` for shared operations.
+ */
+export interface LocalStateAccessor {
+ /**
+ * Get a value from scoped local state.
+ *
+ * @param key The key to retrieve.
+ * @returns The stored value, or null if not found.
+ */
+ get(key: string): Promise<unknown>;
+
+ /**
+ * Set a value in scoped local state.
+ *
+ * @param key The key to store.
+ * @param value The value to store (must be JSON-serializable).
+ */
+ set(key: string, value: unknown): Promise<void>;
+
+ /**
+ * Remove a value from scoped local state.
+ *
+ * @param key The key to remove.
+ */
+ remove(key: string): Promise<void>;
+}
+
+/**
+ * Get a value from user-scoped local state.
+ *
+ * Data is automatically scoped to the current authenticated user.
+ * Other users on the same browser cannot see or modify this data.
+ *
+ * @param key The key to retrieve.
+ * @returns The stored value, or null if not found.
+ *
+ * @example
+ * ```typescript
+ * const isCollapsed = await localState.get('sidebar_collapsed');
+ * if (isCollapsed) {
+ * collapseSidebar();
+ * }
+ * ```
+ */
+export declare function get(key: string): Promise<unknown>;
+
+/**
+ * Set a value in user-scoped local state.
+ *
+ * Data is automatically scoped to the current authenticated user.
+ * Other users on the same browser cannot see or modify this data.
+ *
+ * @param key The key to store.
+ * @param value The value to store (must be JSON-serializable).
+ *
+ * @example
+ * ```typescript
+ * await localState.set('sidebar_collapsed', true);
+ * await localState.set('panel_width', 300);
+ * ```
+ */
+export declare function set(key: string, value: unknown): Promise<void>;
Review Comment:
**Suggestion:** The top-level `set` API type also allows `unknown`, so typed extension code can pass `undefined` and other non-serializable values that become invalid JSON in storage and crash when read back. Restrict the accepted value type to JSON-compatible primitives/objects/arrays so the API contract matches the runtime serializer/parser behavior. [type error]
<details>
<summary><b>Severity Level:</b> Major β οΈ</summary>
```mdx
- β User-scoped localState reads can crash after bad writes.
- β οΈ Breaks extension features relying on persisted UI state.
- β οΈ TS typing misrepresents JSON-only storage contract.
```
</details>
```suggestion
export declare function set(
key: string,
value: string | number | boolean | null | Record<string, unknown> | Array<unknown>,
): Promise<void>;
```
<details>
<summary><b>Steps of Reproduction β
</b></summary>
```mdx
1. An extension written in TypeScript imports the user-scoped API as suggested in
`superset-frontend/packages/superset-core/src/storage/localState.ts:39-45`: `import {
localState } from '@apache-superset/core/storage';`.
2. The extension calls `await localState.set('sidebar_collapsed', maybeCollapsed);` where
`maybeCollapsed` is typed `boolean | undefined`. This compiles because the public `set`
signature `export declare function set(key: string, value: unknown): Promise<void>;` at
`packages/superset-core/src/storage/localState.ts:100-115` accepts `unknown`, so `boolean
| undefined` is assignable with no error.
3. At runtime, the host implementation of user-scoped local state is
`createBrowserStorageImpl` in `superset-frontend/src/core/storage/localState.ts:61-76`.
When `maybeCollapsed` is `undefined`, `set` at lines 70-75 computes
`JSON.stringify(value)` which yields `undefined`; `storage.setItem(storageKey,
JSON.stringify(value));` then stores the string `"undefined"` in `localStorage`.
4. Later, the extension reads the value with `await localState.get('sidebar_collapsed');`,
which executes `get` at `src/core/storage/localState.ts:61-68`.
`storage.getItem(storageKey)` returns `"undefined"`, which is truthy, so
`JSON.parse(value)` on line 67 is invoked with `"undefined"` and throws a `SyntaxError`,
breaking the extension's UI flow whenever it tries to restore this local state.
```
</details>
<details>
<summary><b>Prompt for AI Agent π€ </b></summary>
```mdx
This is a comment left during a code review.
**Path:** superset-frontend/packages/superset-core/src/storage/localState.ts
**Line:** 115:115
**Comment:**
*Type Error: The top-level `set` API type also allows `unknown`, so typed extension code can pass `undefined` and other non-serializable values that become invalid JSON in storage and crash when read back. Restrict the accepted value type to JSON-compatible primitives/objects/arrays so the API contract matches the runtime serializer/parser behavior.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F39171&comment_hash=42537622b0f28c8e2263e211db7bfd12c3aa0aa98c05af0e756b7c7ec551cf1d&reaction=like'>π</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F39171&comment_hash=42537622b0f28c8e2263e211db7bfd12c3aa0aa98c05af0e756b7c7ec551cf1d&reaction=dislike'>π</a>
##########
superset-frontend/packages/superset-core/src/storage/localState.ts:
##########
@@ -0,0 +1,150 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/**
+ * @fileoverview Local State API for Superset extensions (Tier 1 Storage).
+ *
+ * Provides client-side KV storage backed by the browser's localStorage.
+ * Data persists across browser sessions but is per-device (not shared across
+ * devices or synced to the server).
+ *
+ * By default, all operations are user-scoped (private to the current user).
+ * Use shared() to access state visible to all users on the same browser.
+ *
+ * Key patterns:
+ * - User-scoped (default): superset-ext:{extension_id}:user:{user_id}:{key}
+ * - Shared: superset-ext:{extension_id}:{key}
+ *
+ * The API is async to maintain compatibility with a future sandboxed execution
+ * model where storage calls would go through a postMessage bridge.
+ *
+ * @example
+ * ```typescript
+ * import { localState } from '@apache-superset/core/storage';
+ *
+ * // User-scoped state (default - private to current user)
+ * const isCollapsed = await localState.get('sidebar_collapsed');
+ * await localState.set('sidebar_collapsed', true);
+ * await localState.remove('sidebar_collapsed');
+ *
+ * // Shared state (visible to all users on same browser)
+ * const deviceId = await localState.shared().get('device_id');
+ * await localState.shared().set('device_id', 'abc-123');
+ * ```
+ */
+
+/**
+ * Interface for scoped local state access.
+ * Returned by `shared()` for shared operations.
+ */
+export interface LocalStateAccessor {
+ /**
+ * Get a value from scoped local state.
+ *
+ * @param key The key to retrieve.
+ * @returns The stored value, or null if not found.
+ */
+ get(key: string): Promise<unknown>;
+
+ /**
+ * Set a value in scoped local state.
+ *
+ * @param key The key to store.
+ * @param value The value to store (must be JSON-serializable).
+ */
+ set(key: string, value: unknown): Promise<void>;
Review Comment:
**Suggestion:** The shared accessor currently accepts `unknown` for `set`, which includes non-JSON values like `undefined`. In the host implementation this value is serialized with `JSON.stringify` and later parsed with `JSON.parse`; storing `undefined` produces an invalid JSON payload that will throw on read. Narrow the type to JSON-compatible values to prevent writing values that break subsequent `get` calls. [type error]
<details>
<summary><b>Severity Level:</b> Major β οΈ</summary>
```mdx
- β Shared localState reads can crash on stored undefined.
- β οΈ Undermines safety guarantees of Tier 1 shared storage.
- β οΈ Affects any TS extensions using shared localState.
```
</details>
```suggestion
set(
key: string,
value: string | number | boolean | null | Record<string, unknown> | Array<unknown>,
): Promise<void>;
```
<details>
<summary><b>Steps of Reproduction β
</b></summary>
```mdx
1. An extension author uses the Tier 1 storage API as documented in
`superset-frontend/packages/superset-core/src/storage/index.ts:34-53`, importing
`localState` from `@apache-superset/core/storage`.
2. Their TypeScript code calls the shared accessor `await
localState.shared().set('device_id', maybeDeviceId);` where `maybeDeviceId` has type
`string | undefined`. This compiles because the shared accessor type
`LocalStateAccessor.set(key: string, value: unknown)` at
`packages/superset-core/src/storage/localState.ts:56-71` accepts `unknown`, which is a
supertype of `string | undefined`.
3. At runtime, the host binds `localState.shared()` to the `SharedAccessor` implementation
in `superset-frontend/src/core/storage/localState.ts:37-59`. When the extension passes
`undefined`, `SharedAccessor.set` at lines 50-52 executes `storage.setItem(storageKey,
JSON.stringify(value));`. `JSON.stringify(undefined)` returns `undefined`, which
`localStorage.setItem` coerces to the string `"undefined"` and persists.
4. Later, the extension calls `await localState.shared().get('device_id');`, which is
implemented by `SharedAccessor.get` at `src/core/storage/localState.ts:44-48`.
`storage.getItem(storageKey)` returns the string `"undefined"`, which is truthy, so
`JSON.parse(value)` is called on line 47. `JSON.parse('undefined')` throws a
`SyntaxError`, causing the extension's read of shared local state to crash at runtime.
```
</details>
<details>
<summary><b>Prompt for AI Agent π€ </b></summary>
```mdx
This is a comment left during a code review.
**Path:** superset-frontend/packages/superset-core/src/storage/localState.ts
**Line:** 71:71
**Comment:**
*Type Error: The shared accessor currently accepts `unknown` for `set`, which includes non-JSON values like `undefined`. In the host implementation this value is serialized with `JSON.stringify` and later parsed with `JSON.parse`; storing `undefined` produces an invalid JSON payload that will throw on read. Narrow the type to JSON-compatible values to prevent writing values that break subsequent `get` calls.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F39171&comment_hash=25a85d3ff0108608d1f1bd9218406fe3c2bf248d89d178803faf8eb16277cc12&reaction=like'>π</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F39171&comment_hash=25a85d3ff0108608d1f1bd9218406fe3c2bf248d89d178803faf8eb16277cc12&reaction=dislike'>π</a>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(extensions): add a Storage API for extensions [superset]
Posted by "michael-s-molina (via GitHub)" <gi...@apache.org>.
michael-s-molina commented on code in PR #39171:
URL: https://github.com/apache/superset/pull/39171#discussion_r3066485170
##########
docs/developer_docs/extensions/storage.md:
##########
@@ -0,0 +1,231 @@
+---
+title: Storage
+sidebar_position: 8
+---
+
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Storage
+
+Superset Extensions have access to a managed storage API for persisting data. The storage system provides multiple tiers with different persistence characteristics, allowing extensions to choose the right storage for their needs.
+
+Each extension receives its own isolated storage namespace. When Superset loads your extension, it binds storage to your extension's unique identifier, ensuring data privacyβtwo extensions using the same key will never collide, and extensions cannot access each other's data.
+
+## Storage Tiers
+
+| Tier | Storage Type | Context Property | Use Case |
+| ---- | ----------------- | ------------------------------------------ | ------------------------------------------------ |
+| 1 | Browser storage | `ctx.storage.local`, `ctx.storage.session` | UI state, wizard progress, draft forms |
+| 2 | Server-side cache | `ctx.storage.ephemeral` | Job progress, temporary results |
+| 3 | Database | `ctx.storage.persistent` | User preferences, extension config (coming soon) |
+
+## Tier 1: Local State
+
+Browser-based storage that persists on the user's device. Use this for UI state and settings that don't need to sync across devices.
+
+### Why Use the API Instead of localStorage Directly?
+
+You might wonder why extensions should use `ctx.storage.local` instead of directly accessing `window.localStorage`. The managed API provides several benefits:
+
+- **Automatic namespacing**: Each extension's data is isolated. Two extensions using the same key name won't collide.
+- **User isolation**: By default, data is scoped to the current user, preventing data leakage between users on shared devices.
+- **Clean uninstall**: When an extension is uninstalled, all its data can be cleanly removed using prefix-based deletion.
+- **Future sandboxing**: The async API is designed for a future sandboxed execution model where extensions run in isolated contexts without direct DOM access.
+- **Consistent patterns**: The same API shape works across all storage tiers, making it easy to switch between them.
+
+### localState
+
+Data persists across browser sessions until explicitly deleted or the user clears browser storage.
+
+```typescript
+import { getContext } from '@apache-superset/core/extensions';
+
+const ctx = getContext();
+
+// Save sidebar state
+await ctx.storage.local.set('sidebar_collapsed', true);
+
+// Retrieve it later
+const isCollapsed = await ctx.storage.local.get('sidebar_collapsed');
+
+// Remove it
+await ctx.storage.local.remove('sidebar_collapsed');
+```
+
+### sessionState
+
+Data is cleared when the browser tab is closed. Use for transient state within a single session.
+
+```typescript
+import { getContext } from '@apache-superset/core/extensions';
+
+const ctx = getContext();
+
+// Save wizard progress (lost when tab closes)
+await ctx.storage.session.set('wizard_step', 3);
+await ctx.storage.session.set('unsaved_form', { name: 'Draft' });
+
+// Retrieve on page reload within same tab
+const step = await ctx.storage.session.get('wizard_step');
+```
+
+### Shared State
+
+By default, data is scoped to the current user. Use `shared` for data that should be accessible to all users on the same device.
+
+```typescript
+import { getContext } from '@apache-superset/core/extensions';
+
+const ctx = getContext();
+
+// Shared across all users on this device
+await ctx.storage.local.shared.set('device_id', 'abc-123');
+const deviceId = await ctx.storage.local.shared.get('device_id');
+```
+
+### When to Use Tier 1
+
+- UI state (sidebar collapsed, panel sizes)
+- Recently used items
+- Draft form values
+- Any data acceptable to lose if user clears browser
+
+### Limitations
+
+- Per-browser, per-device (not shared across devices)
+- Subject to browser storage quotas (~5-10 MB)
+- Not accessible from backend code
+
+## Tier 2: Ephemeral State
+
+Server-side cache storage with automatic TTL expiration. Use for temporary data that needs to be shared between frontend and backend, or persist across page reloads.
+
+### Frontend Usage
+
+```typescript
+import { getContext } from '@apache-superset/core/extensions';
+
+const ctx = getContext();
+
+// Store with default TTL (1 hour)
+await ctx.storage.ephemeral.set('job_progress', { pct: 42, status: 'running' });
+
+// Store with custom TTL (5 minutes)
+await ctx.storage.ephemeral.set(
+ 'quick_cache',
+ { results: [1, 2, 3] },
+ { ttl: 300 },
+);
+
+// Retrieve
+const progress = await ctx.storage.ephemeral.get('job_progress');
+
+// Remove
+await ctx.storage.ephemeral.remove('job_progress');
+```
+
+### Backend Usage
+
+```python
+from superset_core.extensions.context import get_context
+
+ctx = get_context()
+
+# Store job progress
+ctx.storage.ephemeral.set('job_progress', {'pct': 42, 'status': 'running'}, ttl=3600)
+
+# Retrieve
+progress = ctx.storage.ephemeral.get('job_progress')
+
+# Remove
+ctx.storage.ephemeral.remove('job_progress')
+```
+
+### Shared State
+
+For data that needs to be visible to all users:
+
+```typescript
+import { getContext } from '@apache-superset/core/extensions';
+
+const ctx = getContext();
+
+await ctx.storage.ephemeral.shared.set('shared_result', { data: [1, 2, 3] });
+const result = await ctx.storage.ephemeral.shared.get('shared_result');
+```
+
+```python
+from superset_core.extensions.context import get_context
+
+ctx = get_context()
+
+ctx.storage.ephemeral.shared.set('shared_result', {'data': [1, 2, 3]})
+result = ctx.storage.ephemeral.shared.get('shared_result')
+```
+
+### When to Use Tier 2
+
+- Background job progress indicators
+- Cross-request intermediate state
+- Query result previews
+- Temporary computation results
+- Any data that can be recomputed if lost
+
+### Limitations
+
+- Not guaranteed to survive server restarts
+- Subject to cache eviction under memory pressure
+- TTL-based expiration (data disappears after timeout)
+
+## Tier 3: Persistent State
+
+Coming soon.
+
+## Key Patterns
+
+All storage keys are automatically namespaced:
+
+| Scope | Key Pattern |
+| ----------- | -------------------------------------------------- |
+| User-scoped | `superset-ext:{extension_id}:user:{user_id}:{key}` |
+| Shared | `superset-ext:{extension_id}:shared:{key}` |
+
+This ensures:
+
+- Extensions cannot accidentally access each other's data
+- Users cannot see other users' data (by default)
+- Clean prefix-based deletion on uninstall
+
+## Configuration
+
+Administrators can configure Tier 2 storage in `superset_config.py`:
+
+```python
+EXTENSIONS_STORAGE = {
+ "EPHEMERAL": {
+ # Use Redis for better performance in production
+ "CACHE_TYPE": "RedisCache",
+ "CACHE_REDIS_URL": "redis://localhost:6379/2",
+ "CACHE_DEFAULT_TIMEOUT": 3600, # 1 hour default TTL
+ },
+}
Review Comment:
Addressed in [53fcc21](https://github.com/apache/superset/pull/39171/commits/53fcc219923c5f27b0e278d333cb1fe69db8b493)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(extensions): add a Storage API for extensions [superset]
Posted by "codeant-ai-for-open-source[bot] (via GitHub)" <gi...@apache.org>.
codeant-ai-for-open-source[bot] commented on code in PR #39171:
URL: https://github.com/apache/superset/pull/39171#discussion_r3624206011
##########
superset-core/src/superset_core/extensions/context.py:
##########
@@ -0,0 +1,136 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""
+Extension Context API for superset-core extensions.
+
+Provides access to the current extension's context, including metadata
+and scoped resources like storage. Extensions call `get_context()` to
+access their context during execution.
+
+The context is set by the host (Superset) during extension loading and
+is only available within extension code.
+
+Usage:
+ from superset_core.extensions.context import get_context
+
+ def setup():
+ ctx = get_context()
+
+ # Access extension metadata
+ print(f"Running {ctx.extension.displayName} v{ctx.extension.version}")
+
+ # Access extension-scoped storage
+ ctx.storage.ephemeral.set("lastRun", time.time(), EphemeralSetOptions(ttl=3600))
+ data = ctx.storage.ephemeral.get("cachedData")
+"""
+
+from __future__ import annotations
+
+from typing import Any, Protocol, TYPE_CHECKING
+
+if TYPE_CHECKING:
+ from superset_core.extensions.types import BaseExtension
+
+
+class StorageAccessor(Protocol):
+ """Protocol for storage access with user-scoped and shared modes."""
+
+ def get(self, key: str) -> Any:
+ """Get a value from storage."""
+ ...
+
+ def set(self, key: str, value: Any, ttl: int = 3600) -> None:
+ """Set a value in storage with optional TTL."""
+ ...
Review Comment:
**Suggestion:** The `StorageAccessor.set` contract uses a `ttl` integer argument, but the Tier 2 implementation in this PR expects an `EphemeralSetOptions` object; this API mismatch will let callers pass the wrong argument shape and can trigger runtime `TypeError` when the concrete accessor is invoked. Split the protocol so ephemeral and persistent setters use their real signatures. [possible bug]
<details>
<summary><b>Severity Level:</b> Major β οΈ</summary>
```mdx
- β Backend extensions using Python ephemeral storage can crash.
- β οΈ Ephemeral writes fail, losing short-lived extension state.
```
</details>
<details>
<summary><b>Steps of Reproduction β
</b></summary>
```mdx
1. In a backend extension, import `get_context()` from
`superset-core/src/superset_core/extensions/context.py:104` and obtain `ctx =
get_context()`.
2. Use the `ExtensionStorage.ephemeral` accessor, which is typed as `StorageAccessor` in
`superset-core/src/superset_core/extensions/context.py:71-76`, and follow its documented
signature `set(self, key, value, ttl=3600)` from `context.py:57-58`.
3. Implement code such as `ctx.storage.ephemeral.set("job_progress", {"pct": 42},
ttl=300)` based on that protocol.
4. At runtime the concrete Tier 2 implementation for `ephemeral` follows
`EphemeralStateAccessor.set(self, key, value, options: EphemeralSetOptions)` from
`superset-core/src/superset_core/extensions/storage/ephemeral.py:69-76`, attempts to treat
the third argument as an `EphemeralSetOptions` (accessing `options.ttl` /
`options.codec`), and crashes with an `AttributeError`/`TypeError` because an `int` was
passed instead of the expected dataclass.
```
</details>
[](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=25d1bb1147b3457b88546bffce0ba4b8&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=25d1bb1147b3457b88546bffce0ba4b8&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset)
*(Use Cmd/Ctrl + Click for best experience)*
<details>
<summary><b>Prompt for AI Agent π€ </b></summary>
```mdx
This is a comment left during a code review.
**Path:** superset-core/src/superset_core/extensions/context.py
**Line:** 57:59
**Comment:**
*Possible Bug: The `StorageAccessor.set` contract uses a `ttl` integer argument, but the Tier 2 implementation in this PR expects an `EphemeralSetOptions` object; this API mismatch will let callers pass the wrong argument shape and can trigger runtime `TypeError` when the concrete accessor is invoked. Split the protocol so ephemeral and persistent setters use their real signatures.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F39171&comment_hash=b5a68826a5a7c81ddc3892ed95b040cf044c8dc7fe8230458a9528303d7ccac4&reaction=like'>π</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F39171&comment_hash=b5a68826a5a7c81ddc3892ed95b040cf044c8dc7fe8230458a9528303d7ccac4&reaction=dislike'>π</a>
##########
superset/extensions/storage/persistent.py:
##########
@@ -0,0 +1,244 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""
+Host implementation for Persistent State (Tier 3 Storage).
+
+Provides the concrete database-backed implementation that is injected into
+superset_core.extensions.storage.persistent at startup.
+"""
+
+from __future__ import annotations
+
+from typing import Any, ClassVar
+
+from superset_core.extensions.storage.persistent import (
+ PersistentListEntry,
+ PersistentListOptions,
+ PersistentListResult,
+ PersistentSetOptions,
+ PersistentState as CorePersistentState,
+)
+
+from superset.extensions.storage.codecs import DEFAULT_CODEC, get_codec
+from superset.extensions.storage.persistent_dao import ExtensionStorageDAO
+from superset.extensions.storage.utils import (
+ get_current_extension_id,
+ get_current_user_id,
+)
+from superset.utils.decorators import transaction
+
+
+def _get_extension_id() -> str:
+ """Get the current extension ID from context."""
+ return get_current_extension_id("persistent_state")
+
+
+def _list(
+ extension_id: str,
+ user_fk: int | None,
+ options: PersistentListOptions,
+) -> PersistentListResult:
+ """Shared `list` implementation for both scopes.
+
+ Unlike the REST API, ambient backend code has no SAFE_CODECS
+ restriction β every entry's value is decoded unconditionally, same as
+ `get_decoded_value()`.
+ """
+ entries, count = ExtensionStorageDAO.list_entries(
+ extension_id,
+ user_fk=user_fk,
+ resource_type=options.resource_type,
+ resource_uuid=options.resource_uuid,
+ page=options.page,
+ page_size=options.page_size,
+ )
+ return PersistentListResult(
+ entries=[
+ PersistentListEntry(
+ key=entry.key,
+ value=(
+ get_codec(entry.codec).decode(entry.value)
+ if entry.value is not None
+ else None
+ ),
Review Comment:
**Suggestion:** Listing decodes stored bytes directly with the codec but never decrypts entries marked as encrypted, so encrypted values will fail to decode or return invalid payloads. The list path needs to apply the same decrypt-then-decode behavior as single-item reads. [possible bug]
<details>
<summary><b>Severity Level:</b> Critical π¨</summary>
```mdx
- β Listing encrypted persistent entries fails or returns garbage.
- β οΈ Extensions cannot reliably paginate encrypted persistent state.
```
</details>
<details>
<summary><b>Steps of Reproduction β
</b></summary>
```mdx
1. In a Superset extension frontend, call `ctx.storage.persistent.set('secret', payload, {
encrypt: true })` via `superset-frontend/packages/superset-core/src/storage/index.ts`,
which routes to core `PersistentState.set`
(`superset-core/src/superset_core/extensions/storage/persistent.py:144`) and then to host
`PersistentState.set` in `superset/extensions/storage/persistent.py:185-211`.
2. Host `PersistentState.set` passes `encrypt=True` and the current user id from
`get_current_user_id('persistent_state')` (`superset/extensions/storage/utils.py:40-100`)
into `ExtensionStorageDAO.set` (`superset/extensions/storage/persistent_dao.py:504`),
which stores an encrypted blob and marks `ExtensionStorage.is_encrypted = True` (column
defined in `superset/extensions/storage/persistent_model.py:101-103`).
3. Later, the extension calls `ctx.storage.persistent.list(options)` to page through keys;
this invokes host `PersistentState.list`
(`superset/extensions/storage/persistent.py:214-227`), which delegates to `_list` at
`superset/extensions/storage/persistent.py:50-84`.
4. `_list` iterates the entries from `ExtensionStorageDAO.list_entries`
(`superset/extensions/storage/persistent_dao.py:428`) and constructs `PersistentListEntry`
objects, decoding each `entry.value` via `get_codec(entry.codec).decode(entry.value)` at
lines 74-78. For rows where `is_encrypted` is True, `entry.value` contains ciphertext, so
decoding without first decrypting causes codec-specific errors (e.g. JSON parse failures)
or garbage values, breaking list responses for encrypted data and potentially returning
500 errors to the extension.
```
</details>
[](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=c66cf479ad5f4ebca5a26132d6ff1af8&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=c66cf479ad5f4ebca5a26132d6ff1af8&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset)
*(Use Cmd/Ctrl + Click for best experience)*
<details>
<summary><b>Prompt for AI Agent π€ </b></summary>
```mdx
This is a comment left during a code review.
**Path:** superset/extensions/storage/persistent.py
**Line:** 74:78
**Comment:**
*Possible Bug: Listing decodes stored bytes directly with the codec but never decrypts entries marked as encrypted, so encrypted values will fail to decode or return invalid payloads. The list path needs to apply the same decrypt-then-decode behavior as single-item reads.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F39171&comment_hash=34ddc2555f5dfaa24b33c16df95de7fd0ce29c844110388f2f83fc89c4bfd8a4&reaction=like'>π</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F39171&comment_hash=34ddc2555f5dfaa24b33c16df95de7fd0ce29c844110388f2f83fc89c4bfd8a4&reaction=dislike'>π</a>
##########
superset/extensions/storage/persistent_model.py:
##########
@@ -0,0 +1,136 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import uuid as uuid_module
+
+from flask_appbuilder import Model
+from sqlalchemy import (
+ Boolean,
+ Column,
+ ForeignKey,
+ Index,
+ Integer,
+ LargeBinary,
+ String,
+ UniqueConstraint,
+)
+from sqlalchemy_utils import UUIDType
+from superset_core.extensions.storage.models import (
+ ExtensionStorageEntry as CoreExtensionStorageEntry,
+)
+
+from superset.models.helpers import AuditMixinNullable
+
+
+class ExtensionStorage(CoreExtensionStorageEntry, AuditMixinNullable, Model):
+ """Generic persistent key-value storage for extensions (Tier 3).
+
+ Each row is identified by (extension_id, user_fk, resource_type,
+ resource_uuid, key):
+
+ * Global scope β user_fk IS NULL, resource_type IS NULL
+ * User scope β user_fk set, resource_type IS NULL
+ * Resource scope β resource_type + resource_uuid set (user_fk optional)
+
+ The payload is stored as raw bytes (value) alongside the identifier of
+ the codec used to encode it (codec), so the same bytes can be decoded
+ back into a value on read. When is_encrypted is True the value has been
+ encrypted at the DAO layer using Fernet and must be decrypted before
+ decoding.
+ """
+
+ __tablename__ = "extension_storage"
+
+ id = Column(Integer, primary_key=True, autoincrement=True)
+ uuid = Column(
+ UUIDType(binary=True),
+ default=uuid_module.uuid4,
+ unique=True,
+ nullable=False,
+ )
+
+ # Extension identity
+ extension_id = Column(String(255), nullable=False)
+
+ # Scope discriminators β all nullable; NULLs define the scope (see docstring).
+ # No relationship() is declared for user_fk: extension storage rows are
+ # not deleted when their owning user is (ondelete="SET NULL" demotes them
+ # to global scope instead), and nothing reads through such a relationship,
+ # so there's no ORM object here for a future change to attach a cascade to.
+ user_fk = Column(
+ Integer,
+ ForeignKey(
+ "ab_user.id",
+ ondelete="SET NULL",
+ name="fk_extension_storage_user_fk_ab_user",
+ ),
+ nullable=True,
+ )
Review Comment:
**Suggestion:** Using `ondelete="SET NULL"` on the user foreign key silently moves user-scoped rows into shared scope when a user is deleted, which can expose previously private per-user data to all users through shared access. User-owned rows should be deleted or otherwise kept inaccessible instead of being demoted to global visibility. [security]
<details>
<summary><b>Severity Level:</b> Critical π¨</summary>
```mdx
- β Deleted users' persistent data can become globally readable.
- β οΈ Privacy risk for extensions storing user-specific information.
```
</details>
<details>
<summary><b>Steps of Reproduction β
</b></summary>
```mdx
1. An extension stores per-user data via `ctx.storage.persistent.set('prefs', value)` in
the frontend (`superset-frontend/packages/superset-core/src/storage/index.ts`), which
calls core `PersistentState.set`
(`superset-core/src/superset_core/extensions/storage/persistent.py:144`) and host
`PersistentState.set` in `superset/extensions/storage/persistent.py:185-211`.
2. Host `PersistentState.set` obtains the current user id from
`get_current_user_id('persistent_state')` (`superset/extensions/storage/utils.py:40-100`)
and passes it as `user_fk` into `ExtensionStorageDAO.set`
(`superset/extensions/storage/persistent_dao.py:504`), creating an `ExtensionStorage` row
whose `user_fk` column (defined at
`superset/extensions/storage/persistent_model.py:75-82`) points to the owning user in
`ab_user`.
3. An administrator later deletes that user from the `ab_user` table; because the foreign
key on `ExtensionStorage.user_fk` is declared with `ondelete="SET NULL"` and
`nullable=True` at `superset/extensions/storage/persistent_model.py:78-82`, the database
automatically sets `user_fk` to `NULL`, effectively turning the row into a global (shared)
entry.
4. Another user calls `ctx.storage.persistent.shared.get('prefs')`, which routes to
`SharedPersistentStateAccessor.get` in `superset/extensions/storage/persistent.py:95-105`;
this passes `user_fk=None` to `ExtensionStorageDAO.get_decoded_value`, causing the demoted
row to be returned as shared state and exposing what was originally user-specific data to
all users of the extension.
```
</details>
[](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=211df7eff2814c499a1a28d8eed9aae7&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=211df7eff2814c499a1a28d8eed9aae7&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset)
*(Use Cmd/Ctrl + Click for best experience)*
<details>
<summary><b>Prompt for AI Agent π€ </b></summary>
```mdx
This is a comment left during a code review.
**Path:** superset/extensions/storage/persistent_model.py
**Line:** 75:82
**Comment:**
*Security: Using `ondelete="SET NULL"` on the user foreign key silently moves user-scoped rows into shared scope when a user is deleted, which can expose previously private per-user data to all users through shared access. User-owned rows should be deleted or otherwise kept inaccessible instead of being demoted to global visibility.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F39171&comment_hash=78e408aeeccd23f4f49ddfad47f27cef819d9316a78050158f371e531f2cc351&reaction=like'>π</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F39171&comment_hash=78e408aeeccd23f4f49ddfad47f27cef819d9316a78050158f371e531f2cc351&reaction=dislike'>π</a>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(extensions): add a Storage API for extensions [superset]
Posted by "michael-s-molina (via GitHub)" <gi...@apache.org>.
michael-s-molina commented on code in PR #39171:
URL: https://github.com/apache/superset/pull/39171#discussion_r3622783490
##########
superset/extensions/storage/api.py:
##########
@@ -0,0 +1,738 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""
+REST API for extension storage.
+
+Provides HTTP endpoints for frontend extensions to access server-side
+ephemeral storage without direct backend code.
+
+All operations are user-scoped by default. Use `?shared=true` query param
+to access shared state visible to all users.
+"""
+
+from __future__ import annotations
+
+import base64
+from typing import Any
+
+from flask import g, request
+from flask.wrappers import Response
+from flask_appbuilder.api import BaseApi, expose, protect, safe
+
+from superset.extensions.storage.codecs import DEFAULT_CODEC, get_codec, SAFE_CODECS
+from superset.extensions.storage.ephemeral_dao import (
+ ExtensionEphemeralDAO,
+ ExtensionEphemeralTTLInvalid,
+ ExtensionEphemeralValueTooLarge,
+)
+from superset.extensions.storage.persistent_dao import (
+ ExtensionStorageDAO,
+ ExtensionStorageKeyTooLong,
+ ExtensionStorageListPayloadTooLarge,
+ ExtensionStorageQuotaExceeded,
+ ExtensionStorageValueTooLarge,
+)
+from superset.extensions.storage.utils import get_extension_or_404, parse_ttl
+from superset.key_value.exceptions import KeyValueCodecEncodeException
+from superset.utils.decorators import transaction
+
+
+def _decoded_result_for_wire(decoded: Any) -> tuple[Any, bool]:
+ """Convert a codec-decoded value into its JSON wire representation.
+
+ JSON has no byte type, so a raw `bytes` value is base64-encoded to a
+ string for the response and flagged as such; every other value is
+ already JSON-representable as-is. Checked on the decoded value's
+ actual type, not the codec's name, so this keeps working for a codec
+ this module didn't define.
+
+ :returns: (wire_value, is_binary)
+ """
+ if isinstance(decoded, bytes):
+ return base64.b64encode(decoded).decode("ascii"), True
+ return decoded, False
+
+
+def _wire_value_for_request(value: Any, is_binary: bool) -> Any:
+ """Convert a request body's JSON `value` into a codec's input type.
+
+ `is_binary` is an explicit flag from the caller: JSON has no byte
+ type, so there is no way to tell, from the JSON `value` alone,
+ whether it is a base64 string that must be decoded to bytes before
+ being handed to the codec's `encode`, or a literal value to pass
+ through as-is. Only the caller knows which it sent.
+
+ :raises ValueError: if `is_binary` is set and `value` is not a valid
+ base64 string.
+ """
+ if is_binary:
+ return base64.b64decode(value, validate=True)
+ return value
+
+
+class ExtensionStorageRestApi(BaseApi):
+ """REST API for extension ephemeral (Tier 2) and persistent (Tier 3) storage."""
+
+ allow_browser_login = True
+ route_base = "/api/v1/extensions"
+
+ def response(self, status_code: int, **kwargs: Any) -> Response:
+ """Helper method to create JSON responses."""
+ from flask import jsonify
+
+ return jsonify(kwargs), status_code
+
+ def response_404(self, message: str = "Not found") -> Response:
+ """Helper method to create 404 responses."""
+ from flask import jsonify
+
+ return jsonify({"message": message}), 404
+
+ def response_400(self, message: str) -> Response:
+ """Helper method to create 400 responses."""
+ from flask import jsonify
+
+ return jsonify({"message": message}), 400
Review Comment:
Good catch β this one is real and self-inflicted from an earlier fix in this same review round. `response_400` was being passed `f"...: {ex}"`, interpolating the raw exception message (from `KeyValueCodecEncodeException`/`TypeError`/`ValueError`) directly into the HTTP response body. Fixed by dropping `{ex}` from the message so only the static, safe text (which codec was rejected) reaches the client.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(extensions): add a Storage API for extensions [superset]
Posted by "github-advanced-security[bot] (via GitHub)" <gi...@apache.org>.
github-advanced-security[bot] commented on code in PR #39171:
URL: https://github.com/apache/superset/pull/39171#discussion_r3622759850
##########
superset/extensions/storage/api.py:
##########
@@ -0,0 +1,738 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""
+REST API for extension storage.
+
+Provides HTTP endpoints for frontend extensions to access server-side
+ephemeral storage without direct backend code.
+
+All operations are user-scoped by default. Use `?shared=true` query param
+to access shared state visible to all users.
+"""
+
+from __future__ import annotations
+
+import base64
+from typing import Any
+
+from flask import g, request
+from flask.wrappers import Response
+from flask_appbuilder.api import BaseApi, expose, protect, safe
+
+from superset.extensions.storage.codecs import DEFAULT_CODEC, get_codec, SAFE_CODECS
+from superset.extensions.storage.ephemeral_dao import (
+ ExtensionEphemeralDAO,
+ ExtensionEphemeralTTLInvalid,
+ ExtensionEphemeralValueTooLarge,
+)
+from superset.extensions.storage.persistent_dao import (
+ ExtensionStorageDAO,
+ ExtensionStorageKeyTooLong,
+ ExtensionStorageListPayloadTooLarge,
+ ExtensionStorageQuotaExceeded,
+ ExtensionStorageValueTooLarge,
+)
+from superset.extensions.storage.utils import get_extension_or_404, parse_ttl
+from superset.key_value.exceptions import KeyValueCodecEncodeException
+from superset.utils.decorators import transaction
+
+
+def _decoded_result_for_wire(decoded: Any) -> tuple[Any, bool]:
+ """Convert a codec-decoded value into its JSON wire representation.
+
+ JSON has no byte type, so a raw `bytes` value is base64-encoded to a
+ string for the response and flagged as such; every other value is
+ already JSON-representable as-is. Checked on the decoded value's
+ actual type, not the codec's name, so this keeps working for a codec
+ this module didn't define.
+
+ :returns: (wire_value, is_binary)
+ """
+ if isinstance(decoded, bytes):
+ return base64.b64encode(decoded).decode("ascii"), True
+ return decoded, False
+
+
+def _wire_value_for_request(value: Any, is_binary: bool) -> Any:
+ """Convert a request body's JSON `value` into a codec's input type.
+
+ `is_binary` is an explicit flag from the caller: JSON has no byte
+ type, so there is no way to tell, from the JSON `value` alone,
+ whether it is a base64 string that must be decoded to bytes before
+ being handed to the codec's `encode`, or a literal value to pass
+ through as-is. Only the caller knows which it sent.
+
+ :raises ValueError: if `is_binary` is set and `value` is not a valid
+ base64 string.
+ """
+ if is_binary:
+ return base64.b64decode(value, validate=True)
+ return value
+
+
+class ExtensionStorageRestApi(BaseApi):
+ """REST API for extension ephemeral (Tier 2) and persistent (Tier 3) storage."""
+
+ allow_browser_login = True
+ route_base = "/api/v1/extensions"
+
+ def response(self, status_code: int, **kwargs: Any) -> Response:
+ """Helper method to create JSON responses."""
+ from flask import jsonify
+
+ return jsonify(kwargs), status_code
+
+ def response_404(self, message: str = "Not found") -> Response:
+ """Helper method to create 404 responses."""
+ from flask import jsonify
+
+ return jsonify({"message": message}), 404
+
+ def response_400(self, message: str) -> Response:
+ """Helper method to create 400 responses."""
+ from flask import jsonify
+
+ return jsonify({"message": message}), 400
Review Comment:
## CodeQL / Information exposure through an exception
[Stack trace information](1) flows to this location and may be exposed to an external user.
[Show more details](https://github.com/apache/superset/security/code-scanning/2585)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(extensions): add a Storage API for extensions [superset]
Posted by "michael-s-molina (via GitHub)" <gi...@apache.org>.
michael-s-molina commented on code in PR #39171:
URL: https://github.com/apache/superset/pull/39171#discussion_r3624400134
##########
superset/extensions/storage/persistent_model.py:
##########
@@ -0,0 +1,136 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import uuid as uuid_module
+
+from flask_appbuilder import Model
+from sqlalchemy import (
+ Boolean,
+ Column,
+ ForeignKey,
+ Index,
+ Integer,
+ LargeBinary,
+ String,
+ UniqueConstraint,
+)
+from sqlalchemy_utils import UUIDType
+from superset_core.extensions.storage.models import (
+ ExtensionStorageEntry as CoreExtensionStorageEntry,
+)
+
+from superset.models.helpers import AuditMixinNullable
+
+
+class ExtensionStorage(CoreExtensionStorageEntry, AuditMixinNullable, Model):
+ """Generic persistent key-value storage for extensions (Tier 3).
+
+ Each row is identified by (extension_id, user_fk, resource_type,
+ resource_uuid, key):
+
+ * Global scope β user_fk IS NULL, resource_type IS NULL
+ * User scope β user_fk set, resource_type IS NULL
+ * Resource scope β resource_type + resource_uuid set (user_fk optional)
+
+ The payload is stored as raw bytes (value) alongside the identifier of
+ the codec used to encode it (codec), so the same bytes can be decoded
+ back into a value on read. When is_encrypted is True the value has been
+ encrypted at the DAO layer using Fernet and must be decrypted before
+ decoding.
+ """
+
+ __tablename__ = "extension_storage"
+
+ id = Column(Integer, primary_key=True, autoincrement=True)
+ uuid = Column(
+ UUIDType(binary=True),
+ default=uuid_module.uuid4,
+ unique=True,
+ nullable=False,
+ )
+
+ # Extension identity
+ extension_id = Column(String(255), nullable=False)
+
+ # Scope discriminators β all nullable; NULLs define the scope (see docstring).
+ # No relationship() is declared for user_fk: extension storage rows are
+ # not deleted when their owning user is (ondelete="SET NULL" demotes them
+ # to global scope instead), and nothing reads through such a relationship,
+ # so there's no ORM object here for a future change to attach a cascade to.
+ user_fk = Column(
+ Integer,
+ ForeignKey(
+ "ab_user.id",
+ ondelete="SET NULL",
+ name="fk_extension_storage_user_fk_ab_user",
+ ),
+ nullable=True,
+ )
Review Comment:
Fixed β changed `user_fk`'s FK to `ondelete="CASCADE"` (in the model and the not-yet-released migration that creates this table), so a deleted user's private persistent-storage rows are deleted instead of being demoted to shared/global scope. `created_by_fk`/`changed_by_fk` are left as `SET NULL` since they're pure audit metadata with no bearing on scope, consistent with every other audited model in the codebase.
##########
superset/extensions/storage/api.py:
##########
@@ -0,0 +1,740 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""
+REST API for extension storage.
+
+Provides HTTP endpoints for frontend extensions to access server-side
+ephemeral storage without direct backend code.
+
+All operations are user-scoped by default. Use `?shared=true` query param
+to access shared state visible to all users.
+"""
+
+from __future__ import annotations
+
+import base64
+from typing import Any
+
+from flask import g, request
+from flask.wrappers import Response
+from flask_appbuilder.api import BaseApi, expose, protect, safe
+
+from superset.extensions.storage.codecs import DEFAULT_CODEC, get_codec, SAFE_CODECS
+from superset.extensions.storage.ephemeral_dao import (
+ ExtensionEphemeralDAO,
+ ExtensionEphemeralTTLInvalid,
+ ExtensionEphemeralValueTooLarge,
+)
+from superset.extensions.storage.persistent_dao import (
+ ExtensionStorageDAO,
+ ExtensionStorageKeyTooLong,
+ ExtensionStorageListPayloadTooLarge,
+ ExtensionStorageQuotaExceeded,
+ ExtensionStorageValueTooLarge,
+)
+from superset.extensions.storage.utils import get_extension_or_404, parse_ttl
+from superset.key_value.exceptions import KeyValueCodecEncodeException
+from superset.utils.decorators import transaction
+
+
+def _decoded_result_for_wire(decoded: Any) -> tuple[Any, bool]:
+ """Convert a codec-decoded value into its JSON wire representation.
+
+ JSON has no byte type, so a raw `bytes` value is base64-encoded to a
+ string for the response and flagged as such; every other value is
+ already JSON-representable as-is. Checked on the decoded value's
+ actual type, not the codec's name, so this keeps working for a codec
+ this module didn't define.
+
+ :returns: (wire_value, is_binary)
+ """
+ if isinstance(decoded, bytes):
+ return base64.b64encode(decoded).decode("ascii"), True
+ return decoded, False
+
+
+def _wire_value_for_request(value: Any, is_binary: bool) -> Any:
+ """Convert a request body's JSON `value` into a codec's input type.
+
+ `is_binary` is an explicit flag from the caller: JSON has no byte
+ type, so there is no way to tell, from the JSON `value` alone,
+ whether it is a base64 string that must be decoded to bytes before
+ being handed to the codec's `encode`, or a literal value to pass
+ through as-is. Only the caller knows which it sent.
+
+ :raises ValueError: if `is_binary` is set and `value` is not a valid
+ base64 string.
+ """
+ if is_binary:
+ return base64.b64decode(value, validate=True)
+ return value
Review Comment:
Not a bug β `binascii.Error` is a subclass of `ValueError` in Python 3, so the existing `except (ValueError, TypeError)` in the request-parsing path already catches invalid base64 payloads and returns a 400. No change needed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(extensions): add a Storage API for extensions [superset]
Posted by "michael-s-molina (via GitHub)" <gi...@apache.org>.
michael-s-molina commented on code in PR #39171:
URL: https://github.com/apache/superset/pull/39171#discussion_r3624398861
##########
superset-core/src/superset_core/extensions/context.py:
##########
@@ -0,0 +1,136 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""
+Extension Context API for superset-core extensions.
+
+Provides access to the current extension's context, including metadata
+and scoped resources like storage. Extensions call `get_context()` to
+access their context during execution.
+
+The context is set by the host (Superset) during extension loading and
+is only available within extension code.
+
+Usage:
+ from superset_core.extensions.context import get_context
+
+ def setup():
+ ctx = get_context()
+
+ # Access extension metadata
+ print(f"Running {ctx.extension.displayName} v{ctx.extension.version}")
+
+ # Access extension-scoped storage
+ ctx.storage.ephemeral.set("lastRun", time.time(), EphemeralSetOptions(ttl=3600))
+ data = ctx.storage.ephemeral.get("cachedData")
+"""
+
+from __future__ import annotations
+
+from typing import Any, Protocol, TYPE_CHECKING
+
+if TYPE_CHECKING:
+ from superset_core.extensions.types import BaseExtension
+
+
+class StorageAccessor(Protocol):
+ """Protocol for storage access with user-scoped and shared modes."""
+
+ def get(self, key: str) -> Any:
+ """Get a value from storage."""
+ ...
+
+ def set(self, key: str, value: Any, ttl: int = 3600) -> None:
+ """Set a value in storage with optional TTL."""
+ ...
Review Comment:
Fixed β removed the stale duplicate `StorageAccessor` protocol and pointed `ExtensionStorage.ephemeral`/`.persistent` at the real `EphemeralStateAccessor`/`PersistentStateAccessor` protocols, which already have the correct `options`-based `set()` signatures.
##########
superset/extensions/storage/persistent.py:
##########
@@ -0,0 +1,244 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""
+Host implementation for Persistent State (Tier 3 Storage).
+
+Provides the concrete database-backed implementation that is injected into
+superset_core.extensions.storage.persistent at startup.
+"""
+
+from __future__ import annotations
+
+from typing import Any, ClassVar
+
+from superset_core.extensions.storage.persistent import (
+ PersistentListEntry,
+ PersistentListOptions,
+ PersistentListResult,
+ PersistentSetOptions,
+ PersistentState as CorePersistentState,
+)
+
+from superset.extensions.storage.codecs import DEFAULT_CODEC, get_codec
+from superset.extensions.storage.persistent_dao import ExtensionStorageDAO
+from superset.extensions.storage.utils import (
+ get_current_extension_id,
+ get_current_user_id,
+)
+from superset.utils.decorators import transaction
+
+
+def _get_extension_id() -> str:
+ """Get the current extension ID from context."""
+ return get_current_extension_id("persistent_state")
+
+
+def _list(
+ extension_id: str,
+ user_fk: int | None,
+ options: PersistentListOptions,
+) -> PersistentListResult:
+ """Shared `list` implementation for both scopes.
+
+ Unlike the REST API, ambient backend code has no SAFE_CODECS
+ restriction β every entry's value is decoded unconditionally, same as
+ `get_decoded_value()`.
+ """
+ entries, count = ExtensionStorageDAO.list_entries(
+ extension_id,
+ user_fk=user_fk,
+ resource_type=options.resource_type,
+ resource_uuid=options.resource_uuid,
+ page=options.page,
+ page_size=options.page_size,
+ )
+ return PersistentListResult(
+ entries=[
+ PersistentListEntry(
+ key=entry.key,
+ value=(
+ get_codec(entry.codec).decode(entry.value)
+ if entry.value is not None
+ else None
+ ),
Review Comment:
Not a bug β `ExtensionStorageDAO.list_entries` already decrypts each row via `_decrypt_if_needed` (persistent_dao.py) before returning it, so `entry.value` here is already plaintext bytes by the time `_list` decodes it with the codec. No change needed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(extensions): add a Storage API for extensions [superset]
Posted by "michael-s-molina (via GitHub)" <gi...@apache.org>.
michael-s-molina commented on code in PR #39171:
URL: https://github.com/apache/superset/pull/39171#discussion_r3560995205
##########
superset/extensions/storage/persistent_model.py:
##########
@@ -0,0 +1,129 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import uuid as uuid_module
+
+from flask_appbuilder import Model
+from sqlalchemy import (
+ Boolean,
+ Column,
+ ForeignKey,
+ Index,
+ Integer,
+ LargeBinary,
+ String,
+ Text,
+ UniqueConstraint,
+)
+from sqlalchemy.orm import backref, relationship
+from sqlalchemy_utils import UUIDType
+from superset_core.extensions.storage.models import (
+ ExtensionStorageEntry as CoreExtensionStorageEntry,
+)
+
+from superset.models.helpers import AuditMixinNullable
+
+# 16 MB β matches the KeyValue store limit.
+EXTENSION_STORAGE_MAX_SIZE = 2**24 - 1
+
+
+class ExtensionStorage(CoreExtensionStorageEntry, AuditMixinNullable, Model):
+ """Generic persistent key-value storage for extensions (Tier 3).
+
+ Each row is identified by (extension_id, user_fk, resource_type,
+ resource_uuid, key):
+
+ * Global scope β user_fk IS NULL, resource_type IS NULL
+ * User scope β user_fk set, resource_type IS NULL
+ * Resource scope β resource_type + resource_uuid set (user_fk optional)
+
+ The payload is stored as raw bytes (value) with a MIME-type hint
+ (value_type). When is_encrypted is True the value has been encrypted
+ at the DAO layer using Fernet and must be decrypted before use.
+ """
+
+ __tablename__ = "extension_storage"
+
+ id = Column(Integer, primary_key=True, autoincrement=True)
+ uuid = Column(
+ UUIDType(binary=True),
+ default=uuid_module.uuid4,
+ unique=True,
+ nullable=False,
+ )
+
+ # Extension identity
+ extension_id = Column(String(255), nullable=False)
+
+ # Scope discriminators β all nullable; NULLs define the scope (see docstring)
+ user_fk = Column(
+ Integer,
+ ForeignKey(
+ "ab_user.id",
+ ondelete="SET NULL",
+ name="fk_extension_storage_user_fk_ab_user",
+ ),
+ nullable=True,
+ )
+ resource_type = Column(String(64), nullable=True)
+ resource_uuid = Column(String(36), nullable=True)
+
+ # Storage key within the scope
+ key = Column(String(255), nullable=False)
+
+ # Optional metadata
+ category = Column(String(64), nullable=True)
+ description = Column(Text, nullable=True)
+
+ # Payload
+ value = Column(LargeBinary(EXTENSION_STORAGE_MAX_SIZE), nullable=False)
+ value_type = Column(String(255), nullable=False, default="application/json")
+ is_encrypted = Column(Boolean, nullable=False, default=False)
+
+ user = relationship(
+ "User",
+ backref=backref("extension_storage_entries", cascade="all, delete-orphan"),
Review Comment:
Removed the `backref` as it was not needed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(extensions): add a Storage API for extensions [superset]
Posted by "michael-s-molina (via GitHub)" <gi...@apache.org>.
michael-s-molina commented on PR #39171:
URL: https://github.com/apache/superset/pull/39171#issuecomment-4938191195
@codeant-ai: review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(extensions): add a Storage API for extensions [superset]
Posted by "michael-s-molina (via GitHub)" <gi...@apache.org>.
michael-s-molina commented on code in PR #39171:
URL: https://github.com/apache/superset/pull/39171#discussion_r3560958640
##########
superset/extensions/storage/api.py:
##########
@@ -0,0 +1,499 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""
+REST API for extension storage.
+
+Provides HTTP endpoints for frontend extensions to access server-side
+ephemeral storage without direct backend code.
+
+All operations are user-scoped by default. Use `?shared=true` query param
+to access shared state visible to all users.
+"""
+
+from __future__ import annotations
+
+from typing import Any
+
+from flask import current_app, g, request
+from flask.wrappers import Response
+from flask_appbuilder.api import BaseApi, expose, protect, safe
+
+from superset.extensions import cache_manager
+from superset.extensions.storage.persistent_dao import (
+ ExtensionStorageDAO,
+ ExtensionStorageQuotaExceeded,
+)
+from superset.extensions.types import LoadedExtension
+from superset.extensions.utils import get_extensions
+from superset.utils import json
+from superset.utils.decorators import transaction
+
+# Key separator
+SEPARATOR = ":"
+
+# Key prefix for extension ephemeral state
+KEY_PREFIX = "superset-ext"
+
+
+def _build_cache_key(*parts: Any) -> str:
+ """Build a namespaced cache key from parts."""
+ return SEPARATOR.join(str(part) for part in parts)
+
+
+def _get_extension_or_404(extension_id: str) -> LoadedExtension | None:
+ """Get extension by ID or return None if not found."""
+ extensions = get_extensions()
+ return extensions.get(extension_id)
+
+
+def _parse_ttl(body: dict[str, Any]) -> tuple[int | None, str | None]:
+ """Parse and validate TTL from request body.
+
+ Returns:
+ (ttl, error_message) - error_message is set if the value is missing or invalid.
+ """
+ if "ttl" not in body:
+ return None, "Field 'ttl' is required"
+ try:
+ ttl = int(body["ttl"])
+ except (TypeError, ValueError):
+ return None, "Field 'ttl' must be a positive integer"
+ if ttl <= 0:
+ return None, "Field 'ttl' must be a positive integer"
+ max_ttl = current_app.config.get("EXTENSIONS_EPHEMERAL_STORAGE", {}).get("MAX_TTL")
+ if max_ttl is not None and ttl > max_ttl:
+ return None, f"Field 'ttl' must not exceed {max_ttl} seconds"
+ return ttl, None
+
+
+def _build_storage_key(extension_id: str, key: str, shared: bool) -> str:
+ """Build the cache key based on scope (user or shared)."""
+ if shared:
+ return _build_cache_key(KEY_PREFIX, extension_id, "shared", key)
+ user_id = g.user.id
+ return _build_cache_key(KEY_PREFIX, extension_id, "user", user_id, key)
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(extensions): add a Storage API for extensions [superset]
Posted by "codeant-ai-for-open-source[bot] (via GitHub)" <gi...@apache.org>.
codeant-ai-for-open-source[bot] commented on code in PR #39171:
URL: https://github.com/apache/superset/pull/39171#discussion_r3624439297
##########
superset/extensions/storage/persistent_dao.py:
##########
@@ -0,0 +1,637 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from __future__ import annotations
+
+import hashlib
+import hmac
+import logging
+from dataclasses import dataclass
+from typing import Any
+
+import backoff
+from flask import current_app
+from flask_babel import gettext as _
+from sqlalchemy import and_, desc, func, LargeBinary
+
+from superset import db
+from superset.daos.base import BaseDAO
+from superset.distributed_lock import DistributedLock
+from superset.exceptions import (
+ AcquireDistributedLockFailedException,
+ SupersetGenericErrorException,
+)
+from superset.extensions.storage.codecs import DEFAULT_CODEC, get_codec
+from superset.extensions.storage.filters import ExtensionStorageFilter
+from superset.extensions.storage.persistent_model import ExtensionStorage
+from superset.utils.encrypt import (
+ DEFAULT_ENCRYPTION_ENGINE_NAME,
+ EncryptedType,
+ resolve_encryption_engine,
+)
+
+logger = logging.getLogger(__name__)
+
+
+class ExtensionStorageQuotaExceeded(SupersetGenericErrorException):
+ """Raised when a write would exceed an extension's persistent storage quota."""
+
+ status = 413
+
+ def __init__(self, extension_id: str, quota: int) -> None:
+ super().__init__(
+ message=_(
+ "Extension '%(extension_id)s' has exceeded its persistent "
+ "storage quota of %(quota)d bytes.",
+ extension_id=extension_id,
+ quota=quota,
+ ),
+ )
+
+
+class ExtensionStorageValueTooLarge(SupersetGenericErrorException):
+ """Raised when a `set` call's value exceeds MAX_VALUE_SIZE."""
+
+ status = 400
+
+ def __init__(self, max_size: int) -> None:
+ super().__init__(
+ message=_(
+ "Value exceeds the maximum allowed size of %(max_size)d bytes.",
+ max_size=max_size,
+ ),
+ )
+
+
+class ExtensionStorageListPayloadTooLarge(SupersetGenericErrorException):
+ """Raised when a `list` page's combined value size exceeds
+ MAX_LIST_PAYLOAD_SIZE."""
+
+ status = 400
+
+ def __init__(self, requested_size: int, max_size: int) -> None:
+ super().__init__(
+ message=_(
+ "Requested page would return %(requested_size)d bytes, "
+ "exceeding the maximum allowed list payload size of "
+ "%(max_size)d bytes. Reduce page_size and try again.",
+ requested_size=requested_size,
+ max_size=max_size,
+ ),
+ )
+
+
+class ExtensionStorageKeyTooLong(SupersetGenericErrorException):
+ """Raised when a `set` call's key exceeds MAX_KEY_LENGTH."""
+
+ status = 400
+
+ def __init__(self, max_length: int) -> None:
+ super().__init__(
+ message=_(
+ "Storage key exceeds the maximum allowed length of "
+ "%(max_length)d characters.",
+ max_length=max_length,
+ ),
+ )
+
+
+#: Matches the `key` column's `String(255)` definition in `ExtensionStorage`
+#: (`persistent_model.py`). Enforced here, at the DAO boundary, so every
+#: caller (REST API, ambient accessors, direct DAO use) gets a consistent
+#: error instead of a database integrity/truncation failure β the frontend
+#: SDK enforces the same limit client-side purely so it can fail fast
+#: without a round trip, not as the source of truth.
+MAX_KEY_LENGTH = 255
+
+#: TTL for the lock guarding `ExtensionStorageDAO.set`'s select-then-write.
+#: Short because the guarded section is a handful of local DB operations, no
+#: network calls β long enough to cover normal execution, short enough that
+#: a crashed holder doesn't block the same key for long.
+SET_LOCK_TTL_SECONDS = 5
+
+
+def _validate_key_length(key: str) -> None:
+ """Validate key length against MAX_KEY_LENGTH. Raises if over."""
+ if len(key) > MAX_KEY_LENGTH:
+ raise ExtensionStorageKeyTooLong(MAX_KEY_LENGTH)
+
+
+def _derive_key(secret: str, user_fk: int) -> str:
+ """Derive a per-user encryption key from the master secret and user ID.
+
+ Uses HMAC-SHA256 so the derived key is cryptographically bound to both.
+ Ciphertext encrypted for one user cannot be decrypted as another.
+ """
+ return hmac.new(
+ secret.encode(),
+ str(user_fk).encode(),
+ hashlib.sha256,
+ ).hexdigest()
+
+
+def _enc_type(user_fk: int | None) -> EncryptedType:
+ """Return an EncryptedType for the given scope.
+
+ User-scoped rows use a key derived from SECRET_KEY and the user ID, so
+ ciphertext written for one user cannot be decrypted as another.
+ Shared rows (user_fk=None) use SECRET_KEY directly.
+ """
+ secret = current_app.config["SECRET_KEY"]
+ key = _derive_key(secret, user_fk) if user_fk is not None else secret
+ engine_name = current_app.config.get(
+ "SQLALCHEMY_ENCRYPTED_FIELD_ENGINE", DEFAULT_ENCRYPTION_ENGINE_NAME
+ )
+ return EncryptedType(
+ LargeBinary,
+ key=key,
+ engine=resolve_encryption_engine(engine_name),
+ )
+
+
+def _get_quota() -> int | None:
+ """Return the configured per-extension persistent storage quota in bytes.
+
+ Returns None when no quota is configured (unlimited).
+ """
+ config = current_app.config["EXTENSIONS_PERSISTENT_STORAGE"]
+ return config.get("QUOTA_PER_EXTENSION")
+
+
+def _get_max_list_payload_size() -> int | None:
+ """Return the configured max combined value size for a `list()` page.
+
+ Returns None when no limit is configured (unlimited).
+ """
+ config = current_app.config["EXTENSIONS_PERSISTENT_STORAGE"]
+ return config.get("MAX_LIST_PAYLOAD_SIZE")
+
+
+def _validate_value_size(encoded: bytes) -> None:
+ """Validate encoded value bytes against MAX_VALUE_SIZE. Raises if over.
+
+ Enforced independently of `QUOTA_PER_EXTENSION`: the quota bounds an
+ extension's total storage, while this bounds a single value. A
+ `list_entries()` page's combined size is bounded separately by
+ MAX_LIST_PAYLOAD_SIZE (see `ExtensionStorageDAO.list_entries`).
+ """
+ config = current_app.config["EXTENSIONS_PERSISTENT_STORAGE"]
+ max_size = config.get("MAX_VALUE_SIZE")
+ if max_size is None:
+ return
+ if len(encoded) > max_size:
+ raise ExtensionStorageValueTooLarge(max_size)
+
+
+def _check_quota(extension_id: str, new_size: int, existing_size: int = 0) -> None:
+ """Raise if writing `new_size` bytes would push the extension over quota.
+
+ `existing_size` is the size of the row being replaced (0 for inserts),
+ subtracted from current usage so overwriting a key doesn't double-count
+ it against the quota it already occupies.
+
+ The usage query sums the `value_size` column via the covering
+ `ix_ext_storage_extension_id` index (on `extension_id, value_size`)
+ instead of `LENGTH(value)`, so it can run as an index-only scan rather
+ than reading every one of the extension's stored (and potentially
+ TOASTed) blobs. It is still O(rows) for the extension, so write latency
+ grows with row count β acceptable at the row counts a single extension
+ is expected to accumulate, but not a fully constant-time check.
+ """
+ quota = _get_quota()
+ if quota is None:
+ return
+ current_usage = (
+ db.session.query(func.coalesce(func.sum(ExtensionStorage.value_size), 0))
+ .filter(ExtensionStorage.extension_id == extension_id)
+ .scalar()
+ )
+ if current_usage - existing_size + new_size > quota:
+ raise ExtensionStorageQuotaExceeded(extension_id, quota)
+
+
+def _decrypt_if_needed(
+ entry: ExtensionStorage, extension_id: str, key: str
+) -> bytes | None:
+ """Return `entry.value`, decrypted if `entry.is_encrypted`.
+
+ Returns None (and logs) if decryption fails, e.g. after key rotation.
+ """
+ if not entry.is_encrypted:
+ return entry.value
+ try:
+ return _enc_type(entry.user_fk).process_result_value(
+ entry.value, db.engine.dialect
+ )
+ except Exception: # noqa: BLE001
+ logger.error(
+ "Failed to decrypt extension storage value for "
+ "extension_id=%s key=%s β possible key rotation issue",
+ extension_id,
+ key,
+ )
+ return None
+
+
+def _scope_filter(
+ extension_id: str,
+ key: str,
+ user_fk: int | None = None,
+ resource_type: str | None = None,
+ resource_uuid: str | None = None,
+) -> list[object]:
+ """Build the SQLAlchemy filter list for a scoped lookup."""
+ filters: list[object] = [
+ ExtensionStorage.extension_id == extension_id,
+ ExtensionStorage.key == key,
+ ]
+ if user_fk is None:
+ filters.append(ExtensionStorage.user_fk.is_(None))
+ else:
+ filters.append(ExtensionStorage.user_fk == user_fk)
+ if resource_type is None:
+ filters.append(ExtensionStorage.resource_type.is_(None))
+ else:
+ filters.append(ExtensionStorage.resource_type == resource_type)
+ if resource_uuid is None:
+ filters.append(ExtensionStorage.resource_uuid.is_(None))
+ else:
+ filters.append(ExtensionStorage.resource_uuid == resource_uuid)
+ return filters
+
+
+def _list_scope_filter(
+ extension_id: str,
+ user_fk: int | None = None,
+ resource_type: str | None = None,
+ resource_uuid: str | None = None,
+) -> list[object]:
+ """Build the SQLAlchemy filter list for a `list()` call.
+
+ Same scope semantics as `_scope_filter` (an explicit None means "match
+ the global/unset scope"), minus the `key` filter.
+ """
+ filters: list[object] = [ExtensionStorage.extension_id == extension_id]
+ if user_fk is None:
+ filters.append(ExtensionStorage.user_fk.is_(None))
+ else:
+ filters.append(ExtensionStorage.user_fk == user_fk)
+ if resource_type is None:
+ filters.append(ExtensionStorage.resource_type.is_(None))
+ else:
+ filters.append(ExtensionStorage.resource_type == resource_type)
+ if resource_uuid is None:
+ filters.append(ExtensionStorage.resource_uuid.is_(None))
+ else:
+ filters.append(ExtensionStorage.resource_uuid == resource_uuid)
+ return filters
+
+
+@dataclass
+class ExtensionStorageListEntry:
+ """A single row returned by `ExtensionStorageDAO.list_entries`.
+
+ `value` is the raw (decrypted) payload, mirroring `get_value()` β it is
+ not decoded with `codec` here, since REST callers must first check
+ `codec` against `SAFE_CODECS` before deciding whether decoding it is
+ safe to expose over the API, while ambient backend callers (no such
+ restriction) decode unconditionally, same as `get_decoded_value()`.
+ """
+
+ key: str
+ value: bytes | None
+ codec: str
+ is_encrypted: bool
+
+
+class ExtensionStorageDAO(BaseDAO[ExtensionStorage]):
+ """Persistent key-value store for extensions (Tier 3).
+
+ Every query and mutation is scoped to the extension currently executing
+ (resolved from ambient extension context by `ExtensionStorageFilter`),
+ so an extension can never read or modify another extension's storage
+ through this DAO.
+
+ Two access shapes are supported:
+
+ * `get`/`set`/`delete_by_key`/`list_entries` β single-key or
+ paginated-listing operations with explicit scope discriminators
+ (`user_fk`, `resource_type`, `resource_uuid`), used by the ambient
+ `persistent_state`/`ephemeral_state` accessors and the REST API (both
+ of which establish the ambient extension context themselves before
+ calling in).
+ * `find_all`/`find_one_or_none`/`filter_by`/`query`/`create`/`update`/
+ `delete` (inherited from `BaseDAO`) β general-purpose DAO access for
+ backend extension code that needs to enumerate or bulk-manage its own
+ rows, exposed to extensions via `superset_core.extensions.storage.dao`.
+
+ Scope is one of:
+
+ * Global scope β user_fk=None, resource_type=None
+ * User scope β user_fk=<id>, resource_type=None
+ * Resource scope β resource_type + resource_uuid set
+ """
+
+ base_filter = ExtensionStorageFilter
+
+ # ββ Read βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
+
+ @staticmethod
+ def get(
+ extension_id: str,
+ key: str,
+ user_fk: int | None = None,
+ resource_type: str | None = None,
+ resource_uuid: str | None = None,
+ ) -> ExtensionStorage | None:
+ """Return the raw storage entry. The value field may be encrypted."""
+ entry = (
+ db.session.query(ExtensionStorage)
+ .filter(
+ and_(
+ *_scope_filter(
+ extension_id, key, user_fk, resource_type, resource_uuid
+ )
+ )
+ )
+ .first()
+ )
+ return entry
+
+ @staticmethod
+ def get_value(
+ extension_id: str,
+ key: str,
+ user_fk: int | None = None,
+ resource_type: str | None = None,
+ resource_uuid: str | None = None,
+ ) -> bytes | None:
+ """Return the raw (decrypted) value bytes, or None if not found."""
+ entry = ExtensionStorageDAO.get(
+ extension_id, key, user_fk, resource_type, resource_uuid
+ )
+ if entry is None:
+ return None
+ return _decrypt_if_needed(entry, extension_id, key)
+
+ @staticmethod
+ def get_decoded_value(
+ extension_id: str,
+ key: str,
+ user_fk: int | None = None,
+ resource_type: str | None = None,
+ resource_uuid: str | None = None,
+ ) -> Any:
+ """Return the value decoded with the codec it was written with.
+
+ :returns: The decoded value, or None if not found.
+ """
+ entry = ExtensionStorageDAO.get(
+ extension_id, key, user_fk, resource_type, resource_uuid
+ )
+ if entry is None:
+ return None
+ return ExtensionStorageDAO.decode_entry(entry)
+
+ @staticmethod
+ def decode_entry(entry: ExtensionStorage) -> Any:
+ """Decode an already-fetched entry's value with the codec it was
+ written with.
+
+ Unlike `get_decoded_value`, this takes the entry directly instead
+ of re-querying for it, so a caller that already fetched (and, e.g.,
+ validated the codec of) an entry decodes that exact same row
+ instead of racing a concurrent write between two separate reads.
+
+ :returns: The decoded value, or None if the value fails to decrypt.
+ """
+ raw = _decrypt_if_needed(entry, entry.extension_id, entry.key)
+ if raw is None:
+ return None
+ return get_codec(entry.codec).decode(raw)
+
+ @staticmethod
+ def list_entries(
+ extension_id: str,
+ user_fk: int | None = None,
+ resource_type: str | None = None,
+ resource_uuid: str | None = None,
+ page: int = 0,
+ page_size: int = 10,
+ ) -> tuple[list[ExtensionStorageListEntry], int]:
+ """List entries in the given scope, most recently changed first.
+
+ Named `list_entries` rather than `list` (unlike the single-key
+ `get`/`set`/`delete_by_key` shape) because `BaseDAO` already
+ defines a generic `list()` with a different, incompatible
+ signature (search/sort/column-selection over arbitrary models);
+ overriding it here would break that contract for callers relying
+ on polymorphic DAO access.
+
+ Callers may request any `page_size` β there is no row-count
+ ceiling β but the combined `value_size` of the requested page is
+ checked against MAX_LIST_PAYLOAD_SIZE *before* the (potentially
+ large) `value` column is fetched for any row, so an oversized
+ request fails fast rather than reading data it will then reject.
+
+ :returns: (entries, total_count). `entries[i].value` is the raw
+ (decrypted) payload β see `ExtensionStorageListEntry`.
+ :raises ExtensionStorageListPayloadTooLarge: if the requested page's
+ combined value size exceeds MAX_LIST_PAYLOAD_SIZE.
+ """
+ filters = _list_scope_filter(
+ extension_id, user_fk, resource_type, resource_uuid
+ )
+ base_query = db.session.query(ExtensionStorage).filter(and_(*filters))
+ total_count = base_query.count()
+
+ page = max(page, 0)
+ page_size = max(page_size, 1)
+ page_query = (
+ base_query.order_by(desc(ExtensionStorage.changed_on))
+ .offset(page * page_size)
+ .limit(page_size)
+ )
+
+ if (max_payload_size := _get_max_list_payload_size()) is not None:
+ # Sum value_size for exactly this page's rows, without touching
+ # the `value` column, by reusing the same ordered/offset/limited
+ # query as a subquery.
+ subquery = page_query.with_entities(ExtensionStorage.value_size).subquery()
+ requested_size = db.session.query(
+ func.coalesce(func.sum(subquery.c.value_size), 0)
+ ).scalar()
+ if requested_size > max_payload_size:
+ raise ExtensionStorageListPayloadTooLarge(
+ requested_size, max_payload_size
+ )
+
+ entries = [
+ ExtensionStorageListEntry(
+ key=row.key,
+ value=_decrypt_if_needed(row, extension_id, row.key),
+ codec=row.codec,
+ is_encrypted=row.is_encrypted,
+ )
+ for row in page_query.all()
+ ]
+ return entries, total_count
+
+ # ββ Write (upsert) ββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
+
+ @staticmethod
+ @backoff.on_exception(
+ backoff.expo,
+ AcquireDistributedLockFailedException,
+ factor=0.1,
+ base=2,
+ max_tries=8,
+ )
+ def set(
+ extension_id: str,
+ key: str,
+ value: bytes,
+ codec: str = DEFAULT_CODEC,
+ user_fk: int | None = None,
+ resource_type: str | None = None,
+ resource_uuid: str | None = None,
+ encrypt: bool = False,
+ ) -> ExtensionStorage:
+ """Upsert a storage entry. Encrypts value when encrypt=True.
+
+ The select-then-insert/update below is guarded by a distributed lock
+ scoped to the whole extension, for two independent reasons:
+
+ * The row's unique constraint can't be relied on to catch concurrent
+ writes to the same key, since `user_fk`, `resource_type`, and
+ `resource_uuid` are nullable and standard SQL never treats
+ NULL = NULL, so two concurrent inserts for the same *global* or
+ *resource*-scoped key (where one or more of those columns is NULL)
+ would not violate it and would land as duplicate rows.
+ * `_check_quota` below enforces usage extension-wide, summed across
+ every key. A lock scoped to only this key would let concurrent
+ writes to *different* keys of the same extension each read usage
+ before the other commits, both pass the check, and land a combined
+ write that exceeds the extension's quota.
+
+ :raises ExtensionStorageKeyTooLong: if `key` exceeds MAX_KEY_LENGTH.
+ :raises ExtensionStorageValueTooLarge: if `value` exceeds
+ MAX_VALUE_SIZE.
+ :raises ExtensionStorageQuotaExceeded: if writing `value` would push
+ the extension over its total storage quota.
+ """
+ _validate_key_length(key)
+ _validate_value_size(value)
+ stored_value = (
+ _enc_type(user_fk).process_bind_param(value, db.engine.dialect)
+ if encrypt
+ else value
+ )
+
+ with DistributedLock(
+ namespace="extension_storage_set",
+ ttl_seconds=SET_LOCK_TTL_SECONDS,
+ extension_id=extension_id,
+ ):
+ entry = (
+ db.session.query(ExtensionStorage)
+ .filter(
+ and_(
+ *_scope_filter(
+ extension_id, key, user_fk, resource_type, resource_uuid
+ )
+ )
+ )
+ .first()
+ )
+ new_size = len(stored_value)
+ existing_size = entry.value_size if entry is not None else 0
+ _check_quota(extension_id, new_size, existing_size)
+ if entry is not None:
+ entry.value = stored_value
+ entry.value_size = new_size
+ entry.codec = codec
+ entry.is_encrypted = encrypt
+ else:
+ entry = ExtensionStorage(
+ extension_id=extension_id,
+ key=key,
+ value=stored_value,
+ value_size=new_size,
+ codec=codec,
+ user_fk=user_fk,
+ resource_type=resource_type,
+ resource_uuid=resource_uuid,
+ is_encrypted=encrypt,
+ )
+ db.session.add(entry)
+ db.session.flush()
+ return entry
+
+ # ββ Create (disallowed) βββββββββββββββββββββββββββββββββββββββββββββββββββββ
+
+ @classmethod
+ def create(
+ cls,
+ item: ExtensionStorage | None = None,
+ attributes: dict[str, Any] | None = None,
+ ) -> ExtensionStorage:
+ """Not supported for this DAO.
+
+ `create()` is `BaseDAO`'s raw insert: no upsert dedup against an
+ existing row for the same scoped key, no quota/key-length/value-size
+ validation, and no locking β calling it directly could construct a
+ duplicate row for a key `set()` already wrote (see `set()`'s
+ docstring) without any of the checks that make that safe.
+
+ :raises NotImplementedError: always; use `ExtensionStorageDAO.set()`
+ (or the ambient `ctx.storage.persistent.set()` accessor) instead.
+ """
+ raise NotImplementedError(
+ "ExtensionStorageDAO.create() is not supported for extension "
+ "storage: it bypasses the locking, quota, and validation that "
+ "make writing a value safe. Use ctx.storage.persistent.set() "
+ "(or ExtensionStorageDAO.set() from backend code) instead."
+ )
+
+ # ββ Delete ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
+
+ @staticmethod
+ def delete_by_key(
+ extension_id: str,
+ key: str,
+ user_fk: int | None = None,
+ resource_type: str | None = None,
+ resource_uuid: str | None = None,
+ ) -> bool:
+ """Delete a single entry by key. Returns True if a row was removed."""
+ entry = (
+ db.session.query(ExtensionStorage)
+ .filter(
+ and_(
+ *_scope_filter(
+ extension_id, key, user_fk, resource_type, resource_uuid
+ )
+ )
+ )
+ .first()
+ )
+ if entry is None:
+ return False
+ db.session.delete(entry)
+ db.session.flush()
+ return True
Review Comment:
**Suggestion:** This delete path removes only the first matching row; if duplicate rows exist for the same scoped key (which this file already documents can happen with nullable unique columns under races), stale duplicates remain and `remove` will appear to succeed while data still exists. Delete all rows matching the scope instead of a single `.first()` result. [logic error]
<details>
<summary><b>Severity Level:</b> Major β οΈ</summary>
```mdx
- β Delete operations may not fully remove logical key data.
- β Persistent quotas not reclaimed when duplicates remain.
- β οΈ Extensions can still read values after successful delete.
```
</details>
<details>
<summary><b>Steps of Reproduction β
</b></summary>
```mdx
1. Concurrent writes for a global or resource-scoped key occur as in
`ExtensionStorageDAO.set()` (superset/extensions/storage/persistent_dao.py:504-583): the
docstring at lines 515-535 explains that nullable columns in the uniqueness constraint can
allow duplicate rows under races, hence the need for a distributed lock.
2. Because the distributed lock is released before commit (see reproduction for suggestion
2), it is possible for two transactions to each pass `_check_quota(...)` and insert rows
for the same `(extension_id, key, user_fk, resource_type, resource_uuid)` scope, leaving
duplicate `ExtensionStorage` rows in the database.
3. An extension later calls `ctx.storage.persistent.delete('job_progress', ...)`; the
ambient delete accessor in `superset/extensions/storage/persistent.py` invokes
`ExtensionStorageDAO.delete_by_key()`
(superset/extensions/storage/persistent_dao.py:612-637) to remove the key.
4. `delete_by_key()` constructs a scoped query using `_scope_filter(...)` and calls
`.first()` at lines 622-632, deletes that single `entry`, flushes at lines 635-636, and
returns True; any other duplicate rows matching the same scope remain, so subsequent reads
via `ExtensionStorageDAO.get()` (lines 354-373) still find a value, and the extension
observes the key as present despite a successful delete.
```
</details>
[](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=0ef3457e08bb494a9c6de731b24a8f8e&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=0ef3457e08bb494a9c6de731b24a8f8e&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset)
*(Use Cmd/Ctrl + Click for best experience)*
<details>
<summary><b>Prompt for AI Agent π€ </b></summary>
```mdx
This is a comment left during a code review.
**Path:** superset/extensions/storage/persistent_dao.py
**Line:** 622:637
**Comment:**
*Logic Error: This delete path removes only the first matching row; if duplicate rows exist for the same scoped key (which this file already documents can happen with nullable unique columns under races), stale duplicates remain and `remove` will appear to succeed while data still exists. Delete all rows matching the scope instead of a single `.first()` result.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F39171&comment_hash=ffacfad14a7db449adbd393d19d71e3f8657aadb421e48d80001966bf6ad2588&reaction=like'>π</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F39171&comment_hash=ffacfad14a7db449adbd393d19d71e3f8657aadb421e48d80001966bf6ad2588&reaction=dislike'>π</a>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(extensions): add a Storage API for extensions [superset]
Posted by "codeant-ai-for-open-source[bot] (via GitHub)" <gi...@apache.org>.
codeant-ai-for-open-source[bot] commented on code in PR #39171:
URL: https://github.com/apache/superset/pull/39171#discussion_r3624434959
##########
superset-core/src/superset_core/extensions/storage/ephemeral.py:
##########
@@ -0,0 +1,144 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""
+Ephemeral State API for superset-core extensions (Tier 2 Storage).
+
+Provides short-lived KV storage backed by the configured server-side cache
+backend (Redis, Memcached, or filesystem). Automatically expires based on TTL.
+Not guaranteed to survive server restarts.
+
+Host implementations will replace the EphemeralState class during initialization
+with a concrete implementation providing actual functionality.
+
+Cache keys are namespaced automatically:
+- User-scoped (default): superset-ext:{extension_id}:user:{user_id}:{key}
+- Shared (global): superset-ext:{extension_id}:shared:{key}
+
+Usage (via extension context - preferred):
+ from superset_core.extensions.context import get_context
+
+ ctx = get_context()
+
+ # User-scoped state (default - private to current user)
+ ctx.storage.ephemeral.get('preference')
+ ctx.storage.ephemeral.set('preference', 'compact', EphemeralSetOptions(ttl=3600))
+ ctx.storage.ephemeral.remove('preference')
+
+ # Shared state (explicit opt-in - visible to all users)
+ ctx.storage.ephemeral.shared.get('job_progress')
+ ctx.storage.ephemeral.shared.set(
+ 'job_progress', {'pct': 42}, EphemeralSetOptions(ttl=3600)
+ )
+ ctx.storage.ephemeral.shared.remove('job_progress')
+"""
+
+from dataclasses import dataclass
+from typing import Any, ClassVar, Protocol
+
+
+@dataclass(frozen=True)
+class EphemeralSetOptions:
+ """
+ Options for an ephemeral state `set` call.
+
+ `ttl` is required (no default): every write must have an explicit
+ expiration, matching the frontend `EphemeralSetOptions` and the REST API.
+ """
+
+ ttl: int
+ #: Name of the codec used to encode `value`, e.g. "json" (default).
+ codec: str = "json"
+
+
+class EphemeralStateAccessor(Protocol):
+ """Protocol for scoped ephemeral state access."""
+
+ def get(self, key: str) -> Any:
+ """Get a value from ephemeral state."""
+ ...
+
+ def set(self, key: str, value: Any, options: EphemeralSetOptions) -> None:
+ """Set a value in ephemeral state with TTL."""
+ ...
+
+ def remove(self, key: str) -> None:
+ """Remove a value from ephemeral state."""
+ ...
+
+
+class EphemeralState:
+ """
+ Tier 2 ephemeral state storage for extensions.
+
+ Backed by the configured server-side cache (Redis, Memcached, or filesystem).
+ Data expires based on TTL and is not guaranteed to survive server restarts.
+
+ Host implementations will replace this class during initialization
+ with a concrete implementation providing actual functionality.
+
+ All operations are user-scoped by default (private to the current user).
+ Use `.shared` to access state that is visible to all users.
+ """
+
+ @staticmethod
+ def get(key: str) -> Any:
+ """
+ Get a value from user-scoped ephemeral state.
+
+ Data is automatically scoped to the current authenticated user.
+ Other users cannot see or modify this data.
+
+ :param key: The key to retrieve.
+ :returns: The stored value, or None if not found or expired.
+ """
+ raise NotImplementedError("Class will be replaced during initialization")
+
+ @staticmethod
+ def set(key: str, value: Any, options: EphemeralSetOptions) -> None:
+ """
+ Set a value in user-scoped ephemeral state with TTL.
+
+ Data is automatically scoped to the current authenticated user.
+ Other users cannot see or modify this data.
+
+ :param key: The key to store.
+ :param value: The value to store, encoded with `options.codec`
+ (default "json"). The encoded value must not exceed
+ MAX_VALUE_SIZE from config.
+ :param options: `EphemeralSetOptions`, e.g. `ttl=3600`. Required β
+ `ttl` must not exceed MAX_TTL from config. `codec="pickle"`
+ stores a value that isn't JSON-serializable.
+ """
+ raise NotImplementedError("Class will be replaced during initialization")
+
+ @staticmethod
+ def remove(key: str) -> None:
+ """
+ Remove a value from user-scoped ephemeral state.
+
+ Data is automatically scoped to the current authenticated user.
+
+ :param key: The key to remove.
+ """
+ raise NotImplementedError("Class will be replaced during initialization")
+
+ #: Shared (global) ephemeral state accessor.
+ #: Data stored via this accessor is visible to all users of the extension.
+ #: WARNING: Do not store user-specific or sensitive data here.
+ #: Host implementations will set this during initialization.
+ shared: ClassVar[EphemeralStateAccessor]
Review Comment:
**Suggestion:** `shared` is only annotated but never initialized, so accessing `EphemeralState.shared` before host-side class replacement will raise `AttributeError` instead of the expected placeholder behavior. Initialize it with a stub accessor (that raises `NotImplementedError`) or set a safe default during class definition to keep the API contract consistent. [possible bug]
<details>
<summary><b>Severity Level:</b> Major β οΈ</summary>
```mdx
- β Direct use of shared ephemeral storage crashes with AttributeError.
- β οΈ Misconfigured hosts crash when shared accessor is used.
```
</details>
<details>
<summary><b>Steps of Reproduction β
</b></summary>
```mdx
1. Start a Python shell with the PR code installed and import the class: `from
superset_core.extensions.storage.ephemeral import EphemeralState` (class defined at line
84 in this file).
2. Without any host-side initialization or class replacement, access the shared accessor:
`EphemeralState.shared.get("foo")`, referencing the annotated attribute at line 144.
3. Python attempts to look up the `shared` attribute on `EphemeralState` but finds only a
type annotation; no actual class attribute was created during definition.
4. Observe `AttributeError: type object 'EphemeralState' has no attribute 'shared'`
instead of the placeholder `NotImplementedError` used by the static methods at lines 98,
111, and 129, leading to an inconsistent failure mode for shared ephemeral storage before
host injection.
```
</details>
[](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=33228fcdc43646cabb7479710f10b0c2&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=33228fcdc43646cabb7479710f10b0c2&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset)
*(Use Cmd/Ctrl + Click for best experience)*
<details>
<summary><b>Prompt for AI Agent π€ </b></summary>
```mdx
This is a comment left during a code review.
**Path:** superset-core/src/superset_core/extensions/storage/ephemeral.py
**Line:** 144:144
**Comment:**
*Possible Bug: `shared` is only annotated but never initialized, so accessing `EphemeralState.shared` before host-side class replacement will raise `AttributeError` instead of the expected placeholder behavior. Initialize it with a stub accessor (that raises `NotImplementedError`) or set a safe default during class definition to keep the API contract consistent.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F39171&comment_hash=a43b2ed195e85a6d7da0cba41e0d405e03a2754d58a3faf83b228b9949c326e5&reaction=like'>π</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F39171&comment_hash=a43b2ed195e85a6d7da0cba41e0d405e03a2754d58a3faf83b228b9949c326e5&reaction=dislike'>π</a>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org