You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@superset.apache.org by "aminghadersohi (via GitHub)" <gi...@apache.org> on 2025/09/04 16:59:35 UTC
[PR] Feat/dao updates [superset]
aminghadersohi opened a new pull request, #35018:
URL: https://github.com/apache/superset/pull/35018
<!---
Please write the PR title following the conventions at https://www.conventionalcommits.org/en/v1.0.0/
Example:
fix(dashboard): load charts correctly
-->
### SUMMARY
<!--- Describe the change below, including rationale and design decisions -->
This PR enhances the BaseDAO layer to support structured programmatic filtering, enabling external API integrations (like MCP services) to query Superset data without Flask-AppBuilder dependencies. The changes are fully backward compatible with zero breaking changes.
#### Key Enhancements
**1. Structured Filtering System**
- Added `ColumnOperator` model with 14 filter operators (eq, neq, gt, gte, lt, lte, like, ilike, in, nin, null, nnull, sw, nsw)
- Enables type-safe, programmatic filtering without FAB web dependencies
- Supports complex queries with multiple filters and operators
**2. Enhanced Query Methods**
- New `list()` method with comprehensive filtering, sorting, pagination, and search
- New `count()` method for filtered record counting
- Schema discovery via `get_filterable_columns_and_operators()` for API documentation
**3. UUID Support & Flexible ID Handling**
- Automatic UUID type conversion for gradual migration to UUID primary keys
- Support for non-standard ID field names across models
- Enhanced `find_by_id()`/`find_by_ids()` with string/UUID flexibility
**4. Model-Specific Extensions**
- ChartDAO: Added filterable fields for `tags`, `owner`, `viz_type`, `datasource_name`
- DashboardDAO: Added filterable fields for `tags`, `owner`, `published`, `favorite`
- DatasetDAO: Added filterable fields for `tags`, `owner`
#### Design Rationale
The existing DAO layer lacked programmatic filtering capabilities needed by external services. This enhancement:
- Maintains complete backward compatibility (all existing code continues working)
- Follows Superset's patterns and conventions
- Provides type safety with MyPy compliance
- Enables self-documenting APIs through schema discovery
- Supports the ongoing UUID migration initiative
### BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF
<!--- Skip this if not applicable -->
N/A - Backend infrastructure change with no UI impact
### TESTING INSTRUCTIONS
<!--- Required! What steps can be taken to manually verify the changes? -->
1. **Run the comprehensive test suite:**
```bash
# Run the new DAO tests (2400+ lines of test coverage)
pytest tests/unit_tests/dao/
```
2. **Verify backward compatibility:**
```bash
# Existing DAO tests should all pass unchanged
pytest tests/unit_tests/daos/
pytest tests/integration_tests/charts/
pytest tests/integration_tests/dashboards/
pytest tests/integration_tests/datasets/
```
3. **Test programmatic filtering (Python shell example):**
```python
from superset.daos.chart import ChartDAO
from superset.daos.base import ColumnOperator, ColumnOperatorEnum
# Test filtering
charts, total = ChartDAO.list(
column_operators=[
ColumnOperator(col="viz_type", opr=ColumnOperatorEnum.eq, value="line")
],
page_size=10
)
# Test schema discovery
schema = ChartDAO.get_filterable_columns_and_operators()
assert "viz_type" in schema
```
4. **Verify type checking:**
```bash
pre-commit run mypy
```
### ADDITIONAL INFORMATION
<!--- Check any relevant boxes with "x" -->
<!--- HINT: Include "Fixes #nnn" if you are fixing an existing issue -->
- [ ] Has associated issue:
- [ ] Required feature flags:
- [ ] Changes UI
- [ ] Includes DB Migration (follow approval process in [SIP-59](https://github.com/apache/superset/issues/13351))
- [ ] Migration is atomic, supports rollback & is backwards-compatible
- [ ] Confirm DB migration upgrade and downgrade tested
- [ ] Runtime estimates and downtime expectations provided
- [x] Introduces new feature or API
- New DAO methods: `list()`, `count()`, `get_filterable_columns_and_operators()`
- New filtering system via `ColumnOperator` and `ColumnOperatorEnum`
- Fully backward compatible - no breaking changes
- [ ] Removes existing feature or API
**Additional Notes:**
- This PR is the foundation for the MCP (Model Context Protocol) service integration
- Comprehensive test coverage: 2400+ lines of tests covering all operators and edge cases
- Follows Superset's ongoing initiatives: UUID migration, type safety (MyPy), and API standardization
- Zero performance impact on existing code paths - new features are opt-in via optional parameters
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat: Add BaseDAO improvements and test reorganization [superset]
Posted by "aminghadersohi (via GitHub)" <gi...@apache.org>.
aminghadersohi commented on code in PR #35018:
URL: https://github.com/apache/superset/pull/35018#discussion_r2334689726
##########
superset/daos/base.py:
##########
@@ -251,3 +455,205 @@ def filter_by(cls, **filter_by: Any) -> list[T]:
cls.id_column_name, data_model
).apply(query, None)
return query.filter_by(**filter_by).all()
+
+ @classmethod
+ def apply_column_operators(
+ cls, query: Any, column_operators: Optional[List[ColumnOperator]] = None
+ ) -> Any:
+ """
+ Apply column operators (list of ColumnOperator) to the query using
+ ColumnOperatorEnum logic. Raises ValueError if a filter references a
+ non-existent column.
+ """
+ if not column_operators:
+ return query
+ for c in column_operators:
+ if not isinstance(c, ColumnOperator):
+ continue
+ col = c.col
+ opr = c.opr
+ value = c.value
+ if not col or not hasattr(cls.model_cls, col):
+ model_name = cls.model_cls.__name__ if cls.model_cls else "Unknown"
+ logging.error(
+ f"Invalid filter: column '{col}' does not exist on {model_name}"
+ )
+ raise ValueError(
+ f"Invalid filter: column '{col}' does not exist on {model_name}"
+ )
+ column = getattr(cls.model_cls, col)
+ try:
+ # Always use ColumnOperatorEnum's apply method
+ operator_enum = ColumnOperatorEnum(opr)
+ query = query.filter(operator_enum.apply(column, value))
+ except Exception as e:
+ logging.error(f"Error applying filter on column '{col}': {e}")
+ raise
+ return query
+
+ @classmethod
+ def get_filterable_columns_and_operators(cls) -> Dict[str, List[str]]:
+ """
+ Returns a dict mapping filterable columns (including hybrid/computed fields if
+ present) to their supported operators. Used by MCP tools to dynamically expose
+ filter options. Custom fields supported by the DAO but not present on the model
+ should be documented here.
+ """
+
+ mapper = inspect(cls.model_cls)
+ columns = {c.key: c for c in mapper.columns}
+ # Add hybrid properties
+ hybrids = {
+ name: attr
+ for name, attr in vars(cls.model_cls).items()
+ if isinstance(attr, hybrid_property)
+ }
+ # You may add custom fields here, e.g.:
+ # custom_fields = {"tags": ["eq", "in_", "like"], ...}
+ custom_fields: Dict[str, List[str]] = {}
+
+ filterable = {}
+ for name, col in columns.items():
+ if isinstance(col.type, (sa.String, sa.Text)):
+ filterable[name] = TYPE_OPERATOR_MAP["string"]
+ elif isinstance(col.type, (sa.Boolean,)):
+ filterable[name] = TYPE_OPERATOR_MAP["boolean"]
+ elif isinstance(col.type, (sa.Integer, sa.Float, sa.Numeric)):
+ filterable[name] = TYPE_OPERATOR_MAP["number"]
+ elif isinstance(col.type, (sa.DateTime, sa.Date, sa.Time)):
+ filterable[name] = TYPE_OPERATOR_MAP["datetime"]
+ else:
+ # Fallback to eq/ne/null
+ filterable[name] = ["eq", "ne", "is_null", "is_not_null"]
+ # Add hybrid properties as string fields by default
+ for name in hybrids:
+ filterable[name] = TYPE_OPERATOR_MAP["string"]
+ # Add custom fields
+ filterable.update(custom_fields)
+ return filterable
+
+ @classmethod
+ def _build_query(
+ cls,
+ column_operators: Optional[List[ColumnOperator]] = None,
+ search: Optional[str] = None,
+ search_columns: Optional[List[str]] = None,
+ custom_filters: Optional[Dict[str, BaseFilter]] = None,
+ skip_base_filter: bool = False,
+ data_model: Optional[SQLAInterface] = None,
+ ) -> Any:
+ """
+ Build a SQLAlchemy query with base filter, column operators, search, and
+ custom filters.
+ """
+ if data_model is None:
+ data_model = SQLAInterface(cls.model_cls, db.session)
+ query = data_model.session.query(cls.model_cls)
+ query = cls._apply_base_filter(
+ query, skip_base_filter=skip_base_filter, data_model=data_model
+ )
+ if search and search_columns:
+ search_filters = []
+ for column_name in search_columns:
+ if hasattr(cls.model_cls, column_name):
+ column = getattr(cls.model_cls, column_name)
+ search_filters.append(cast(column, Text).ilike(f"%{search}%"))
+ if search_filters:
+ query = query.filter(or_(*search_filters))
+ if custom_filters:
+ for filter_class in custom_filters.values():
+ query = filter_class.apply(query, None)
+ if column_operators:
+ query = cls.apply_column_operators(query, column_operators)
+ return query
+
+ @classmethod
+ def list( # noqa: C901
+ cls,
+ column_operators: Optional[List[ColumnOperator]] = None,
+ order_column: str = "changed_on",
+ order_direction: str = "desc",
+ page: int = 0,
+ page_size: int = 100,
+ search: Optional[str] = None,
+ search_columns: Optional[List[str]] = None,
+ custom_filters: Optional[Dict[str, BaseFilter]] = None,
+ columns: Optional[List[str]] = None,
+ ) -> Tuple[List[Any], int]:
+ """
+ Generic list method for filtered, sorted, and paginated results.
+ If columns is specified, returns a list of tuples (one per row),
+ otherwise returns model instances.
+ """
+ data_model = SQLAInterface(cls.model_cls, db.session)
+
+ column_attrs = []
+ relationship_loads = []
+ if columns is None:
+ columns = []
+ for name in columns:
+ attr = getattr(cls.model_cls, name, None)
+ if attr is None:
+ continue
+ prop = getattr(attr, "property", None)
+ if isinstance(prop, ColumnProperty):
+ column_attrs.append(attr)
+ elif isinstance(prop, RelationshipProperty):
+ relationship_loads.append(joinedload(attr))
Review Comment:
You're absolutely right! Using joinedload() with many-to-many relationships will break pagination:
1. The count happens after joins (line 627), giving inflated counts
2. LIMIT/OFFSET apply to joined rows, not entities
3. No tests exist for relationship loading in the list method
We should either:
- Use subqueryload() instead of joinedload() for collection
relationships
- Move the count before adding joins
- Add distinct() when many-to-many relationships are detected
I'll add tests for this scenario.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat: Add BaseDAO improvements and test reorganization [superset]
Posted by "aminghadersohi (via GitHub)" <gi...@apache.org>.
aminghadersohi commented on PR #35018:
URL: https://github.com/apache/superset/pull/35018#issuecomment-3272290944
I have addressed all comments so far. Thank you for the very thorough review @dpgaspar !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat: Add BaseDAO improvements and test reorganization [superset]
Posted by "bito-code-review[bot] (via GitHub)" <gi...@apache.org>.
bito-code-review[bot] commented on code in PR #35018:
URL: https://github.com/apache/superset/pull/35018#discussion_r2330784881
##########
superset/daos/dashboard.py:
##########
@@ -44,10 +44,25 @@
logger = logging.getLogger(__name__)
+# Custom filterable fields for dashboards
+DASHBOARD_CUSTOM_FIELDS = {
+ "tags": ["eq", "in_", "like"],
+ "owner": ["eq", "in_"],
+ "published": ["eq"],
+ "favorite": ["eq"],
Review Comment:
<div>
<div id="suggestion">
<div id="issue"><b>Invalid filterable field</b></div>
<div id="fix">
The 'favorite' field is not a valid column or relationship on the Dashboard model. Filtering by favorite status is handled through the separate `FavStar` relationship via the `favorited_ids` method (lines 284-296), not through direct dashboard filtering. Including 'favorite' in DASHBOARD_CUSTOM_FIELDS will cause filtering failures since there's no corresponding database column or relationship to filter on.
</div>
<details>
<summary>
<b>Code suggestion</b>
</summary>
<blockquote>Check the AI-generated fix before applying</blockquote>
<div id="code">
```suggestion
DASHBOARD_CUSTOM_FIELDS = {
"tags": ["eq", "in_", "like"],
"owner": ["eq", "in_"],
"published": ["eq"],
```
</div>
</details>
</div>
<small><i>Code Review Run <a href=https://github.com/apache/superset/pull/35018#issuecomment-3267072028>#3718e8</a></i></small>
</div>
---
Should Bito avoid suggestions like this for future reviews? (<a href=https://alpha.bito.ai/home/ai-agents/review-rules>Manage Rules</a>)
- [ ] Yes, avoid them
##########
requirements/development.in:
##########
@@ -16,5 +16,5 @@
# specific language governing permissions and limitations
# under the License.
#
--e .[development,bigquery,druid,duckdb,gevent,gsheets,mysql,postgres,presto,prophet,trino,thumbnails]
+-e .[development,bigquery,druid,duckdb,gevent,gsheets,mysql,postgres,presto,prophet,pydantic,trino,thumbnails]
Review Comment:
<div>
<div id="suggestion">
<div id="issue"><b>Invalid extra dependency</b></div>
<div id="fix">
The change attempts to add `pydantic` as an optional dependency extra, but this will break the installation because pydantic is already defined as a core dependency in pyproject.toml line 87 and no `[pydantic]` extra is defined in the `[project.optional-dependencies]` section. This will cause pip to fail with 'no such option' error when trying to install the non-existent pydantic extra. Remove pydantic from the extras list since it's already included as a core dependency.
</div>
<details>
<summary>
<b>Code suggestion</b>
</summary>
<blockquote>Check the AI-generated fix before applying</blockquote>
<div id="code">
```
--e .[development,bigquery,druid,duckdb,gevent,gsheets,mysql,postgres,presto,prophet,pydantic,trino,thumbnails]
+-e .[development,bigquery,druid,duckdb,gevent,gsheets,mysql,postgres,presto,prophet,trino,thumbnails]
```
</div>
</details>
</div>
<small><i>Code Review Run <a href=https://github.com/apache/superset/pull/35018#issuecomment-3267072028>#3718e8</a></i></small>
</div>
---
Should Bito avoid suggestions like this for future reviews? (<a href=https://alpha.bito.ai/home/ai-agents/review-rules>Manage Rules</a>)
- [ ] Yes, avoid them
##########
tests/integration_tests/dao/base_dao_test.py:
##########
@@ -0,0 +1,1397 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""
+Integration tests for BaseDAO functionality.
+
+This module contains comprehensive integration tests for the BaseDAO class and its
+subclasses, covering database operations, CRUD methods, flexible column support,
+column operators, and error handling.
+
+Tests use an in-memory SQLite database for isolation and to replicate the unit test
+environment behavior. User model deletions are avoided due to circular dependency
+constraints with self-referential foreign keys.
+"""
+
+import datetime
+import time
+import uuid
+
+import pytest
+from flask_appbuilder.models.filters import BaseFilter
+from flask_appbuilder.security.sqla.models import User
+from sqlalchemy import Column, DateTime, Integer, String
+from sqlalchemy.ext.declarative import declarative_base
+from sqlalchemy.orm.session import Session
+
+from superset.daos.base import BaseDAO, ColumnOperator, ColumnOperatorEnum
+from superset.daos.chart import ChartDAO
+from superset.daos.dashboard import DashboardDAO
+from superset.daos.database import DatabaseDAO
+from superset.daos.user import UserDAO
+from superset.extensions import db
+from superset.models.core import Database
+from superset.models.dashboard import Dashboard
+from superset.models.slice import Slice
+
+# Create a test model for comprehensive testing
+Base = declarative_base()
+
+
+class ExampleModel(Base): # type: ignore
+ __tablename__ = "example_model"
+ id = Column(Integer, primary_key=True)
+ uuid = Column(String(36), unique=True, nullable=False)
+ slug = Column(String(100), unique=True)
+ name = Column(String(100))
+ code = Column(String(50), unique=True)
+ created_on = Column(DateTime, default=datetime.datetime.utcnow)
+
+
+class ExampleModelDAO(BaseDAO[ExampleModel]):
+ model_cls = ExampleModel
+ id_column_name = "id"
+ base_filter = None
+
+
+class MockModel:
+ def __init__(self, id=1, name="test"):
+ self.id = id
+ self.name = name
+
+
+class TestDAO(BaseDAO[MockModel]):
+ model_cls = MockModel
+
+
+@pytest.fixture(autouse=True)
+def mock_g_user(app_context):
+ """Mock the flask g.user for security context."""
+ # Within app context, we can safely mock g
+ from flask import g
+
+ mock_user = User()
+ mock_user.id = 1
+ mock_user.username = "test_user"
+
+ # Set g.user directly instead of patching
+ g.user = mock_user
+ yield
+
+ # Clean up
+ if hasattr(g, "user"):
+ delattr(g, "user")
+
+
+# =============================================================================
+# Integration Tests - These tests use the actual database
+# =============================================================================
+
+
+def test_column_operator_enum_complete_coverage(user_with_data: Session) -> None:
+ """
+ Test that every single ColumnOperatorEnum operator is covered by tests.
+ This ensures we have comprehensive test coverage for all operators.
+ """
+ # Simply verify that we can create queries with all operators
+ for operator in ColumnOperatorEnum:
+ column_operator = ColumnOperator(
+ col="username", opr=operator, value="test_value"
+ )
+ # Just check it doesn't raise an error
+ assert column_operator.opr == operator
+
+
+def test_find_by_id_with_default_column(app_context: Session) -> None:
+ """Test find_by_id with default 'id' column."""
+ # Create a user to test with
+ user = User(
+ username="test_find_by_id",
+ first_name="Test",
+ last_name="User",
+ email="test@example.com",
+ active=True,
+ )
+ db.session.add(user)
+ db.session.commit()
+
+ # Find by numeric id
+ found = UserDAO.find_by_id(user.id, skip_base_filter=True)
+ assert found is not None
+ assert found.id == user.id
+ assert found.username == "test_find_by_id"
+
+ # Test with non-existent id
+ not_found = UserDAO.find_by_id(999999, skip_base_filter=True)
+ assert not_found is None
+
+
+def test_find_by_id_with_uuid_column(app_context: Session) -> None:
+ """Test find_by_id with custom uuid column."""
+ # Create a dashboard with uuid
+ dashboard = Dashboard(
+ dashboard_title="Test UUID Dashboard",
+ slug="test-uuid-dashboard",
+ published=True,
+ )
+ db.session.add(dashboard)
+ db.session.commit()
+
+ # Find by uuid string using the uuid column
+ found = DashboardDAO.find_by_id(
+ str(dashboard.uuid), id_column="uuid", skip_base_filter=True
+ )
+ assert found is not None
+ assert found.uuid == dashboard.uuid
+ assert found.dashboard_title == "Test UUID Dashboard"
+
+ # Find by numeric id (should still work)
+ found_by_id = DashboardDAO.find_by_id(dashboard.id, skip_base_filter=True)
+ assert found_by_id is not None
+ assert found_by_id.id == dashboard.id
+
+ # Test with non-existent uuid
+ not_found = DashboardDAO.find_by_id(str(uuid.uuid4()), skip_base_filter=True)
+ assert not_found is None
+
+
+def test_find_by_id_with_slug_column(app_context: Session) -> None:
+ """Test find_by_id with slug column fallback."""
+ # Create a dashboard with slug
+ dashboard = Dashboard(
+ dashboard_title="Test Slug Dashboard",
+ slug="test-slug-dashboard",
+ published=True,
+ )
+ db.session.add(dashboard)
+ db.session.commit()
+
+ # Find by slug using the slug column
+ found = DashboardDAO.find_by_id(
+ "test-slug-dashboard", id_column="slug", skip_base_filter=True
+ )
+ assert found is not None
+ assert found.slug == "test-slug-dashboard"
+ assert found.dashboard_title == "Test Slug Dashboard"
+
+ # Test with non-existent slug
+ not_found = DashboardDAO.find_by_id("non-existent-slug", skip_base_filter=True)
+ assert not_found is None
+
+
+def test_find_by_id_with_invalid_column(app_context: Session) -> None:
+ """Test find_by_id returns None when column doesn't exist."""
+ # This should return None gracefully
+ result = UserDAO.find_by_id("not_a_valid_id", skip_base_filter=True)
+ assert result is None
+
+
+def test_find_by_id_skip_base_filter(app_context: Session) -> None:
+ """Test find_by_id with skip_base_filter parameter."""
+ # Create users with different active states
+ active_user = User(
+ username="active_user",
+ first_name="Active",
+ last_name="User",
+ email="active@example.com",
+ active=True,
+ )
+ inactive_user = User(
+ username="inactive_user",
+ first_name="Inactive",
+ last_name="User",
+ email="inactive@example.com",
+ active=False,
+ )
+ db.session.add_all([active_user, inactive_user])
+ db.session.commit()
+
+ # Without skipping base filter (if one exists)
+ found_active = UserDAO.find_by_id(active_user.id, skip_base_filter=False)
+ assert found_active is not None
+
+ # With skipping base filter
+ found_active_skip = UserDAO.find_by_id(active_user.id, skip_base_filter=True)
+ assert found_active_skip is not None
+
+ # Both should find the user since UserDAO might not have a base filter
+ assert found_active.id == active_user.id
+ assert found_active_skip.id == active_user.id
+
+
+def test_find_by_ids_with_default_column(app_context: Session) -> None:
+ """Test find_by_ids with default 'id' column."""
+ # Create multiple users
+ users = []
+ for i in range(3):
+ user = User(
+ username=f"test_find_by_ids_{i}",
+ first_name=f"Test{i}",
+ last_name="User",
+ email=f"test{i}@example.com",
+ active=True,
+ )
+ users.append(user)
+ db.session.add(user)
+ db.session.commit()
+
+ # Find by multiple ids
+ ids = [user.id for user in users]
+ found = UserDAO.find_by_ids(ids, skip_base_filter=True)
+ assert len(found) == 3
+ found_ids = [u.id for u in found]
+ assert set(found_ids) == set(ids)
+
+ # Test with mix of existent and non-existent ids
+ mixed_ids = [users[0].id, 999999, users[1].id]
+ found_mixed = UserDAO.find_by_ids(mixed_ids, skip_base_filter=True)
+ assert len(found_mixed) == 2
+
+ # Test with empty list
+ found_empty = UserDAO.find_by_ids([], skip_base_filter=True)
+ assert found_empty == []
+
+
+def test_find_by_ids_with_uuid_column(app_context: Session) -> None:
+ """Test find_by_ids with uuid column."""
+ # Create multiple dashboards
+ dashboards = []
+ for i in range(3):
+ dashboard = Dashboard(
+ dashboard_title=f"Test UUID Dashboard {i}",
+ slug=f"test-uuid-dashboard-{i}",
+ published=True,
+ )
+ dashboards.append(dashboard)
+ db.session.add(dashboard)
+ db.session.commit()
+
+ # Find by multiple uuids
+ uuids = [str(dashboard.uuid) for dashboard in dashboards]
+ found = DashboardDAO.find_by_ids(uuids, id_column="uuid", skip_base_filter=True)
+ assert len(found) == 3
+ found_uuids = [str(d.uuid) for d in found]
+ assert set(found_uuids) == set(uuids)
+
+ # Test with mix of ids and uuids - search separately by column
+ found_by_id = DashboardDAO.find_by_ids([dashboards[0].id], skip_base_filter=True)
+ found_by_uuid = DashboardDAO.find_by_ids(
+ [str(dashboards[1].uuid)], id_column="uuid", skip_base_filter=True
+ )
+ assert len(found_by_id) == 1
+ assert len(found_by_uuid) == 1
+
+
+def test_find_by_ids_with_slug_column(app_context: Session) -> None:
+ """Test find_by_ids with slug column."""
+ # Create multiple dashboards
+ dashboards = []
+ for i in range(3):
+ dashboard = Dashboard(
+ dashboard_title=f"Test Slug Dashboard {i}",
+ slug=f"test-slug-dashboard-{i}",
+ published=True,
+ )
+ dashboards.append(dashboard)
+ db.session.add(dashboard)
+ db.session.commit()
+
+ # Find by multiple slugs
+ slugs = [dashboard.slug for dashboard in dashboards]
+ found = DashboardDAO.find_by_ids(slugs, id_column="slug", skip_base_filter=True)
+ assert len(found) == 3
+ found_slugs = [d.slug for d in found]
+ assert set(found_slugs) == set(slugs)
+
+
+def test_find_by_ids_with_invalid_column(app_context: Session) -> None:
+ """Test find_by_ids returns empty list when column doesn't exist."""
+ # This should return empty list gracefully
+ result = UserDAO.find_by_ids(["not_a_valid_id"], skip_base_filter=True)
+ assert result == []
+
+
+def test_find_by_ids_skip_base_filter(app_context: Session) -> None:
+ """Test find_by_ids with skip_base_filter parameter."""
+ # Create users
+ users = []
+ for i in range(3):
+ user = User(
+ username=f"test_skip_filter_{i}",
+ first_name=f"Test{i}",
+ last_name="User",
+ email=f"test{i}@example.com",
+ active=True,
+ )
+ users.append(user)
+ db.session.add(user)
+ db.session.commit()
+
+ ids = [user.id for user in users]
+
+ # Without skipping base filter
+ found_no_skip = UserDAO.find_by_ids(ids, skip_base_filter=False)
+ assert len(found_no_skip) == 3
+
+ # With skipping base filter
+ found_skip = UserDAO.find_by_ids(ids, skip_base_filter=True)
+ assert len(found_skip) == 3
+
+
+def test_base_dao_create_with_item(app_context: Session) -> None:
+ """Test BaseDAO.create with an item parameter."""
+ # Create a user item
+ user = User(
+ username="created_with_item",
+ first_name="Created",
+ last_name="Item",
+ email="created@example.com",
+ active=True,
+ )
+
+ # Create using the item
+ created = UserDAO.create(item=user)
+ assert created is not None
+ assert created.username == "created_with_item"
+ assert created.first_name == "Created"
+
+ # Verify it's in the session
+ assert created in db.session
+
+ # Commit and verify it persists
+ db.session.commit()
+
+ # Find it again to ensure it was saved
+ found = UserDAO.find_by_id(created.id, skip_base_filter=True)
+ assert found is not None
+ assert found.username == "created_with_item"
+
+
+def test_base_dao_create_with_attributes(app_context: Session) -> None:
+ """Test BaseDAO.create with attributes parameter."""
+ # Create using attributes dict
+ attributes = {
+ "username": "created_with_attrs",
+ "first_name": "Created",
+ "last_name": "Attrs",
+ "email": "attrs@example.com",
+ "active": True,
+ }
+
+ created = UserDAO.create(attributes=attributes)
+ assert created is not None
+ assert created.username == "created_with_attrs"
+ assert created.email == "attrs@example.com"
+
+ # Commit and verify
+ db.session.commit()
+ found = UserDAO.find_by_id(created.id, skip_base_filter=True)
+ assert found is not None
+ assert found.username == "created_with_attrs"
+
+
+def test_base_dao_create_with_both_item_and_attributes(app_context: Session) -> None:
+ """Test BaseDAO.create with both item and attributes (override behavior)."""
+ # Create a user item
+ user = User(
+ username="item_username",
+ first_name="Item",
+ last_name="User",
+ email="item@example.com",
+ active=False,
+ )
+
+ # Override some attributes
+ attributes = {
+ "username": "override_username",
+ "active": True,
+ }
+
+ created = UserDAO.create(item=user, attributes=attributes)
+ assert created is not None
+ assert created.username == "override_username" # Should be overridden
+ assert created.active is True # Should be overridden
+ assert created.first_name == "Item" # Should keep original
+ assert created.last_name == "User" # Should keep original
+
+ db.session.commit()
+
+
+def test_base_dao_update_with_item(app_context: Session) -> None:
+ """Test BaseDAO.update with an item parameter."""
+ # Create a user first
+ user = User(
+ username="update_test",
+ first_name="Original",
+ last_name="User",
+ email="original@example.com",
+ active=True,
+ )
+ db.session.add(user)
+ db.session.commit()
+
+ # Update the user
+ user.first_name = "Updated"
+ updated = UserDAO.update(item=user)
+ assert updated is not None
+ assert updated.first_name == "Updated"
+
+ db.session.commit()
+
+ # Verify the update persisted
+ found = UserDAO.find_by_id(user.id, skip_base_filter=True)
+ assert found is not None
+ assert found.first_name == "Updated"
+
+
+def test_base_dao_update_with_attributes(app_context: Session) -> None:
+ """Test BaseDAO.update with attributes parameter."""
+ # Create a user first
+ user = User(
+ username="update_attrs_test",
+ first_name="Original",
+ last_name="User",
+ email="original@example.com",
+ active=True,
+ )
+ db.session.add(user)
+ db.session.commit()
+
+ # Update using attributes
+ attributes = {"first_name": "Updated", "last_name": "Attr User"}
+ updated = UserDAO.update(item=user, attributes=attributes)
+ assert updated is not None
+ assert updated.first_name == "Updated"
+ assert updated.last_name == "Attr User"
+
+ db.session.commit()
+
+
+def test_base_dao_update_detached_item(app_context: Session) -> None:
+ """Test BaseDAO.update with a detached item."""
+ # Create a user first
+ user = User(
+ username="detached_test",
+ first_name="Original",
+ last_name="User",
+ email="detached@example.com",
+ active=True,
+ )
+ db.session.add(user)
+ db.session.commit()
+
+ user_id = user.id
+
+ # Expunge to detach from session
+ db.session.expunge(user)
+
+ # Update the detached user
+ user.first_name = "Updated Detached"
+ updated = UserDAO.update(item=user)
+ assert updated is not None
+ assert updated.first_name == "Updated Detached"
+
+ db.session.commit()
+
+ # Verify the update persisted
+ found = UserDAO.find_by_id(user_id, skip_base_filter=True)
+ assert found is not None
+ assert found.first_name == "Updated Detached"
+
+
+def test_base_dao_delete_single_item(app_context: Session) -> None:
+ """Test BaseDAO.delete with a single item."""
+ # Create a dashboard instead of user to avoid circular dependencies
+ dashboard = Dashboard(
+ dashboard_title="Delete Test",
+ slug="delete-test",
+ published=True,
+ )
+ db.session.add(dashboard)
+ db.session.commit()
+
+ dashboard_id = dashboard.id
+
+ DashboardDAO.delete([dashboard])
+ db.session.commit()
+
+ # Verify it's gone
+ found = DashboardDAO.find_by_id(dashboard_id, skip_base_filter=True)
+ assert found is None
+
+
+def test_base_dao_delete_multiple_items(app_context: Session) -> None:
+ """Test BaseDAO.delete with multiple items."""
+ # Create multiple dashboards instead of users to avoid circular dependencies
+ dashboards = []
+ for i in range(3):
+ dashboard = Dashboard(
+ dashboard_title=f"Delete Multi {i}",
+ slug=f"delete-multi-{i}",
+ published=True,
+ )
+ dashboards.append(dashboard)
+ db.session.add(dashboard)
+ db.session.commit()
+
+ dashboard_ids = [dashboard.id for dashboard in dashboards]
+
+ DashboardDAO.delete(dashboards)
+ db.session.commit()
+
+ # Verify they're all gone
+ for dashboard_id in dashboard_ids:
+ found = DashboardDAO.find_by_id(dashboard_id, skip_base_filter=True)
+ assert found is None
Review Comment:
<div>
<div id="suggestion">
<div id="issue"><b>Assertion outside verification loop</b></div>
<div id="fix">
The assertion `assert found is None` is incorrectly placed outside the verification loop in `test_base_dao_delete_multiple_items`. This only checks the last dashboard deletion and would allow the test to pass even if earlier dashboards failed to be deleted. Move the assertion inside the loop to ensure each dashboard deletion is properly verified.
</div>
<details>
<summary>
<b>Code suggestion</b>
</summary>
<blockquote>Check the AI-generated fix before applying</blockquote>
<div id="code">
```suggestion
for dashboard_id in dashboard_ids:
found = DashboardDAO.find_by_id(dashboard_id, skip_base_filter=True)
assert found is None
```
</div>
</details>
</div>
<small><i>Code Review Run <a href=https://github.com/apache/superset/pull/35018#issuecomment-3267072028>#3718e8</a></i></small>
</div>
---
Should Bito avoid suggestions like this for future reviews? (<a href=https://alpha.bito.ai/home/ai-agents/review-rules>Manage Rules</a>)
- [ ] Yes, avoid them
##########
tests/integration_tests/dao/base_dao_test.py:
##########
@@ -0,0 +1,1397 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""
+Integration tests for BaseDAO functionality.
+
+This module contains comprehensive integration tests for the BaseDAO class and its
+subclasses, covering database operations, CRUD methods, flexible column support,
+column operators, and error handling.
+
+Tests use an in-memory SQLite database for isolation and to replicate the unit test
+environment behavior. User model deletions are avoided due to circular dependency
+constraints with self-referential foreign keys.
+"""
+
+import datetime
+import time
+import uuid
+
+import pytest
+from flask_appbuilder.models.filters import BaseFilter
+from flask_appbuilder.security.sqla.models import User
+from sqlalchemy import Column, DateTime, Integer, String
+from sqlalchemy.ext.declarative import declarative_base
+from sqlalchemy.orm.session import Session
+
+from superset.daos.base import BaseDAO, ColumnOperator, ColumnOperatorEnum
+from superset.daos.chart import ChartDAO
+from superset.daos.dashboard import DashboardDAO
+from superset.daos.database import DatabaseDAO
+from superset.daos.user import UserDAO
+from superset.extensions import db
+from superset.models.core import Database
+from superset.models.dashboard import Dashboard
+from superset.models.slice import Slice
+
+# Create a test model for comprehensive testing
+Base = declarative_base()
+
+
+class ExampleModel(Base): # type: ignore
+ __tablename__ = "example_model"
+ id = Column(Integer, primary_key=True)
+ uuid = Column(String(36), unique=True, nullable=False)
+ slug = Column(String(100), unique=True)
+ name = Column(String(100))
+ code = Column(String(50), unique=True)
+ created_on = Column(DateTime, default=datetime.datetime.utcnow)
+
+
+class ExampleModelDAO(BaseDAO[ExampleModel]):
+ model_cls = ExampleModel
+ id_column_name = "id"
+ base_filter = None
+
+
+class MockModel:
+ def __init__(self, id=1, name="test"):
+ self.id = id
+ self.name = name
+
+
+class TestDAO(BaseDAO[MockModel]):
+ model_cls = MockModel
+
+
+@pytest.fixture(autouse=True)
+def mock_g_user(app_context):
+ """Mock the flask g.user for security context."""
+ # Within app context, we can safely mock g
+ from flask import g
+
+ mock_user = User()
+ mock_user.id = 1
+ mock_user.username = "test_user"
+
+ # Set g.user directly instead of patching
+ g.user = mock_user
+ yield
+
+ # Clean up
+ if hasattr(g, "user"):
+ delattr(g, "user")
+
+
+# =============================================================================
+# Integration Tests - These tests use the actual database
+# =============================================================================
+
+
+def test_column_operator_enum_complete_coverage(user_with_data: Session) -> None:
+ """
+ Test that every single ColumnOperatorEnum operator is covered by tests.
+ This ensures we have comprehensive test coverage for all operators.
+ """
+ # Simply verify that we can create queries with all operators
+ for operator in ColumnOperatorEnum:
+ column_operator = ColumnOperator(
+ col="username", opr=operator, value="test_value"
+ )
+ # Just check it doesn't raise an error
+ assert column_operator.opr == operator
+
+
+def test_find_by_id_with_default_column(app_context: Session) -> None:
+ """Test find_by_id with default 'id' column."""
+ # Create a user to test with
+ user = User(
+ username="test_find_by_id",
+ first_name="Test",
+ last_name="User",
+ email="test@example.com",
+ active=True,
+ )
+ db.session.add(user)
+ db.session.commit()
+
+ # Find by numeric id
+ found = UserDAO.find_by_id(user.id, skip_base_filter=True)
+ assert found is not None
+ assert found.id == user.id
+ assert found.username == "test_find_by_id"
+
+ # Test with non-existent id
+ not_found = UserDAO.find_by_id(999999, skip_base_filter=True)
+ assert not_found is None
+
+
+def test_find_by_id_with_uuid_column(app_context: Session) -> None:
+ """Test find_by_id with custom uuid column."""
+ # Create a dashboard with uuid
+ dashboard = Dashboard(
+ dashboard_title="Test UUID Dashboard",
+ slug="test-uuid-dashboard",
+ published=True,
+ )
+ db.session.add(dashboard)
+ db.session.commit()
+
+ # Find by uuid string using the uuid column
+ found = DashboardDAO.find_by_id(
+ str(dashboard.uuid), id_column="uuid", skip_base_filter=True
+ )
+ assert found is not None
+ assert found.uuid == dashboard.uuid
+ assert found.dashboard_title == "Test UUID Dashboard"
+
+ # Find by numeric id (should still work)
+ found_by_id = DashboardDAO.find_by_id(dashboard.id, skip_base_filter=True)
+ assert found_by_id is not None
+ assert found_by_id.id == dashboard.id
+
+ # Test with non-existent uuid
+ not_found = DashboardDAO.find_by_id(str(uuid.uuid4()), skip_base_filter=True)
+ assert not_found is None
+
+
+def test_find_by_id_with_slug_column(app_context: Session) -> None:
+ """Test find_by_id with slug column fallback."""
+ # Create a dashboard with slug
+ dashboard = Dashboard(
+ dashboard_title="Test Slug Dashboard",
+ slug="test-slug-dashboard",
+ published=True,
+ )
+ db.session.add(dashboard)
+ db.session.commit()
+
+ # Find by slug using the slug column
+ found = DashboardDAO.find_by_id(
+ "test-slug-dashboard", id_column="slug", skip_base_filter=True
+ )
+ assert found is not None
+ assert found.slug == "test-slug-dashboard"
+ assert found.dashboard_title == "Test Slug Dashboard"
+
+ # Test with non-existent slug
+ not_found = DashboardDAO.find_by_id("non-existent-slug", skip_base_filter=True)
+ assert not_found is None
+
+
+def test_find_by_id_with_invalid_column(app_context: Session) -> None:
+ """Test find_by_id returns None when column doesn't exist."""
+ # This should return None gracefully
+ result = UserDAO.find_by_id("not_a_valid_id", skip_base_filter=True)
+ assert result is None
+
+
+def test_find_by_id_skip_base_filter(app_context: Session) -> None:
+ """Test find_by_id with skip_base_filter parameter."""
+ # Create users with different active states
+ active_user = User(
+ username="active_user",
+ first_name="Active",
+ last_name="User",
+ email="active@example.com",
+ active=True,
+ )
+ inactive_user = User(
+ username="inactive_user",
+ first_name="Inactive",
+ last_name="User",
+ email="inactive@example.com",
+ active=False,
+ )
+ db.session.add_all([active_user, inactive_user])
+ db.session.commit()
+
+ # Without skipping base filter (if one exists)
+ found_active = UserDAO.find_by_id(active_user.id, skip_base_filter=False)
+ assert found_active is not None
+
+ # With skipping base filter
+ found_active_skip = UserDAO.find_by_id(active_user.id, skip_base_filter=True)
+ assert found_active_skip is not None
+
+ # Both should find the user since UserDAO might not have a base filter
+ assert found_active.id == active_user.id
+ assert found_active_skip.id == active_user.id
+
+
+def test_find_by_ids_with_default_column(app_context: Session) -> None:
+ """Test find_by_ids with default 'id' column."""
+ # Create multiple users
+ users = []
+ for i in range(3):
+ user = User(
+ username=f"test_find_by_ids_{i}",
+ first_name=f"Test{i}",
+ last_name="User",
+ email=f"test{i}@example.com",
+ active=True,
+ )
+ users.append(user)
+ db.session.add(user)
+ db.session.commit()
+
+ # Find by multiple ids
+ ids = [user.id for user in users]
+ found = UserDAO.find_by_ids(ids, skip_base_filter=True)
+ assert len(found) == 3
+ found_ids = [u.id for u in found]
+ assert set(found_ids) == set(ids)
+
+ # Test with mix of existent and non-existent ids
+ mixed_ids = [users[0].id, 999999, users[1].id]
+ found_mixed = UserDAO.find_by_ids(mixed_ids, skip_base_filter=True)
+ assert len(found_mixed) == 2
+
+ # Test with empty list
+ found_empty = UserDAO.find_by_ids([], skip_base_filter=True)
+ assert found_empty == []
+
+
+def test_find_by_ids_with_uuid_column(app_context: Session) -> None:
+ """Test find_by_ids with uuid column."""
+ # Create multiple dashboards
+ dashboards = []
+ for i in range(3):
+ dashboard = Dashboard(
+ dashboard_title=f"Test UUID Dashboard {i}",
+ slug=f"test-uuid-dashboard-{i}",
+ published=True,
+ )
+ dashboards.append(dashboard)
+ db.session.add(dashboard)
+ db.session.commit()
+
+ # Find by multiple uuids
+ uuids = [str(dashboard.uuid) for dashboard in dashboards]
+ found = DashboardDAO.find_by_ids(uuids, id_column="uuid", skip_base_filter=True)
+ assert len(found) == 3
+ found_uuids = [str(d.uuid) for d in found]
+ assert set(found_uuids) == set(uuids)
+
+ # Test with mix of ids and uuids - search separately by column
+ found_by_id = DashboardDAO.find_by_ids([dashboards[0].id], skip_base_filter=True)
+ found_by_uuid = DashboardDAO.find_by_ids(
+ [str(dashboards[1].uuid)], id_column="uuid", skip_base_filter=True
+ )
+ assert len(found_by_id) == 1
+ assert len(found_by_uuid) == 1
+
+
+def test_find_by_ids_with_slug_column(app_context: Session) -> None:
+ """Test find_by_ids with slug column."""
+ # Create multiple dashboards
+ dashboards = []
+ for i in range(3):
+ dashboard = Dashboard(
+ dashboard_title=f"Test Slug Dashboard {i}",
+ slug=f"test-slug-dashboard-{i}",
+ published=True,
+ )
+ dashboards.append(dashboard)
+ db.session.add(dashboard)
+ db.session.commit()
+
+ # Find by multiple slugs
+ slugs = [dashboard.slug for dashboard in dashboards]
+ found = DashboardDAO.find_by_ids(slugs, id_column="slug", skip_base_filter=True)
+ assert len(found) == 3
+ found_slugs = [d.slug for d in found]
+ assert set(found_slugs) == set(slugs)
+
+
+def test_find_by_ids_with_invalid_column(app_context: Session) -> None:
+ """Test find_by_ids returns empty list when column doesn't exist."""
+ # This should return empty list gracefully
+ result = UserDAO.find_by_ids(["not_a_valid_id"], skip_base_filter=True)
+ assert result == []
+
+
+def test_find_by_ids_skip_base_filter(app_context: Session) -> None:
+ """Test find_by_ids with skip_base_filter parameter."""
+ # Create users
Review Comment:
<div>
<div id="suggestion">
<div id="issue"><b>Duplicate code after return</b></div>
<div id="fix">
The `find_by_ids` method in `BaseDAO` has critical duplicate code after the return statement. Lines 331-346 contain unreachable duplicate query logic and exception handling that will never execute, creating dead code and potential confusion. This appears to be leftover from a merge conflict or incomplete refactoring. The duplicate code includes another complete query execution path that duplicates the logic from lines 320-330.
</div>
<details>
<summary>
<b>Code suggestion</b>
</summary>
<blockquote>Check the AI-generated fix before applying</blockquote>
<div id="code">
```
- return results
- query = db.session.query(cls.model_cls).filter(id_col.in_(converted_ids))
- if cls.base_filter and not skip_base_filter:
- data_model = SQLAInterface(cls.model_cls, db.session)
- query = cls.base_filter( # pylint: disable=not-callable
- cls.id_column_name, data_model
- ).apply(query, None)
-
- try:
- results = query.all()
- except SQLAlchemyError as ex:
- model_name = cls.model_cls.__name__ if cls.model_cls else "Unknown"
- raise DAOFindFailedError(
- f"Failed to find {model_name} with ids: {model_ids}"
- ) from ex
-
- return results
+ return results
```
</div>
</details>
</div>
<small><i>Code Review Run <a href=https://github.com/apache/superset/pull/35018#issuecomment-3267072028>#3718e8</a></i></small>
</div>
---
Should Bito avoid suggestions like this for future reviews? (<a href=https://alpha.bito.ai/home/ai-agents/review-rules>Manage Rules</a>)
- [ ] Yes, avoid them
##########
tests/unit_tests/dao/base_dao_test.py:
##########
@@ -37,6 +44,103 @@ class TestDAOWithNoneModel(BaseDAO[MockModel]):
model_cls = None
+# =============================================================================
+# Unit Tests - These tests use mocks and don't touch the database
+# =============================================================================
+
+
+def test_column_operator_enum_apply_method() -> None: # noqa: C901
+ """
+ Test that the apply method works correctly for each operator.
+ This verifies the actual SQL generation for each operator.
+ """
+ Base_test = declarative_base() # noqa: N806
+
+ class TestModel(Base_test): # type: ignore
+ __tablename__ = "test_model"
+ id = Column(Integer, primary_key=True)
+ name = Column(String(50))
+ age = Column(Integer)
+ active = Column(Boolean)
+
+ # Test each operator's apply method
+ test_cases = [
+ # (operator, column, value, expected_sql_fragment)
+ (ColumnOperatorEnum.eq, TestModel.name, "test", "name = 'test'"),
+ (ColumnOperatorEnum.ne, TestModel.name, "test", "name != 'test'"),
+ (ColumnOperatorEnum.sw, TestModel.name, "test", "name LIKE 'test%'"),
+ (ColumnOperatorEnum.ew, TestModel.name, "test", "name LIKE '%test'"),
+ (ColumnOperatorEnum.in_, TestModel.id, [1, 2, 3], "id IN (1, 2, 3)"),
+ (ColumnOperatorEnum.nin, TestModel.id, [1, 2, 3], "id NOT IN (1, 2, 3)"),
+ (ColumnOperatorEnum.gt, TestModel.age, 25, "age > 25"),
+ (ColumnOperatorEnum.gte, TestModel.age, 25, "age >= 25"),
+ (ColumnOperatorEnum.lt, TestModel.age, 25, "age < 25"),
+ (ColumnOperatorEnum.lte, TestModel.age, 25, "age <= 25"),
+ (ColumnOperatorEnum.like, TestModel.name, "test", "name LIKE '%test%'"),
+ (ColumnOperatorEnum.ilike, TestModel.name, "test", "name ILIKE '%test%'"),
+ (ColumnOperatorEnum.is_null, TestModel.name, None, "name IS NULL"),
+ (ColumnOperatorEnum.is_not_null, TestModel.name, None, "name IS NOT NULL"),
+ ]
+
+ for operator, column, value, expected_sql_fragment in test_cases:
+ # Apply the operator
+ result = operator.apply(column, value)
+
+ # Convert to string to check SQL generation
+ sql_str = str(result.compile(compile_kwargs={"literal_binds": True}))
+
+ # Verify the SQL contains the expected fragment
+ # Note: SQLAlchemy might format this differently, so we check for key parts
+ if "=" in expected_sql_fragment:
+ assert "=" in sql_str
+ elif "!=" in expected_sql_fragment:
+ assert "!=" in sql_str
+ elif "LIKE" in expected_sql_fragment:
+ assert "LIKE" in sql_str
+ elif "ILIKE" in expected_sql_fragment:
+ assert "ILIKE" in sql_str
+ elif "IN" in expected_sql_fragment:
+ assert "IN" in sql_str
+ elif "NOT IN" in expected_sql_fragment:
+ assert "NOT IN" in sql_str
+ elif ">" in expected_sql_fragment:
+ assert ">" in sql_str
+ elif ">=" in expected_sql_fragment:
+ assert ">=" in sql_str
+ elif "<" in expected_sql_fragment:
+ assert "<" in sql_str
+ elif "<=" in expected_sql_fragment:
+ assert "<=" in sql_str
+ elif "IS NULL" in expected_sql_fragment:
+ assert "IS NULL" in sql_str
+ elif "IS NOT NULL" in expected_sql_fragment:
+ assert "IS NOT NULL" in sql_str
Review Comment:
<div>
<div id="suggestion">
<div id="issue"><b>Weak SQL verification logic</b></div>
<div id="fix">
The test assertions for SQL generation are too weak and use substring matching instead of exact verification. This allows incorrect SQL generation to pass tests. Replace the complex conditional assertions with direct SQL fragment matching that validates the complete expected SQL structure.
</div>
<details>
<summary>
<b>Code suggestion</b>
</summary>
<blockquote>Check the AI-generated fix before applying</blockquote>
<div id="code">
```suggestion
# Verify the SQL contains the expected fragment
assert expected_sql_fragment in sql_str, f"Expected '{expected_sql_fragment}' in '{sql_str}'"
```
</div>
</details>
</div>
<small><i>Code Review Run <a href=https://github.com/apache/superset/pull/35018#issuecomment-3267072028>#3718e8</a></i></small>
</div>
---
Should Bito avoid suggestions like this for future reviews? (<a href=https://alpha.bito.ai/home/ai-agents/review-rules>Manage Rules</a>)
- [ ] Yes, avoid them
##########
superset/daos/base.py:
##########
@@ -83,39 +196,138 @@ def find_by_id_or_uuid(
return None
@classmethod
- def find_by_id(
- cls,
- model_id: str | int,
- skip_base_filter: bool = False,
- ) -> T | None:
+ def _apply_base_filter(
+ cls, query: Any, skip_base_filter: bool = False, data_model: Any = None
+ ) -> Any:
"""
- Find a model by id, if defined applies `base_filter`
+ Apply the base_filter to the query if it exists and skip_base_filter is False.
"""
- query = db.session.query(cls.model_cls)
if cls.base_filter and not skip_base_filter:
- data_model = SQLAInterface(cls.model_cls, db.session)
+ if data_model is None:
+ data_model = SQLAInterface(cls.model_cls, db.session)
query = cls.base_filter( # pylint: disable=not-callable
cls.id_column_name, data_model
).apply(query, None)
- id_column = getattr(cls.model_cls, cls.id_column_name)
+ return query
+
+ @classmethod
+ def _convert_value_for_column(cls, column: Any, value: Any) -> Any:
+ """
+ Convert a value to the appropriate type for a given SQLAlchemy column.
+
+ Args:
+ column: SQLAlchemy column object
+ value: Value to convert
+
+ Returns:
+ Converted value or None if conversion fails
+ """
+ if (
+ hasattr(column.type, "python_type")
+ and column.type.python_type == uuid_lib.UUID
+ ):
+ if isinstance(value, str):
+ try:
+ return uuid_lib.UUID(value)
+ except (ValueError, AttributeError):
+ return None
+ return value
+
+ @classmethod
+ def _find_by_column(
+ cls,
+ column_name: str,
+ value: str | int,
+ skip_base_filter: bool = False,
+ ) -> T | None:
+ """
+ Private method to find a model by any column value.
+
+ Args:
+ column_name: Name of the column to search by
+ value: Value to search for
+ skip_base_filter: Whether to skip base filtering
+
+ Returns:
+ Model instance or None if not found
+ """
+ query = db.session.query(cls.model_cls)
+ query = cls._apply_base_filter(query, skip_base_filter)
+
+ if not hasattr(cls.model_cls, column_name):
+ return None
+
+ column = getattr(cls.model_cls, column_name)
+ converted_value = cls._convert_value_for_column(column, value)
+ if converted_value is None:
+ return None
+
try:
- return query.filter(id_column == model_id).one_or_none()
+ return query.filter(column == converted_value).one_or_none()
except StatementError:
# can happen if int is passed instead of a string or similar
return None
+ @classmethod
+ def find_by_id(
+ cls,
+ model_id: str | int,
+ skip_base_filter: bool = False,
+ id_column: str | None = None,
+ ) -> T | None:
+ """
+ Find a model by ID using specified or default ID column.
+
+ Args:
+ model_id: ID value to search for
+ skip_base_filter: Whether to skip base filtering
+ id_column: Column name to use (defaults to cls.id_column_name)
+
+ Returns:
+ Model instance or None if not found
+ """
+ column = id_column or cls.id_column_name
+ return cls._find_by_column(column, model_id, skip_base_filter)
+
@classmethod
def find_by_ids(
cls,
- model_ids: list[str] | list[int],
+ model_ids: Sequence[str | int],
skip_base_filter: bool = False,
+ id_column: str | None = None,
) -> list[T]:
"""
Find a List of models by a list of ids, if defined applies `base_filter`
+
+ :param model_ids: List of IDs to find
+ :param skip_base_filter: If true, skip applying the base filter
+ :param id_column: Optional column name to use for ID lookup
+ (defaults to id_column_name)
"""
- id_col = getattr(cls.model_cls, cls.id_column_name, None)
+ column = id_column or cls.id_column_name
+ id_col = getattr(cls.model_cls, column, None)
if id_col is None or not model_ids:
return []
+
+ # Convert IDs to appropriate types based on column type
+ converted_ids: list[str | int | uuid_lib.UUID] = []
+ for id_val in model_ids:
+ converted_value = cls._convert_value_for_column(id_col, id_val)
+ if converted_value is not None:
+ converted_ids.append(converted_value)
+
+ query = db.session.query(cls.model_cls).filter(id_col.in_(converted_ids))
+ query = cls._apply_base_filter(query, skip_base_filter)
+
+ try:
+ results = query.all()
+ except SQLAlchemyError as ex:
+ model_name = cls.model_cls.__name__ if cls.model_cls else "Unknown"
+ raise DAOFindFailedError(
+ f"Failed to find {model_name} with ids: {model_ids}"
+ ) from ex
+
+ return results
Review Comment:
<div>
<div id="suggestion">
<div id="issue"><b>Dead code after return</b></div>
<div id="fix">
The `find_by_ids` method contains unreachable duplicate code after the return statement. Lines 331-346 contain the old implementation that will never execute since the method already returns at line 330. This creates dead code that should be removed to prevent confusion and maintain clean codebase.
</div>
<details>
<summary>
<b>Code suggestion</b>
</summary>
<blockquote>Check the AI-generated fix before applying</blockquote>
<div id="code">
```
- return results
- query = db.session.query(cls.model_cls).filter(id_col.in_(model_ids))
- if cls.base_filter and not skip_base_filter:
- data_model = SQLAInterface(cls.model_cls, db.session)
- query = cls.base_filter( # pylint: disable=not-callable
- cls.id_column_name, data_model
- ).apply(query, None)
-
- try:
- results = query.all()
- except SQLAlchemyError as ex:
- model_name = cls.model_cls.__name__ if cls.model_cls else "Unknown"
- raise DAOFindFailedError(
- f"Failed to find {model_name} with ids: {model_ids}"
- ) from ex
-
- return results
+ return results
```
</div>
</details>
</div>
<small><i>Code Review Run <a href=https://github.com/apache/superset/pull/35018#issuecomment-3267072028>#3718e8</a></i></small>
</div>
---
Should Bito avoid suggestions like this for future reviews? (<a href=https://alpha.bito.ai/home/ai-agents/review-rules>Manage Rules</a>)
- [ ] Yes, avoid them
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] Feat/dao updates [superset]
Posted by "villebro (via GitHub)" <gi...@apache.org>.
villebro commented on PR #35018:
URL: https://github.com/apache/superset/pull/35018#issuecomment-3259379838
This is great work and will also be very useful for list view filters.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat: Add BaseDAO improvements and test reorganization [superset]
Posted by "mistercrunch (via GitHub)" <gi...@apache.org>.
mistercrunch commented on code in PR #35018:
URL: https://github.com/apache/superset/pull/35018#discussion_r2335175579
##########
superset/daos/base.py:
##########
@@ -251,3 +455,205 @@ def filter_by(cls, **filter_by: Any) -> list[T]:
cls.id_column_name, data_model
).apply(query, None)
return query.filter_by(**filter_by).all()
+
+ @classmethod
+ def apply_column_operators(
+ cls, query: Any, column_operators: Optional[List[ColumnOperator]] = None
+ ) -> Any:
+ """
+ Apply column operators (list of ColumnOperator) to the query using
+ ColumnOperatorEnum logic. Raises ValueError if a filter references a
+ non-existent column.
+ """
+ if not column_operators:
+ return query
+ for c in column_operators:
+ if not isinstance(c, ColumnOperator):
+ continue
+ col = c.col
+ opr = c.opr
+ value = c.value
+ if not col or not hasattr(cls.model_cls, col):
+ model_name = cls.model_cls.__name__ if cls.model_cls else "Unknown"
+ logging.error(
+ f"Invalid filter: column '{col}' does not exist on {model_name}"
Review Comment:
oh right, I feel linters used to catch these ... maybe we lost that rule when moving from pylint to ruff?
```
[tool.ruff]
select = ["E", "F", "G"] # add "G" to enable flake8-logging-format rules
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat: Add BaseDAO improvements and test reorganization [superset]
Posted by "aminghadersohi (via GitHub)" <gi...@apache.org>.
aminghadersohi commented on code in PR #35018:
URL: https://github.com/apache/superset/pull/35018#discussion_r2334543195
##########
superset/daos/base.py:
##########
@@ -251,3 +455,205 @@ def filter_by(cls, **filter_by: Any) -> list[T]:
cls.id_column_name, data_model
).apply(query, None)
return query.filter_by(**filter_by).all()
+
+ @classmethod
+ def apply_column_operators(
+ cls, query: Any, column_operators: Optional[List[ColumnOperator]] = None
+ ) -> Any:
+ """
+ Apply column operators (list of ColumnOperator) to the query using
+ ColumnOperatorEnum logic. Raises ValueError if a filter references a
+ non-existent column.
+ """
+ if not column_operators:
+ return query
+ for c in column_operators:
+ if not isinstance(c, ColumnOperator):
+ continue
+ col = c.col
+ opr = c.opr
+ value = c.value
+ if not col or not hasattr(cls.model_cls, col):
+ model_name = cls.model_cls.__name__ if cls.model_cls else "Unknown"
+ logging.error(
+ f"Invalid filter: column '{col}' does not exist on {model_name}"
Review Comment:
good catch. i will fix.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] Feat/dao updates [superset]
Posted by "aminghadersohi (via GitHub)" <gi...@apache.org>.
aminghadersohi commented on PR #35018:
URL: https://github.com/apache/superset/pull/35018#issuecomment-3266806587
> Few small nits but overall solid code pushing the right logic where it belongs, excited to see base classes get strengthened up and more test coverage!
>
> Wondering if this refactor means some code in the derived classes will automatically benefit or whether we could push subclasses to use the new methods(?)
Thanks!
The added code does not affect existing usages of daos - so subclasses don't use this code yet. I think a future refactor would be beneficial.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat: Add BaseDAO improvements and test reorganization [superset]
Posted by "mistercrunch (via GitHub)" <gi...@apache.org>.
mistercrunch commented on code in PR #35018:
URL: https://github.com/apache/superset/pull/35018#discussion_r2335181915
##########
superset/daos/base.py:
##########
@@ -251,3 +455,205 @@ def filter_by(cls, **filter_by: Any) -> list[T]:
cls.id_column_name, data_model
).apply(query, None)
return query.filter_by(**filter_by).all()
+
+ @classmethod
+ def apply_column_operators(
+ cls, query: Any, column_operators: Optional[List[ColumnOperator]] = None
+ ) -> Any:
+ """
+ Apply column operators (list of ColumnOperator) to the query using
+ ColumnOperatorEnum logic. Raises ValueError if a filter references a
+ non-existent column.
+ """
+ if not column_operators:
+ return query
+ for c in column_operators:
+ if not isinstance(c, ColumnOperator):
+ continue
+ col = c.col
+ opr = c.opr
+ value = c.value
+ if not col or not hasattr(cls.model_cls, col):
+ model_name = cls.model_cls.__name__ if cls.model_cls else "Unknown"
+ logging.error(
+ f"Invalid filter: column '{col}' does not exist on {model_name}"
Review Comment:
Oops:
```
Found 190 errors.
[*] 1 fixable with the `--fix` option.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat: Add BaseDAO improvements and test reorganization [superset]
Posted by "aminghadersohi (via GitHub)" <gi...@apache.org>.
aminghadersohi commented on code in PR #35018:
URL: https://github.com/apache/superset/pull/35018#discussion_r2334813922
##########
superset/daos/base.py:
##########
@@ -83,52 +197,145 @@ def find_by_id_or_uuid(
return None
@classmethod
- def find_by_id(
- cls,
- model_id: str | int,
- skip_base_filter: bool = False,
- ) -> T | None:
+ def _apply_base_filter(
+ cls, query: Any, skip_base_filter: bool = False, data_model: Any = None
+ ) -> Any:
"""
- Find a model by id, if defined applies `base_filter`
+ Apply the base_filter to the query if it exists and skip_base_filter is False.
"""
- query = db.session.query(cls.model_cls)
if cls.base_filter and not skip_base_filter:
- data_model = SQLAInterface(cls.model_cls, db.session)
+ if data_model is None:
+ data_model = SQLAInterface(cls.model_cls, db.session)
query = cls.base_filter( # pylint: disable=not-callable
cls.id_column_name, data_model
).apply(query, None)
- id_column = getattr(cls.model_cls, cls.id_column_name)
+ return query
+
+ @classmethod
+ def _convert_value_for_column(cls, column: Any, value: Any) -> Any:
+ """
+ Convert a value to the appropriate type for a given SQLAlchemy column.
+
+ Args:
+ column: SQLAlchemy column object
+ value: Value to convert
+
+ Returns:
+ Converted value or None if conversion fails
+ """
+ if (
+ hasattr(column.type, "python_type")
+ and column.type.python_type == uuid_lib.UUID
+ ):
+ if isinstance(value, str):
+ try:
+ return uuid_lib.UUID(value)
+ except (ValueError, AttributeError):
+ return None
+ return value
+
+ @classmethod
+ def _find_by_column(
+ cls,
+ column_name: str,
+ value: str | int,
+ skip_base_filter: bool = False,
+ ) -> T | None:
+ """
+ Private method to find a model by any column value.
+
+ Args:
+ column_name: Name of the column to search by
+ value: Value to search for
+ skip_base_filter: Whether to skip base filtering
+
+ Returns:
+ Model instance or None if not found
+ """
+ query = db.session.query(cls.model_cls)
+ query = cls._apply_base_filter(query, skip_base_filter)
+
+ if not hasattr(cls.model_cls, column_name):
+ return None
+
+ column = getattr(cls.model_cls, column_name)
+ converted_value = cls._convert_value_for_column(column, value)
+ if converted_value is None:
+ return None
+
try:
- return query.filter(id_column == model_id).one_or_none()
+ return query.filter(column == converted_value).one_or_none()
except StatementError:
# can happen if int is passed instead of a string or similar
return None
+ @classmethod
+ def find_by_id(
+ cls,
+ model_id: str | int,
+ skip_base_filter: bool = False,
+ id_column: str | None = None,
+ ) -> T | None:
+ """
+ Find a model by ID using specified or default ID column.
+
+ Args:
+ model_id: ID value to search for
+ skip_base_filter: Whether to skip base filtering
+ id_column: Column name to use (defaults to cls.id_column_name)
+
+ Returns:
+ Model instance or None if not found
+ """
+ column = id_column or cls.id_column_name
+ return cls._find_by_column(column, model_id, skip_base_filter)
+
@classmethod
def find_by_ids(
cls,
- model_ids: list[str] | list[int],
+ model_ids: Sequence[str | int],
skip_base_filter: bool = False,
+ id_column: str | None = None,
) -> list[T]:
"""
Find a List of models by a list of ids, if defined applies `base_filter`
+
+ :param model_ids: List of IDs to find
+ :param skip_base_filter: If true, skip applying the base filter
+ :param id_column: Optional column name to use for ID lookup
+ (defaults to id_column_name)
"""
- id_col = getattr(cls.model_cls, cls.id_column_name, None)
+ column = id_column or cls.id_column_name
+ id_col = getattr(cls.model_cls, column, None)
if id_col is None or not model_ids:
return []
- query = db.session.query(cls.model_cls).filter(id_col.in_(model_ids))
- if cls.base_filter and not skip_base_filter:
- data_model = SQLAInterface(cls.model_cls, db.session)
- query = cls.base_filter( # pylint: disable=not-callable
- cls.id_column_name, data_model
- ).apply(query, None)
+
+ # Convert IDs to appropriate types based on column type
+ converted_ids: list[str | int | uuid_lib.UUID] = []
+ for id_val in model_ids:
+ converted_value = cls._convert_value_for_column(id_col, id_val)
+ if converted_value is not None:
+ converted_ids.append(converted_value)
+
+ # Validate type consistency for better error handling
+ if len(converted_ids) > 1:
+ types_found = set(map(type, converted_ids))
+ if len(types_found) > 1:
+ logger.warning(
+ "Mixed ID types detected for %s: %s",
+ cls.model_cls.__name__ if cls.model_cls else "Unknown",
+ [t.__name__ for t in types_found],
+ )
Review Comment:
@dpgaspar what do you think of this approach?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] Feat/dao updates [superset]
Posted by "aminghadersohi (via GitHub)" <gi...@apache.org>.
aminghadersohi commented on code in PR #35018:
URL: https://github.com/apache/superset/pull/35018#discussion_r2328828302
##########
tests/unit_tests/dao/base_dao_test.py:
##########
@@ -37,6 +79,1924 @@ class TestDAOWithNoneModel(BaseDAO[MockModel]):
model_cls = None
+@pytest.fixture(autouse=True)
+def mock_g_user():
+ """Mock the flask g.user for security context."""
+ with patch("superset.security.manager.g") as mock_g:
+ mock_user = User()
+ mock_user.id = 1
+ mock_user.username = "test_user"
+ mock_g.user = mock_user
+ yield mock_g
+
+
+# =============================================================================
+# Column Operator Enum Tests
+# =============================================================================
+
+
+def test_column_operator_enum_apply_method() -> None: # noqa: C901
+ """
+ Test that the apply method works correctly for each operator.
+ This verifies the actual SQL generation for each operator.
+ """
+ Base_test = declarative_base() # noqa: N806
+
+ class TestModel(Base_test): # type: ignore
+ __tablename__ = "test_model"
+ id = Column(Integer, primary_key=True)
+ name = Column(String(50))
+ age = Column(Integer)
+ active = Column(Boolean)
+
+ # Test each operator's apply method
+ test_cases = [
+ # (operator, column, value, expected_sql_fragment)
+ (ColumnOperatorEnum.eq, TestModel.name, "test", "name = 'test'"),
+ (ColumnOperatorEnum.ne, TestModel.name, "test", "name != 'test'"),
+ (ColumnOperatorEnum.sw, TestModel.name, "test", "name LIKE 'test%'"),
+ (ColumnOperatorEnum.ew, TestModel.name, "test", "name LIKE '%test'"),
+ (ColumnOperatorEnum.in_, TestModel.id, [1, 2, 3], "id IN (1, 2, 3)"),
+ (ColumnOperatorEnum.nin, TestModel.id, [1, 2, 3], "id NOT IN (1, 2, 3)"),
+ (ColumnOperatorEnum.gt, TestModel.age, 25, "age > 25"),
+ (ColumnOperatorEnum.gte, TestModel.age, 25, "age >= 25"),
+ (ColumnOperatorEnum.lt, TestModel.age, 25, "age < 25"),
+ (ColumnOperatorEnum.lte, TestModel.age, 25, "age <= 25"),
+ (ColumnOperatorEnum.like, TestModel.name, "test", "name LIKE '%test%'"),
+ (ColumnOperatorEnum.ilike, TestModel.name, "test", "name ILIKE '%test%'"),
+ (ColumnOperatorEnum.is_null, TestModel.name, None, "name IS NULL"),
+ (ColumnOperatorEnum.is_not_null, TestModel.name, None, "name IS NOT NULL"),
+ ]
+
+ for operator, column, value, expected_sql_fragment in test_cases:
+ # Apply the operator
+ result = operator.apply(column, value)
+
+ # Convert to string to check SQL generation
+ sql_str = str(result.compile(compile_kwargs={"literal_binds": True}))
+
+ # Verify the SQL contains the expected fragment
+ # Note: SQLAlchemy might format this differently, so we check for key parts
+ if "=" in expected_sql_fragment:
+ assert "=" in sql_str
+ elif "!=" in expected_sql_fragment:
+ assert "!=" in sql_str
+ elif "LIKE" in expected_sql_fragment:
+ assert "LIKE" in sql_str
+ elif "ILIKE" in expected_sql_fragment:
+ assert "ILIKE" in sql_str
+ elif "IN" in expected_sql_fragment:
+ assert "IN" in sql_str
+ elif "NOT IN" in expected_sql_fragment:
+ assert "NOT IN" in sql_str
+ elif ">" in expected_sql_fragment:
+ assert ">" in sql_str
+ elif ">=" in expected_sql_fragment:
+ assert ">=" in sql_str
+ elif "<" in expected_sql_fragment:
+ assert "<" in sql_str
+ elif "<=" in expected_sql_fragment:
+ assert "<=" in sql_str
+ elif "IS NULL" in expected_sql_fragment:
+ assert "IS NULL" in sql_str
+ elif "IS NOT NULL" in expected_sql_fragment:
+ assert "IS NOT NULL" in sql_str
+
+ # Test that all operators are covered
+ all_operators = set(ColumnOperatorEnum)
+ tested_operators = {
+ ColumnOperatorEnum.eq,
+ ColumnOperatorEnum.ne,
+ ColumnOperatorEnum.sw,
+ ColumnOperatorEnum.ew,
+ ColumnOperatorEnum.in_,
+ ColumnOperatorEnum.nin,
+ ColumnOperatorEnum.gt,
+ ColumnOperatorEnum.gte,
+ ColumnOperatorEnum.lt,
+ ColumnOperatorEnum.lte,
+ ColumnOperatorEnum.like,
+ ColumnOperatorEnum.ilike,
+ ColumnOperatorEnum.is_null,
+ ColumnOperatorEnum.is_not_null,
+ }
+
+ # Ensure we've tested all operators
+ assert tested_operators == all_operators, (
+ f"Missing operators: {all_operators - tested_operators}"
+ )
+
+
+def test_column_operator_enum_complete_coverage(user_with_data: Session) -> None:
+ """
+ Test that every single ColumnOperatorEnum operator is covered by tests.
+ This ensures we have comprehensive test coverage for all operators.
+ """
+ # Create test users for comprehensive operator testing
+ test_users = [
+ User(
+ id=900,
+ username="testuser1",
+ first_name="Test",
+ last_name="User1",
+ email="test1@example.com",
+ active=True,
+ ),
+ User(
+ id=901,
+ username="testuser2",
+ first_name="Test",
+ last_name="User2",
+ email="test2@example.com",
+ active=False,
+ ),
+ User(
+ id=902,
+ username="adminuser",
+ first_name="Admin",
+ last_name="User",
+ email="admin@example.com",
+ active=True,
+ ),
+ ]
+
+ for user in test_users:
+ user_with_data.add(user)
+ user_with_data.commit()
+
+ # Test eq (equals)
+ results, _ = UserDAO.list(
+ column_operators=[
+ ColumnOperator(col="username", opr=ColumnOperatorEnum.eq, value="testuser1")
+ ]
+ )
+ assert any(u.username == "testuser1" for u in results)
+
+ # Test ne (not equals)
+ results, _ = UserDAO.list(
+ column_operators=[
+ ColumnOperator(col="username", opr=ColumnOperatorEnum.ne, value="testuser1")
+ ]
+ )
+ assert all(u.username != "testuser1" for u in results)
+
+ # Test sw (starts with)
+ results, _ = UserDAO.list(
+ column_operators=[
+ ColumnOperator(col="username", opr=ColumnOperatorEnum.sw, value="test")
+ ]
+ )
+ assert any(u.username.startswith("test") for u in results)
+
+ # Test ew (ends with)
+ results, _ = UserDAO.list(
+ column_operators=[
+ ColumnOperator(col="username", opr=ColumnOperatorEnum.ew, value="user1")
+ ]
+ )
+ assert any(u.username.endswith("user1") for u in results)
+
+ # Test in_ (in list)
+ user_ids = [900, 901]
+ results, _ = UserDAO.list(
+ column_operators=[
+ ColumnOperator(col="id", opr=ColumnOperatorEnum.in_, value=user_ids)
+ ]
+ )
+ result_ids = [u.id for u in results]
+ assert all(uid in result_ids for uid in user_ids)
+
+ # Test nin (not in list)
+ results, _ = UserDAO.list(
+ column_operators=[
+ ColumnOperator(col="id", opr=ColumnOperatorEnum.nin, value=user_ids)
+ ]
+ )
+ result_ids = [u.id for u in results]
+ assert all(uid not in result_ids for uid in user_ids)
+
+ # Test gt (greater than)
+ results, _ = UserDAO.list(
+ column_operators=[
+ ColumnOperator(col="id", opr=ColumnOperatorEnum.gt, value=900)
+ ]
+ )
+ assert all(u.id > 900 for u in results)
+
+ # Test gte (greater than or equal)
+ results, _ = UserDAO.list(
+ column_operators=[
+ ColumnOperator(col="id", opr=ColumnOperatorEnum.gte, value=901)
+ ]
+ )
+ assert all(u.id >= 901 for u in results)
+
+ # Test lt (less than)
+ results, _ = UserDAO.list(
+ column_operators=[
+ ColumnOperator(col="id", opr=ColumnOperatorEnum.lt, value=902)
+ ]
+ )
+ assert all(u.id < 902 for u in results)
+
+ # Test lte (less than or equal)
+ results, _ = UserDAO.list(
+ column_operators=[
+ ColumnOperator(col="id", opr=ColumnOperatorEnum.lte, value=901)
+ ]
+ )
+ assert all(u.id <= 901 for u in results)
+
+ # Test like (like pattern)
+ results, _ = UserDAO.list(
+ column_operators=[
+ ColumnOperator(col="username", opr=ColumnOperatorEnum.like, value="test")
+ ]
+ )
+ assert any("test" in u.username for u in results)
+
+ # Test ilike (case insensitive like)
+ results, _ = UserDAO.list(
+ column_operators=[
+ ColumnOperator(col="username", opr=ColumnOperatorEnum.ilike, value="TEST")
+ ]
+ )
+ assert any("test" in u.username.lower() for u in results)
+
+ # Test is_null (is null)
+ # Create a user with a null field that allows nulls
+ user_with_data.add(
+ User(
+ id=903,
+ username="nulluser",
+ first_name="Test",
+ last_name="User",
+ email="nulluser@example.com",
+ active=True,
+ )
+ )
+ user_with_data.commit()
+
+ # Update to set a field that can be null (we'll use a field that allows nulls)
+ # For this test, we'll check if we can find users where a field is null
+ # Since we can't easily set first_name to null, we'll test the is_null operator
+ # by checking if it works with fields that might be null
+ results, _ = UserDAO.list(
+ column_operators=[
+ ColumnOperator(col="first_name", opr=ColumnOperatorEnum.is_null, value=None)
+ ]
+ )
+ # This should return empty since all users have first_name set
+ assert len(results) == 0
+
+ # Test is_not_null (is not null)
+ results, _ = UserDAO.list(
+ column_operators=[
+ ColumnOperator(
+ col="first_name", opr=ColumnOperatorEnum.is_not_null, value=None
+ )
+ ]
+ )
+ # This should return all users since all have first_name set
+ assert len(results) > 0
+
+ # Test boolean operators
+ results, _ = UserDAO.list(
+ column_operators=[
+ ColumnOperator(col="active", opr=ColumnOperatorEnum.eq, value=True)
+ ]
+ )
+ assert all(u.active for u in results)
+
+ results, _ = UserDAO.list(
+ column_operators=[
+ ColumnOperator(col="active", opr=ColumnOperatorEnum.eq, value=False)
+ ]
+ )
+ assert all(not u.active for u in results)
+
+ # Test that all operators are covered
+ all_operators = set(ColumnOperatorEnum)
+ tested_operators = {
+ ColumnOperatorEnum.eq,
+ ColumnOperatorEnum.ne,
+ ColumnOperatorEnum.sw,
+ ColumnOperatorEnum.ew,
+ ColumnOperatorEnum.in_,
+ ColumnOperatorEnum.nin,
+ ColumnOperatorEnum.gt,
+ ColumnOperatorEnum.gte,
+ ColumnOperatorEnum.lt,
+ ColumnOperatorEnum.lte,
+ ColumnOperatorEnum.like,
+ ColumnOperatorEnum.ilike,
+ ColumnOperatorEnum.is_null,
+ ColumnOperatorEnum.is_not_null,
+ }
+
+ # Ensure we've tested all operators
+ assert tested_operators == all_operators, (
+ f"Missing operators: {all_operators - tested_operators}"
+ )
+
+
+# =============================================================================
+# Find by ID Tests - Testing flexible column support
+# =============================================================================
+
+
+def test_find_by_id_with_default_column(app_context: None) -> None:
+ """Test find_by_id using the default id column."""
+ # Create test data
+ user = User(
+ id=1000,
+ username="test_find_user",
+ first_name="Test",
+ last_name="User",
+ email="testfind@example.com",
+ active=True,
+ )
+ db.session.add(user)
+ db.session.commit()
+
+ # Find by ID using default column
+ found_user = UserDAO.find_by_id(1000)
+ assert found_user is not None
+ assert found_user.id == 1000
+ assert found_user.username == "test_find_user"
+
+ # Test not found
+ not_found = UserDAO.find_by_id(9999)
+ assert not_found is None
+
+ # Cleanup
+ db.session.delete(user)
+ db.session.commit()
+
+
+def test_find_by_id_with_uuid_column(app_context: None) -> None:
+ """Test find_by_id using UUID column."""
+ # Create dashboard with UUID
+ dashboard = Dashboard(
+ dashboard_title="Test UUID Dashboard",
+ slug="test-uuid-dashboard",
+ published=True,
+ )
+ db.session.add(dashboard)
+ db.session.commit()
+
+ # Verify dashboard exists (skip base filter for test)
+ found_by_id = DashboardDAO.find_by_id(dashboard.id, skip_base_filter=True)
+ assert found_by_id is not None
+
+ # Get the UUID as string for lookup
+ dashboard_uuid_str = str(dashboard.uuid)
+
+ # Find by UUID (skip base filter)
+ found_dashboard = DashboardDAO.find_by_id(
+ dashboard_uuid_str, id_column="uuid", skip_base_filter=True
+ )
+ assert found_dashboard is not None
+ assert str(found_dashboard.uuid) == dashboard_uuid_str
+ assert found_dashboard.dashboard_title == "Test UUID Dashboard"
+
+ # Test not found with invalid UUID
+ not_found = DashboardDAO.find_by_id("invalid-uuid", id_column="uuid")
+ assert not_found is None
+
+ # Cleanup
+ db.session.delete(dashboard)
+ db.session.commit()
+
+
+def test_find_by_id_with_slug_column(app_context: None) -> None:
+ """Test find_by_id using slug column."""
+ # Create test dashboard with slug
+ dashboard = Dashboard(
+ dashboard_title="Test Slug Dashboard",
+ slug="unique-test-slug",
+ published=True,
+ )
+ db.session.add(dashboard)
+ db.session.commit()
+
+ # Find by slug (skip base filter)
+ found_dashboard = DashboardDAO.find_by_id(
+ "unique-test-slug", id_column="slug", skip_base_filter=True
+ )
+ assert found_dashboard is not None
+ assert found_dashboard.slug == "unique-test-slug"
+ assert found_dashboard.dashboard_title == "Test Slug Dashboard"
+
+ # Test not found
+ not_found = DashboardDAO.find_by_id(
+ "non-existent-slug", id_column="slug", skip_base_filter=True
+ )
+ assert not_found is None
+
+ # Cleanup
+ db.session.delete(dashboard)
+ db.session.commit()
+
+
+def test_find_by_id_with_invalid_column(app_context: None) -> None:
+ """Test find_by_id with invalid column name."""
+ # Should return None when column doesn't exist
+ result = UserDAO.find_by_id(1, id_column="non_existent_column")
+ assert result is None
+
+
+def test_find_by_id_skip_base_filter(app_context: None) -> None:
+ """Test find_by_id with skip_base_filter option."""
+ # Create two users - one active, one inactive
+ active_user = User(
+ id=1001,
+ username="active_user",
+ first_name="Active",
+ last_name="User",
+ email="active@example.com",
+ active=True,
+ )
+ inactive_user = User(
+ id=1002,
+ username="inactive_user",
+ first_name="Inactive",
+ last_name="User",
+ email="inactive@example.com",
+ active=False,
+ )
+ db.session.add_all([active_user, inactive_user])
+ db.session.commit()
+
+ # Test basic finding without base filter
+ found_active = UserDAO.find_by_id(1001)
+ assert found_active is not None
+ assert found_active.id == 1001
+
+ found_inactive = UserDAO.find_by_id(1002)
+ assert found_inactive is not None
+ assert found_inactive.id == 1002
+
+ # Test skip_base_filter parameter works (should have same result for UserDAO)
+ found_with_skip = UserDAO.find_by_id(1001, skip_base_filter=True)
+ assert found_with_skip is not None
+ assert found_with_skip.id == 1001
+
+ # Cleanup
+ db.session.delete(active_user)
+ db.session.delete(inactive_user)
+ db.session.commit()
+
+
+# =============================================================================
+# Find by IDs Tests - Testing bulk operations with flexible columns
+# =============================================================================
+
+
+def test_find_by_ids_with_default_column(app_context: None) -> None:
+ """Test find_by_ids using the default id column."""
+ # Create multiple test users
+ users = []
+ for i in range(1100, 1105):
+ user = User(
+ id=i,
+ username=f"bulk_user_{i}",
+ first_name="Bulk",
+ last_name=f"User{i}",
+ email=f"bulk{i}@example.com",
+ active=True,
+ )
+ users.append(user)
+ db.session.add(user)
+ db.session.commit()
+
+ # Find multiple by IDs
+ ids_to_find = [1100, 1102, 1104]
+ found_users = UserDAO.find_by_ids(ids_to_find)
+ assert len(found_users) == 3
+ found_ids = [u.id for u in found_users]
+ assert set(found_ids) == set(ids_to_find)
+
+ # Test with some non-existent IDs
+ mixed_ids = [1101, 1103, 9999, 8888]
+ found_users = UserDAO.find_by_ids(mixed_ids)
+ assert len(found_users) == 2
+ found_ids = [u.id for u in found_users]
+ assert set(found_ids) == {1101, 1103}
+
+ # Test empty list
+ found_users = UserDAO.find_by_ids([])
+ assert found_users == []
+
+ # Cleanup
+ for user in users:
+ db.session.delete(user)
+ db.session.commit()
+
+
+def test_find_by_ids_with_uuid_column(app_context: None) -> None:
+ """Test find_by_ids using UUID column."""
+ # Create multiple charts with UUIDs
+ charts = []
+ chart_uuids = []
+ for i in range(3):
+ chart_uuid = str(uuid.uuid4())
+ chart_uuids.append(chart_uuid)
+ chart = Slice(
+ slice_name=f"Test Chart {i}",
+ viz_type="table",
+ datasource_id=1,
+ datasource_type="table",
+ params="{}",
+ uuid=chart_uuid,
+ )
+ charts.append(chart)
+ db.session.add(chart)
+ db.session.commit()
+
+ # Find multiple by UUIDs (skip base filter)
+ found_charts = ChartDAO.find_by_ids(
+ chart_uuids[:2], id_column="uuid", skip_base_filter=True
+ )
+ assert len(found_charts) == 2
+ found_uuids = [str(c.uuid) for c in found_charts]
+ assert set(found_uuids) == set(chart_uuids[:2])
+
+ # Test with mixed valid and invalid UUIDs (skip base filter)
+ mixed_uuids = [chart_uuids[0], "invalid-uuid", chart_uuids[2]]
+ found_charts = ChartDAO.find_by_ids(
+ mixed_uuids, id_column="uuid", skip_base_filter=True
+ )
+ assert len(found_charts) == 2
+ found_uuids = [str(c.uuid) for c in found_charts]
+ assert set(found_uuids) == {chart_uuids[0], chart_uuids[2]}
+
+ # Cleanup
+ for chart in charts:
+ db.session.delete(chart)
+ db.session.commit()
+
+
+def test_find_by_ids_with_slug_column(app_context: None) -> None:
+ """Test find_by_ids using slug column."""
+ # Create multiple dashboards with slugs
+ dashboards = []
+ slugs = []
+ for i in range(3):
+ slug = f"test-dashboard-{i}"
+ slugs.append(slug)
+ dashboard = Dashboard(
+ dashboard_title=f"Test Dashboard {i}",
+ slug=slug,
+ published=True,
+ )
+ dashboards.append(dashboard)
+ db.session.add(dashboard)
+ db.session.commit()
+
+ # Find multiple by slugs (skip base filter)
+ found_dashboards = DashboardDAO.find_by_ids(
+ slugs[:2], id_column="slug", skip_base_filter=True
+ )
+ assert len(found_dashboards) == 2
+ found_slugs = [d.slug for d in found_dashboards]
+ assert set(found_slugs) == set(slugs[:2])
+
+ # Test with non-existent slugs
+ mixed_slugs = [slugs[0], "non-existent-slug", slugs[2]]
+ found_dashboards = DashboardDAO.find_by_ids(
+ mixed_slugs, id_column="slug", skip_base_filter=True
+ )
+ assert len(found_dashboards) == 2
+ found_slugs = [d.slug for d in found_dashboards]
+ assert set(found_slugs) == {slugs[0], slugs[2]}
+
+ # Cleanup
+ for dashboard in dashboards:
+ db.session.delete(dashboard)
+ db.session.commit()
+
+
+def test_find_by_ids_with_invalid_column(app_context: None) -> None:
+ """Test find_by_ids with invalid column name."""
+ # Should return empty list when column doesn't exist
+ result = UserDAO.find_by_ids([1, 2, 3], id_column="non_existent_column")
+ assert result == []
+
+
+def test_find_by_ids_skip_base_filter(app_context: None) -> None:
+ """Test find_by_ids with skip_base_filter option."""
+ # Create mix of active and inactive users
+ users = []
+ user_ids = []
+ for i, active in enumerate([True, False, True, False]):
+ user_id = 1200 + i
+ user_ids.append(user_id)
+ user = User(
+ id=user_id,
+ username=f"filter_user_{user_id}",
+ first_name="Filter",
+ last_name=f"User{user_id}",
+ email=f"filter{user_id}@example.com",
+ active=active,
+ )
+ users.append(user)
+ db.session.add(user)
+ db.session.commit()
+
+ # Test finding all users
+ found_users = UserDAO.find_by_ids(user_ids)
+ assert len(found_users) == 4
+ found_ids = [u.id for u in found_users]
+ assert set(found_ids) == set(user_ids)
+
+ # Test skip_base_filter parameter works (should have same result for UserDAO)
+ found_users_skip = UserDAO.find_by_ids(user_ids, skip_base_filter=True)
+ assert len(found_users_skip) == 4
+ found_ids_skip = [u.id for u in found_users_skip]
+ assert set(found_ids_skip) == set(user_ids)
+
+ # Cleanup
+ for user in users:
+ db.session.delete(user)
+ db.session.commit()
+
+
+# =============================================================================
+# CRUD Operations Tests - Create, Read, Update, Delete
+# =============================================================================
+
+
+def test_base_dao_create_with_item(app_context: None) -> None:
+ """Test BaseDAO create method with an item."""
+ # Create a new user
+ user = User(
+ id=5000,
+ username="test_create_user",
+ first_name="Test",
+ last_name="Create",
+ email="testcreate@example.com",
+ active=True,
+ )
+ created = UserDAO.create(item=user)
+
+ # Verify it's added to session
+ assert created is user
+ assert created in db.session
+
+ # Commit and verify it persists
+ db.session.commit()
+ assert created.id is not None
+
+ # Verify we can find it
+ found = UserDAO.find_by_id(created.id)
+ assert found is not None
+ assert found.username == "test_create_user"
+ assert found.first_name == "Test"
+
+ # Cleanup
+ db.session.delete(created)
+ db.session.commit()
+
+
+def test_base_dao_create_with_attributes(app_context: None) -> None:
+ """Test BaseDAO create method with attributes dict."""
+ # Create with attributes
+ attributes = {
+ "id": 5001,
+ "username": "attr_user",
+ "first_name": "Attr",
+ "last_name": "User",
+ "email": "attr@example.com",
+ "active": True,
+ }
+ created = UserDAO.create(attributes=attributes)
+
+ # Verify attributes were set
+ assert created.username == "attr_user"
+ assert created.first_name == "Attr"
+ assert created in db.session
+
+ # Commit and verify
+ db.session.commit()
+ assert created.id is not None
+
+ # Cleanup
+ db.session.delete(created)
+ db.session.commit()
+
+
+def test_base_dao_create_with_both_item_and_attributes(app_context: None) -> None:
+ """Test BaseDAO create with both item and attributes."""
+ # Create user with some values
+ user = User(
+ id=5002,
+ username="initial_user",
+ first_name="Initial",
+ last_name="User",
+ email="initial@example.com",
+ active=True,
+ )
+
+ # Override with attributes
+ attributes = {
+ "username": "updated_user",
+ "first_name": "Updated",
+ }
+ created = UserDAO.create(item=user, attributes=attributes)
+
+ # Attributes should override item values
+ assert created.username == "updated_user"
+ assert created.first_name == "Updated"
+ assert created.last_name == "User" # Should keep original
+
+ db.session.commit()
+
+ # Cleanup
+ db.session.delete(created)
+ db.session.commit()
+
+
+def test_base_dao_update_with_item(app_context: None) -> None:
+ """Test BaseDAO update method with an item."""
+ # Create a user first
+ user = User(
+ id=5003,
+ username="original_user",
+ first_name="Original",
+ last_name="User",
+ email="original@example.com",
+ active=True,
+ )
+ db.session.add(user)
+ db.session.commit()
+
+ # Update it
+ user.first_name = "Updated"
+ updated = UserDAO.update(item=user)
+
+ assert updated is user
+ assert updated.first_name == "Updated"
+
+ db.session.commit()
+
+ # Verify update persisted
+ found = UserDAO.find_by_id(user.id)
+ assert found is not None
+ assert found.first_name == "Updated"
+
+ # Cleanup
+ db.session.delete(user)
+ db.session.commit()
+
+
+def test_base_dao_update_with_attributes(app_context: None) -> None:
+ """Test BaseDAO update method with attributes."""
+ # Create a user first
+ user = User(
+ id=5004,
+ username="attr_update_user",
+ first_name="Original",
+ last_name="User",
+ email="attrupdate@example.com",
+ active=True,
+ )
+ db.session.add(user)
+ db.session.commit()
+
+ # Update with attributes
+ attributes = {
+ "first_name": "Attr Updated",
+ "last_name": "Attr User",
+ }
+ updated = UserDAO.update(item=user, attributes=attributes)
+
+ assert updated.first_name == "Attr Updated"
+ assert updated.last_name == "Attr User"
+
+ db.session.commit()
+
+ # Cleanup
+ db.session.delete(user)
+ db.session.commit()
+
+
+def test_base_dao_update_detached_item(app_context: None) -> None:
+ """Test BaseDAO update with a detached item."""
+ # Create and commit a user
+ user = User(
+ id=5005,
+ username="detached_user",
+ first_name="Detached",
+ last_name="User",
+ email="detached@example.com",
+ active=True,
+ )
+ db.session.add(user)
+ db.session.commit()
+ user_id = user.id
+
+ # Detach it from session
+ db.session.expunge(user)
+
+ # Update the detached user
+ user.first_name = "Updated Detached"
+ updated = UserDAO.update(item=user)
+
+ # Should merge it back into session
+ assert updated in db.session
+ assert updated.first_name == "Updated Detached"
+
+ db.session.commit()
+
+ # Verify update
+ found = UserDAO.find_by_id(user_id)
+ assert found is not None
+ assert found.first_name == "Updated Detached"
+
+ # Cleanup
+ db.session.delete(found)
+ db.session.commit()
+
+
+def test_base_dao_delete_single_item(app_context: None) -> None:
+ """Test BaseDAO delete method with single item."""
+ # Create a user
+ user = User(
+ id=5006,
+ username="delete_user",
+ first_name="Delete",
+ last_name="User",
+ email="delete@example.com",
+ active=True,
+ )
+ db.session.add(user)
+ db.session.commit()
+ user_id = user.id
+
+ # Delete it
+ UserDAO.delete([user])
+ db.session.commit()
+
+ # Verify it's gone
+ found = UserDAO.find_by_id(user_id)
+ assert found is None
+
+
+def test_base_dao_delete_multiple_items(app_context: None) -> None:
+ """Test BaseDAO delete method with multiple items."""
+ # Create multiple users
+ users = []
+ for i in range(3):
+ user = User(
+ id=5007 + i,
+ username=f"delete_user_{i}",
+ first_name="Delete",
+ last_name=f"User{i}",
+ email=f"delete{i}@example.com",
+ active=True,
+ )
+ users.append(user)
+ db.session.add(user)
+ db.session.commit()
+
+ user_ids = [user.id for user in users]
+
+ # Delete all at once
+ UserDAO.delete(users)
+ db.session.commit()
+
+ # Verify all are gone
+ for user_id in user_ids:
+ found = UserDAO.find_by_id(user_id)
+ assert found is None
+
+
+def test_base_dao_delete_empty_list(app_context: None) -> None:
+ """Test BaseDAO delete with empty list."""
+ # Should not raise any errors
+ UserDAO.delete([])
+ db.session.commit()
+
+
+def test_base_dao_find_all(app_context: None) -> None:
+ """Test BaseDAO find_all method."""
+ # Create test users
+ users = []
+ for i in range(3):
+ user = User(
+ id=5010 + i,
+ username=f"find_all_user_{i}",
+ first_name="FindAll",
+ last_name=f"User{i}",
+ email=f"findall{i}@example.com",
+ active=True,
+ )
+ users.append(user)
+ db.session.add(user)
+ db.session.commit()
+
+ # Find all
+ all_users = UserDAO.find_all()
+
+ # Should include our created users
+ usernames = [user.username for user in all_users]
+ for i in range(3):
+ assert f"find_all_user_{i}" in usernames
+
+ # Cleanup
+ for user in users:
+ db.session.delete(user)
+ db.session.commit()
+
+
+def test_base_dao_find_one_or_none(app_context: None) -> None:
+ """Test BaseDAO find_one_or_none method."""
+ # Create test users
+ user1 = User(
+ id=5013,
+ username="unique_user",
+ first_name="Unique",
+ last_name="User",
+ email="unique@example.com",
+ active=True,
+ )
+ user2 = User(
+ id=5014,
+ username="duplicate_user_1",
+ first_name="Duplicate",
+ last_name="First",
+ email="dup1@example.com",
+ active=True,
+ )
+ user3 = User(
+ id=5015,
+ username="duplicate_user_2",
+ first_name="Duplicate",
+ last_name="Second",
+ email="dup2@example.com",
+ active=True,
+ )
+ db.session.add_all([user1, user2, user3])
+ db.session.commit()
+
+ # Find unique user
+ found = UserDAO.find_one_or_none(username="unique_user")
+ assert found is not None
+ assert found.first_name == "Unique"
+
+ # Find non-existent user
+ not_found = UserDAO.find_one_or_none(username="does_not_exist")
+ assert not_found is None
+
+ # Cleanup
+ db.session.delete(user1)
+ db.session.delete(user2)
+ db.session.delete(user3)
+ db.session.commit()
+
+
+# =============================================================================
+# List and Filtering Tests
+# =============================================================================
+
+
+def test_base_dao_list_returns_results(user_with_data: Session) -> None:
+ results, total = UserDAO.list()
+ assert total >= 1
+ assert any(u.username == "testuser" for u in results)
+
+
+def test_base_dao_list_with_column_operators(user_with_data: Session) -> None:
+ results, total = UserDAO.list(
+ column_operators=[
+ ColumnOperator(col="username", opr=ColumnOperatorEnum.eq, value="testuser")
+ ]
+ )
+ assert total >= 1
+ assert all(u.username == "testuser" for u in results)
+
+
+def test_base_dao_list_with_non_matching_column_operator(
+ user_with_data: Session,
+) -> None:
+ results, total = UserDAO.list(
+ column_operators=[
+ ColumnOperator(
+ col="username", opr=ColumnOperatorEnum.eq, value="doesnotexist"
+ )
+ ]
+ )
+ assert total == 0
+ assert results == []
+
+
+def test_base_dao_count_returns_value(user_with_data: Session) -> None:
+ count = UserDAO.count()
+ assert count >= 1
+
+
+def test_base_dao_count_with_column_operators(user_with_data: Session) -> None:
+ count = UserDAO.count(
+ column_operators=[
+ ColumnOperator(col="username", opr=ColumnOperatorEnum.eq, value="testuser")
+ ]
+ )
+ assert count >= 1
+ count = UserDAO.count(
+ column_operators=[
+ ColumnOperator(
+ col="username", opr=ColumnOperatorEnum.eq, value="doesnotexist"
+ )
+ ]
+ )
+ assert count == 0
+
+
+def test_base_dao_list_and_count_skip_base_filter(user_with_data: Session) -> None:
+ results, total = UserDAO.list()
+ results_skip, total_skip = UserDAO.list()
+ assert total == total_skip
+ count = UserDAO.count()
+ count_skip = UserDAO.count(skip_base_filter=True)
+ assert count_skip == count
+
+
+def test_base_dao_list_ordering(user_with_data: Session) -> None:
+ user_with_data.add_all(
+ [
+ User(
+ id=201,
+ username="buser",
+ first_name="B",
+ last_name="User",
+ email="buser@example.com",
+ active=True,
+ ),
+ User(
+ id=202,
+ username="auser",
+ first_name="A",
+ last_name="User",
+ email="auser@example.com",
+ active=True,
+ ),
+ User(
+ id=203,
+ username="zuser",
+ first_name="Z",
+ last_name="User",
+ email="zuser@example.com",
+ active=True,
+ ),
+ ]
+ )
+ user_with_data.commit()
+ results, _ = UserDAO.list(order_column="username", order_direction="asc")
+ usernames = [u.username for u in results]
+ assert usernames == sorted(usernames)
+ results, _ = UserDAO.list(order_column="id", order_direction="desc")
+ ids = [u.id for u in results]
+ assert ids == sorted(ids, reverse=True)
+
+
+def test_base_dao_list_paging(user_with_data: Session) -> None:
+ users = [
+ User(
+ id=300 + i,
+ username=f"user{i}",
+ first_name=f"F{i}",
+ last_name="User",
+ email=f"user{i}@example.com",
+ active=True,
+ )
+ for i in range(10)
+ ]
+ user_with_data.add_all(users)
+ user_with_data.commit()
+ results, total = UserDAO.list(
+ order_column="id", order_direction="asc", page=0, page_size=5
+ )
+ ids = [u.id for u in results]
+ all_results, _ = UserDAO.list(order_column="id", order_direction="asc")
+ all_ids = [u.id for u in all_results]
+ assert ids == all_ids[:5]
+ assert total == len(all_ids)
+ results, total = UserDAO.list(
+ order_column="id", order_direction="asc", page=1, page_size=5
+ )
+ ids = [u.id for u in results]
+ assert ids == all_ids[5:10]
+ results, total = UserDAO.list(
+ order_column="id", order_direction="asc", page=10, page_size=5
+ )
+ assert results == []
+
+
+def test_base_dao_list_search(user_with_data: Session) -> None:
+ user_with_data.add_all(
+ [
+ User(
+ id=400,
+ username="searchuser1",
+ first_name="Alice",
+ last_name="Wonderland",
+ email="alice@example.com",
+ active=True,
+ ),
+ User(
+ id=401,
+ username="searchuser2",
+ first_name="Bob",
+ last_name="Builder",
+ email="bob@example.com",
+ active=True,
+ ),
+ User(
+ id=402,
+ username="searchuser3",
+ first_name="Charlie",
+ last_name="Chocolate",
+ email="charlie@example.com",
+ active=True,
+ ),
+ ]
+ )
+ user_with_data.commit()
+ results, _ = UserDAO.list(search="Alice", search_columns=["first_name"])
+ assert any(u.first_name == "Alice" for u in results)
+ results, _ = UserDAO.list(search="Builder", search_columns=["last_name"])
+ assert any(u.last_name == "Builder" for u in results)
+ results, _ = UserDAO.list(search="ar", search_columns=["first_name", "last_name"])
+ names = [(u.first_name, u.last_name) for u in results]
+ assert ("Charlie", "Chocolate") in names or ("Bob", "Builder") in names
+
+
+def test_base_dao_list_custom_filter(user_with_data: Session) -> None:
+ datamodel = SQLAInterface(User, user_with_data)
+
+ class EmailDomainFilter(BaseFilter):
+ def apply(self, query, value):
+ return query.filter(User.email.like(f"%@{value}"))
+
+ user_with_data.add_all(
+ [
+ User(
+ id=500,
+ username="customuser1",
+ first_name="Dom",
+ last_name="Ain",
+ email="dom@domain.com",
+ active=True,
+ ),
+ User(
+ id=501,
+ username="customuser2",
+ first_name="Jane",
+ last_name="Doe",
+ email="jane@other.com",
+ active=True,
+ ),
+ ]
+ )
+ user_with_data.commit()
+ results, _ = UserDAO.list(
+ column_operators=None,
+ custom_filters={"email_domain": EmailDomainFilter("email_domain", datamodel)},
+ )
+ assert all(u.email.endswith("@domain.com") for u in results)
+
+
+def test_base_dao_list_base_filter(user_with_data: Session) -> None:
+ class OnlyActiveFilter(BaseFilter):
+ def apply(self, query, value):
+ return query.filter(User.active)
+
+ class ActiveUserDAO(UserDAO):
+ base_filter = OnlyActiveFilter
+
+ user_with_data.add_all(
+ [
+ User(
+ id=600,
+ username="inactiveuser",
+ first_name="Inactive",
+ last_name="User",
+ email="inactive@example.com",
+ active=False,
+ ),
+ User(
+ id=601,
+ username="activeuser",
+ first_name="Active",
+ last_name="User",
+ email="active@example.com",
+ active=True,
+ ),
+ ]
+ )
+ user_with_data.commit()
+ results, _ = ActiveUserDAO.list()
+ assert all(u.active for u in results)
+
+
+def test_base_dao_list_edge_cases(user_with_data: Session) -> None:
+ user_with_data.add_all(
+ [
+ User(
+ id=700,
+ username="booluser1",
+ first_name="Bool",
+ last_name="User",
+ email="bool1@example.com",
+ active=True,
+ ),
+ User(
+ id=701,
+ username="booluser2",
+ first_name="Bool",
+ last_name="User",
+ email="bool2@example.com",
+ active=False,
+ ),
+ ]
+ )
+ user_with_data.commit()
+ results, _ = UserDAO.list(
+ column_operators=[
+ ColumnOperator(col="active", opr=ColumnOperatorEnum.eq, value=True)
+ ]
+ )
+ assert all(u.active for u in results)
+ results, _ = UserDAO.list(
+ column_operators=[
+ ColumnOperator(col="active", opr=ColumnOperatorEnum.eq, value=False)
+ ]
+ )
+ assert all(not u.active for u in results)
+ results, _ = UserDAO.list(
+ column_operators=[
+ ColumnOperator(col="first_name", opr=ColumnOperatorEnum.eq, value=None)
+ ]
+ )
+ ids = [700, 701]
+ results, _ = UserDAO.list(
+ column_operators=[
+ ColumnOperator(col="id", opr=ColumnOperatorEnum.in_, value=ids)
+ ]
+ )
+ result_ids = [u.id for u in results]
+ assert set(result_ids) == set(ids)
+ results, _ = UserDAO.list(page=100, page_size=10)
+ assert results == []
+
+
+def test_base_dao_list_with_select_columns(user_with_data: Session) -> None:
+ # Add a user to ensure at least one exists
+ user_with_data.add(
+ User(
+ id=900,
+ username="coluser",
+ first_name="Col",
+ last_name="User",
+ email="coluser@example.com",
+ active=True,
+ )
+ )
+ user_with_data.commit()
+ # Request only username and email columns
+ results, total = UserDAO.list(columns=["username", "email"])
+ assert total >= 1
+ # Should return Row objects with correct values
+ found = False
+ for row in results:
+ # SQLAlchemy Row supports both index and key access
+ if row[0] == "coluser" and row[1] == "coluser@example.com":
+ found = True
+ assert found
+ # Request only id column
+ results, total = UserDAO.list(columns=["id"])
+ found = False
+ for row in results:
+ if row[0] == 900:
+ found = True
+ assert found
+
+
+def test_base_dao_list_with_default_columns(user_with_data: Session) -> None:
+ user_with_data.add(
+ User(
+ id=901,
+ username="defaultuser",
+ first_name="Default",
+ last_name="User",
+ email="defaultuser@example.com",
+ active=True,
+ )
+ )
+ user_with_data.commit()
+ results, total = UserDAO.list()
+ assert total >= 1
+ # Should return model instances
+ found = False
+ for user in results:
+ if hasattr(user, "id") and hasattr(user, "username") and hasattr(user, "email"):
+ if (
+ user.id == 901
+ and user.username == "defaultuser"
+ and user.email == "defaultuser@example.com"
+ ):
+ found = True
+ assert found
+
+
+# =============================================================================
+# Advanced Features Tests - Filtering, Operators, Performance
+# =============================================================================
+
+
+def test_base_dao_apply_column_operators_invalid_column(app_context: None) -> None:
+ """Test apply_column_operators with invalid column."""
+ # Create a query
+ query = db.session.query(User)
+
+ # Apply operator with invalid column
+ with patch("superset.daos.base.logging.error") as mock_log:
+ column_operators = [
+ ColumnOperator(
+ col="non_existent_column", opr=ColumnOperatorEnum.eq, value="test"
+ )
+ ]
+
+ # Should raise ValueError
+ with pytest.raises(ValueError, match="Invalid filter") as exc_info:
+ UserDAO.apply_column_operators(query, column_operators)
+
+ assert "Invalid filter" in str(exc_info.value)
+ mock_log.assert_called_once()
+
+
+def test_base_dao_apply_column_operators_none_value(app_context: None) -> None:
+ """Test apply_column_operators with None col value."""
+ query = db.session.query(User)
+
+ # Operator with None column name
+ column_operators = [
+ ColumnOperator(
+ col="", # Empty string instead of None
+ opr=ColumnOperatorEnum.eq,
+ value="test",
+ )
+ ]
+
+ # Should raise ValueError
+ with pytest.raises(ValueError, match="Invalid filter") as exc_info:
+ UserDAO.apply_column_operators(query, column_operators)
+
+ assert "Invalid filter" in str(exc_info.value)
+
+
+def test_base_dao_list_with_invalid_operator(app_context: None) -> None:
+ """Test list method with invalid column operator type."""
+ # Pass a non-ColumnOperator object
+ invalid_operators = [{"col": "username", "opr": "eq", "value": "test"}]
+
+ # Should handle gracefully (skip invalid operators)
+ results, total = UserDAO.list(column_operators=invalid_operators) # type: ignore
+
+ # Should return results without applying invalid filter
+ assert isinstance(results, list)
+ assert isinstance(total, int)
+
+
+def test_base_dao_get_filterable_columns(app_context: None) -> None:
+ """Test get_filterable_columns_and_operators method."""
+ # This is typically implemented by FAB, but we can test the interface
+ filterable = UserDAO.get_filterable_columns_and_operators()
+
+ # Should return a dict
+ assert isinstance(filterable, dict)
+
+ # Common user columns should be filterable
+ if filterable: # Only test if FAB integration is available
+ common_columns = ["username", "email", "active", "first_name", "last_name"]
+ for col in common_columns:
+ if col in filterable:
+ # Should have operator information
+ assert isinstance(filterable[col], (list, tuple, set))
+
+
+def test_base_dao_count_with_filters(app_context: None) -> None:
+ """Test count method with various filters."""
+ # Create test users
+ users = []
+ for i in range(5):
+ user = User(
+ id=2000 + i,
+ username=f"count_user_{i}",
+ first_name="Count",
+ last_name=f"User{i}",
+ email=f"count{i}@example.com",
+ active=(i % 2 == 0), # Alternate active/inactive
+ )
+ users.append(user)
+ db.session.add(user)
+ db.session.commit()
+
+ # Count all
+ total_count = UserDAO.count()
+ assert total_count >= 5
+
+ # Count with filter
+ active_count = UserDAO.count(
+ column_operators=[
+ ColumnOperator(col="active", opr=ColumnOperatorEnum.eq, value=True)
+ ]
+ )
+ assert active_count >= 3 # We created 3 active users (0, 2, 4)
+
+ # Count with multiple filters
+ specific_count = UserDAO.count(
+ column_operators=[
+ ColumnOperator(col="first_name", opr=ColumnOperatorEnum.eq, value="Count"),
+ ColumnOperator(col="active", opr=ColumnOperatorEnum.eq, value=False),
+ ]
+ )
+ assert specific_count >= 2 # We created 2 inactive Count users (1, 3)
+
+ # Cleanup
+ for user in users:
+ db.session.delete(user)
+ db.session.commit()
+
+
+# =============================================================================
+# Edge Cases and Special Scenarios
+# =============================================================================
+
+
+def test_find_by_ids_preserves_order(app_context: None) -> None:
+ """Test that find_by_ids returns results correctly."""
+ # Create test users
+ users = []
+ for i in [1300, 1301, 1302, 1303]:
+ user = User(
+ id=i,
+ username=f"order_user_{i}",
+ first_name="Order",
+ last_name=f"User{i}",
+ email=f"order{i}@example.com",
+ active=True,
+ )
+ users.append(user)
+ db.session.add(user)
+ db.session.commit()
+
+ # Request in specific order
+ requested_ids = [1302, 1300, 1303, 1301]
+ found_users = UserDAO.find_by_ids(requested_ids)
+
+ # Note: find_by_ids doesn't guarantee order preservation
+ # but should return all requested items
+ found_ids = [u.id for u in found_users]
+ assert set(found_ids) == set(requested_ids)
+
+ # Cleanup
+ for user in users:
+ db.session.delete(user)
+ db.session.commit()
+
+
+def test_find_by_ids_with_mixed_types(app_context: None) -> None:
+ """Test find_by_ids with integer IDs."""
+ # Create test data
+ database = Database(
+ id=100,
+ database_name="Test DB 100",
+ sqlalchemy_uri="sqlite:///:memory:",
+ )
+ db.session.add(database)
+ db.session.commit()
+
+ # Should handle integer IDs (skip base filter)
+ found_dbs = DatabaseDAO.find_by_ids([100, 100], skip_base_filter=True)
+ # Should find the database
+ assert len(found_dbs) >= 1
+ assert any(d.id == 100 for d in found_dbs)
+
+ # Cleanup
+ db.session.delete(database)
+ db.session.commit()
+
+
+def test_find_by_column_helper_method(app_context: None) -> None:
+ """Test the internal _find_by_column helper method."""
+ # Create test user
+ user = User(
+ id=1400,
+ username="helper_test_user",
+ first_name="Helper",
+ last_name="Test",
+ email="helper@example.com",
+ active=True,
+ )
+ db.session.add(user)
+ db.session.commit()
+
+ # Test the helper method directly
+ found_user = UserDAO._find_by_column("username", "helper_test_user")
+ assert found_user is not None
+ assert found_user.id == 1400
+
+ # Test with non-existent value
+ not_found = UserDAO._find_by_column("username", "non_existent")
+ assert not_found is None
+
+ # Test with invalid column
+ not_found = UserDAO._find_by_column("invalid_column", "value")
+ assert not_found is None
+
+ # Cleanup
+ db.session.delete(user)
+ db.session.commit()
+
+
+def test_find_methods_with_special_characters(app_context: None) -> None:
+ """Test find methods with values containing special characters."""
+ # Create dashboard with special characters in slug
+ special_slug = "test-dashboard-with-special-@#$"
+ dashboard = Dashboard(
+ dashboard_title="Special Dashboard",
+ slug=special_slug,
+ published=True,
+ )
+ db.session.add(dashboard)
+ db.session.commit()
+
+ # Find by slug with special characters (skip base filter)
+ found = DashboardDAO.find_by_id(
+ special_slug, id_column="slug", skip_base_filter=True
+ )
+ assert found is not None
+ assert found.slug == special_slug
+
+ # Test find_by_ids with special characters (skip base filter)
+ found_list = DashboardDAO.find_by_ids(
+ [special_slug], id_column="slug", skip_base_filter=True
+ )
+ assert len(found_list) == 1
+ assert found_list[0].slug == special_slug
+
+ # Cleanup
+ db.session.delete(dashboard)
+ db.session.commit()
+
+
+def test_find_methods_case_sensitivity(app_context: None) -> None:
+ """Test find methods with case-sensitive columns."""
+ # Create users with similar usernames differing in case
+ user1 = User(
+ id=1500,
+ username="CaseSensitive",
+ first_name="Case",
+ last_name="Sensitive",
+ email="case1@example.com",
+ active=True,
+ )
+ user2 = User(
+ id=1501,
+ username="casesensitive",
+ first_name="Case",
+ last_name="Sensitive",
+ email="case2@example.com",
+ active=True,
+ )
+ db.session.add_all([user1, user2])
+
+ try:
+ db.session.commit()
+
+ # Find by exact case
+ found = UserDAO.find_by_id("CaseSensitive", id_column="username")
+ assert found is not None
+ assert found.id == 1500
+
+ found = UserDAO.find_by_id("casesensitive", id_column="username")
+ assert found is not None
+ assert found.id == 1501
+
+ # Cleanup
+ db.session.delete(user1)
+ db.session.delete(user2)
+ db.session.commit()
+ except Exception:
+ # Some databases may have case-insensitive unique constraints
+ db.session.rollback()
+ pytest.skip("Database has case-insensitive unique constraint on username")
+
+
+def test_find_by_ids_empty_and_none_handling(app_context: None) -> None:
+ """Test find_by_ids with empty lists."""
+ # Test with empty list
+ result = UserDAO.find_by_ids([])
+ assert result == []
+
+ # Create test user
+ user = User(
+ id=1600,
+ username="none_test_user",
+ first_name="None",
+ last_name="Test",
+ email="none@example.com",
+ active=True,
+ )
+ db.session.add(user)
+ db.session.commit()
+
+ # Find with valid IDs only
+ found = UserDAO.find_by_ids([1600, 1601])
+ assert len(found) == 1
+ assert found[0].id == 1600
+
+ # Cleanup
+ db.session.delete(user)
+ db.session.commit()
+
+
+def test_find_methods_performance_with_large_lists(app_context: None) -> None:
+ """Test find_by_ids performance with large ID lists."""
+ # Create a batch of users
+ users = []
+ user_ids = []
+ for i in range(1700, 1750): # 50 users
+ user_ids.append(i)
+ user = User(
+ id=i,
+ username=f"perf_user_{i}",
+ first_name="Perf",
+ last_name=f"User{i}",
+ email=f"perf{i}@example.com",
+ active=True,
+ )
+ users.append(user)
+ db.session.add(user)
+ db.session.commit()
+
+ # Find all at once
+ start_time = time.time()
+ found_users = UserDAO.find_by_ids(user_ids)
+ end_time = time.time()
+
+ assert len(found_users) == 50
+ assert (end_time - start_time) < 1.0 # Should complete within 1 second
+
+ # Cleanup
+ for user in users:
+ db.session.delete(user)
+ db.session.commit()
+
+
+# =============================================================================
+# DAO Properties and Configuration Tests
+# =============================================================================
+
+
+def test_base_dao_model_cls_property(app_context: None) -> None:
+ """Test that model_cls property is properly set."""
+ assert UserDAO.model_cls == User
+
+ # Test with a DAO that doesn't set model_cls
+ class InvalidDAO(BaseDAO[User]):
+ pass
+
+ # Should have model_cls attribute set by __init_subclass__
+ # Just verify it exists and is the expected type
+ assert hasattr(InvalidDAO, "model_cls")
+ assert InvalidDAO.model_cls == User # It inherits the generic type
+
+
+def test_base_dao_id_column_name_property(app_context: None) -> None:
+ """Test that id_column_name property is properly used."""
+ assert UserDAO.id_column_name == "id"
+
+ # Test custom id column name
+ class CustomIdDAO(BaseDAO[User]):
+ model_cls = User
+ id_column_name = "username" # Use username as ID column
+
+ # Create test user
+ user = User(
+ id=5016,
+ username="unique-username-id",
+ first_name="Custom",
+ last_name="User",
+ email="custom@example.com",
+ active=True,
+ )
+ db.session.add(user)
+ db.session.commit()
+
+ # Find by custom ID column
+ found = CustomIdDAO.find_by_id("unique-username-id")
+ assert found is not None
+ assert found.first_name == "Custom"
+
+ # Cleanup
+ db.session.delete(user)
+ db.session.commit()
+
+
+def test_base_dao_base_filter_integration(app_context: None) -> None:
+ """Test base_filter integration with list operations."""
+ # Create mix of users
+ users = []
+ for i, active in enumerate([True, False, True, False, True]):
+ user = User(
+ id=2100 + i,
+ username=f"base_filter_user_{i}",
+ first_name="BaseFilter",
+ last_name=f"User{i}",
+ email=f"basefilter{i}@example.com",
+ active=active,
+ )
+ users.append(user)
+ db.session.add(user)
+ db.session.commit()
+
+ # Test basic list functionality
+ results, total = UserDAO.list()
+ assert isinstance(results, list)
+ assert isinstance(total, int)
+ assert total >= 5 # Should include our 5 test users
+
+ # Test skip_base_filter functionality
+ results_skip, total_skip = UserDAO.list()
+ assert isinstance(results_skip, list)
+ assert isinstance(total_skip, int)
+
+ # Test count functionality
+ count_all = UserDAO.count()
+ assert count_all >= 5
+
+ count_skip = UserDAO.count(skip_base_filter=True)
+ assert count_skip >= 5
+
+ # Cleanup
+ for user in users:
+ db.session.delete(user)
+ db.session.commit()
+
+
+def test_base_dao_edge_cases(app_context: None) -> None:
+ """Test BaseDAO edge cases and error conditions."""
+ # Test create without item or attributes
+ created = UserDAO.create()
+ assert created is not None
+ # User model has required fields, so we expect them to be None
+ assert created.username is None
+
+ # Don't commit - would fail due to constraints
+ db.session.rollback()
Review Comment:
yeah i think you're right. some of these are integration tests, and should be in `tests/integration_tests` as such. I will refactor.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat: Add BaseDAO improvements and test reorganization [superset]
Posted by "bito-code-review[bot] (via GitHub)" <gi...@apache.org>.
bito-code-review[bot] commented on code in PR #35018:
URL: https://github.com/apache/superset/pull/35018#discussion_r2353218311
##########
superset/daos/chart.py:
##########
@@ -35,10 +35,24 @@
logger = logging.getLogger(__name__)
+# Custom filterable fields for charts
+CHART_CUSTOM_FIELDS = {
+ "tags": ["eq", "in_", "like"],
+ "viz_type": ["eq", "in_", "like"],
+ "datasource_name": ["eq", "in_", "like"],
Review Comment:
<div>
<div id="suggestion">
<div id="issue"><b>Invalid field in filters</b></div>
<div id="fix">
The 'tags' field cannot be included in CHART_CUSTOM_FIELDS because it's a relationship field (not a direct database column). The get_filterable_columns_and_operators method only handles direct columns and hybrid properties. Including 'tags' will cause runtime errors when the filtering system tries to apply column-based filters on a relationship field. Remove 'tags' from the custom fields and implement proper relationship filtering through ChartFilter if needed.
</div>
<details>
<summary>
<b>Code suggestion</b>
</summary>
<blockquote>Check the AI-generated fix before applying</blockquote>
<div id="code">
```suggestion
# Custom filterable fields for charts
CHART_CUSTOM_FIELDS = {
"viz_type": ["eq", "in_", "like"],
"datasource_name": ["eq", "in_", "like"],
```
</div>
</details>
</div>
<small><i>Code Review Run <a href=https://github.com/apache/superset/pull/35018#issuecomment-3299702064>#b918c4</a></i></small>
</div>
---
Should Bito avoid suggestions like this for future reviews? (<a href=https://alpha.bito.ai/home/ai-agents/review-rules>Manage Rules</a>)
- [ ] Yes, avoid them
##########
superset/daos/dashboard.py:
##########
@@ -45,10 +45,24 @@
logger = logging.getLogger(__name__)
+# Custom filterable fields for dashboards
+DASHBOARD_CUSTOM_FIELDS = {
+ "tags": ["eq", "in_", "like"],
+ "owner": ["eq", "in_"],
+ "published": ["eq"],
+}
Review Comment:
<div>
<div id="suggestion">
<div id="issue"><b>Field name mismatch</b></div>
<div id="fix">
The custom field mapping uses `"owner"` but the Dashboard model defines the relationship as `"owners"` (plural). This mismatch will cause filtering by owner to fail since the filter mechanism won't find a field named "owner". Change to `"owners"` to match the actual relationship name defined in the Dashboard model.
</div>
<details>
<summary>
<b>Code suggestion</b>
</summary>
<blockquote>Check the AI-generated fix before applying</blockquote>
<div id="code">
```suggestion
DASHBOARD_CUSTOM_FIELDS = {
"tags": ["eq", "in_", "like"],
"owners": ["eq", "in_"],
"published": ["eq"],
}
```
</div>
</details>
</div>
<small><i>Code Review Run <a href=https://github.com/apache/superset/pull/35018#issuecomment-3299702064>#b918c4</a></i></small>
</div>
---
Should Bito avoid suggestions like this for future reviews? (<a href=https://alpha.bito.ai/home/ai-agents/review-rules>Manage Rules</a>)
- [ ] Yes, avoid them
##########
superset/daos/base.py:
##########
@@ -83,80 +202,167 @@ def find_by_id_or_uuid(
return None
@classmethod
- def find_by_id(
- cls,
- model_id: str | int,
- skip_base_filter: bool = False,
- ) -> T | None:
+ def _apply_base_filter(
+ cls, query: Any, skip_base_filter: bool = False, data_model: Any = None
+ ) -> Any:
"""
- Find a model by id, if defined applies `base_filter`
+ Apply the base_filter to the query if it exists and skip_base_filter is False.
"""
- query = db.session.query(cls.model_cls)
if cls.base_filter and not skip_base_filter:
- data_model = SQLAInterface(cls.model_cls, db.session)
+ if data_model is None:
+ data_model = SQLAInterface(cls.model_cls, db.session)
query = cls.base_filter( # pylint: disable=not-callable
cls.id_column_name, data_model
).apply(query, None)
- id_column = getattr(cls.model_cls, cls.id_column_name)
+ return query
+
+ @classmethod
+ def _convert_value_for_column(cls, column: Any, value: Any) -> Any:
+ """
+ Convert a value to the appropriate type for a given SQLAlchemy column.
+
+ Args:
+ column: SQLAlchemy column object
+ value: Value to convert
+
+ Returns:
+ Converted value or None if conversion fails
+ """
+ if (
+ hasattr(column.type, "python_type")
+ and column.type.python_type == uuid_lib.UUID
+ ):
+ if isinstance(value, str):
+ try:
+ return uuid_lib.UUID(value)
+ except (ValueError, AttributeError):
+ return None
+ return value
+
+ @classmethod
+ def _find_by_column(
+ cls,
+ column_name: str,
+ value: str | int,
+ skip_base_filter: bool = False,
+ ) -> T | None:
+ """
+ Private method to find a model by any column value.
+
+ Args:
+ column_name: Name of the column to search by
+ value: Value to search for
+ skip_base_filter: Whether to skip base filtering
+
+ Returns:
+ Model instance or None if not found
+ """
+ query = db.session.query(cls.model_cls)
+ query = cls._apply_base_filter(query, skip_base_filter)
+
+ if not hasattr(cls.model_cls, column_name):
+ return None
+
+ column = getattr(cls.model_cls, column_name)
+ converted_value = cls._convert_value_for_column(column, value)
+ if converted_value is None:
+ return None
+
try:
- return query.filter(id_column == model_id).one_or_none()
+ return query.filter(column == converted_value).one_or_none()
except StatementError:
# can happen if int is passed instead of a string or similar
return None
+ @classmethod
+ def find_by_id(
+ cls,
+ model_id: str | int,
+ skip_base_filter: bool = False,
+ id_column: str | None = None,
+ ) -> T | None:
+ """
+ Find a model by ID using specified or default ID column.
+
+ Args:
+ model_id: ID value to search for
+ skip_base_filter: Whether to skip base filtering
+ id_column: Column name to use (defaults to cls.id_column_name)
+
+ Returns:
+ Model instance or None if not found
+ """
+ column = id_column or cls.id_column_name
+ return cls._find_by_column(column, model_id, skip_base_filter)
+
@classmethod
def find_by_ids(
cls,
- model_ids: list[str] | list[int],
+ model_ids: Sequence[str | int],
skip_base_filter: bool = False,
+ id_column: str | None = None,
) -> list[T]:
"""
Find a List of models by a list of ids, if defined applies `base_filter`
+
+ :param model_ids: List of IDs to find
+ :param skip_base_filter: If true, skip applying the base filter
+ :param id_column: Optional column name to use for ID lookup
+ (defaults to id_column_name)
"""
- id_col = getattr(cls.model_cls, cls.id_column_name, None)
+ column = id_column or cls.id_column_name
+ id_col = getattr(cls.model_cls, column, None)
if id_col is None or not model_ids:
return []
- query = db.session.query(cls.model_cls).filter(id_col.in_(model_ids))
- if cls.base_filter and not skip_base_filter:
- data_model = SQLAInterface(cls.model_cls, db.session)
- query = cls.base_filter( # pylint: disable=not-callable
- cls.id_column_name, data_model
- ).apply(query, None)
+
+ # Convert IDs to appropriate types based on column type
+ converted_ids: list[str | int | uuid_lib.UUID] = []
+ for id_val in model_ids:
+ converted_value = cls._convert_value_for_column(id_col, id_val)
+ if converted_value is not None:
+ converted_ids.append(converted_value)
+
+ # Validate type consistency for better error handling
+ if len(converted_ids) > 1:
+ types_found = set(map(type, converted_ids))
+ if len(types_found) > 1:
Review Comment:
<div>
<div id="suggestion">
<div id="issue"><b>ID conversion data loss</b></div>
<div id="fix">
The ID conversion logic in `find_by_ids` silently drops invalid ID conversions by filtering out None values, which can lead to silent data loss. When `_convert_value_for_column` returns None for failed conversions (like invalid UUID strings), these IDs are removed from the query entirely, causing valid-looking requests to return incomplete results. This affects downstream callers like `ChartDAO.find_by_ids`, `DashboardDAO.find_by_ids`, and all DAO classes that inherit from this base.
</div>
<details>
<summary>
<b>Code suggestion</b>
</summary>
<blockquote>Check the AI-generated fix before applying</blockquote>
<div id="code">
```suggestion
# Convert IDs to appropriate types based on column type
converted_ids: list[str | int | uuid_lib.UUID] = []
for id_val in model_ids:
converted_value = cls._convert_value_for_column(id_col, id_val)
if converted_value is None:
# Preserve original value if conversion fails to avoid silent data loss
converted_ids.append(id_val)
else:
converted_ids.append(converted_value)
# Validate type consistency for better error handling
if len(converted_ids) > 1:
types_found = set(map(type, converted_ids))
if len(types_found) > 1:
```
</div>
</details>
</div>
<small><i>Code Review Run <a href=https://github.com/apache/superset/pull/35018#issuecomment-3299702064>#b918c4</a></i></small>
</div>
---
Should Bito avoid suggestions like this for future reviews? (<a href=https://alpha.bito.ai/home/ai-agents/review-rules>Manage Rules</a>)
- [ ] Yes, avoid them
##########
tests/integration_tests/dao/base_dao_test.py:
##########
@@ -0,0 +1,1610 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""
+Integration tests for BaseDAO functionality.
+
+This module contains comprehensive integration tests for the BaseDAO class and its
+subclasses, covering database operations, CRUD methods, flexible column support,
+column operators, and error handling.
+
+Tests use an in-memory SQLite database for isolation and to replicate the unit test
+environment behavior. User model deletions are avoided due to circular dependency
+constraints with self-referential foreign keys.
+"""
+
+import datetime
+import time
+import uuid
+
+import pytest
+from flask_appbuilder.models.filters import BaseFilter
+from flask_appbuilder.security.sqla.models import User
+from sqlalchemy import Column, DateTime, Integer, String
+from sqlalchemy.ext.declarative import declarative_base
+from sqlalchemy.orm.session import Session
+
+from superset.daos.base import BaseDAO, ColumnOperator, ColumnOperatorEnum
+from superset.daos.chart import ChartDAO
+from superset.daos.dashboard import DashboardDAO
+from superset.daos.database import DatabaseDAO
+from superset.daos.user import UserDAO
+from superset.extensions import db
+from superset.models.core import Database
+from superset.models.dashboard import Dashboard
+from superset.models.slice import Slice
+
+# Create a test model for comprehensive testing
+Base = declarative_base()
+
+
+class ExampleModel(Base): # type: ignore
+ __tablename__ = "example_model"
+ id = Column(Integer, primary_key=True)
+ uuid = Column(String(36), unique=True, nullable=False)
+ slug = Column(String(100), unique=True)
+ name = Column(String(100))
+ code = Column(String(50), unique=True)
+ created_on = Column(DateTime, default=datetime.datetime.utcnow)
+
+
+class ExampleModelDAO(BaseDAO[ExampleModel]):
+ model_cls = ExampleModel
+ id_column_name = "id"
+ base_filter = None
+
+
+@pytest.fixture(autouse=True)
+def mock_g_user(app_context):
+ """Mock the flask g.user for security context."""
+ # Within app context, we can safely mock g
+ from flask import g
+
+ mock_user = User()
+ mock_user.id = 1
+ mock_user.username = "test_user"
+
+ # Set g.user directly instead of patching
+ g.user = mock_user
+ yield
+
+ # Clean up
+ if hasattr(g, "user"):
+ delattr(g, "user")
+
+
+# =============================================================================
+# Integration Tests - These tests use the actual database
+# =============================================================================
+
+
+def test_column_operator_enum_complete_coverage(user_with_data: Session) -> None:
+ """
+ Test that every single ColumnOperatorEnum operator is covered by tests.
+ This ensures we have comprehensive test coverage for all operators.
+ """
+ # Simply verify that we can create queries with all operators
+ for operator in ColumnOperatorEnum:
+ column_operator = ColumnOperator(
+ col="username", opr=operator, value="test_value"
+ )
+ # Just check it doesn't raise an error
+ assert column_operator.opr == operator
+
+
+def test_find_by_id_with_default_column(app_context: Session) -> None:
+ """Test find_by_id with default 'id' column."""
+ # Create a user to test with
+ user = User(
+ username="test_find_by_id",
+ first_name="Test",
+ last_name="User",
+ email="test@example.com",
+ active=True,
+ )
+ db.session.add(user)
+ db.session.commit()
+
+ # Find by numeric id
+ found = UserDAO.find_by_id(user.id, skip_base_filter=True)
+ assert found is not None
+ assert found.id == user.id
+ assert found.username == "test_find_by_id"
+
+ # Test with non-existent id
+ not_found = UserDAO.find_by_id(999999, skip_base_filter=True)
+ assert not_found is None
+
+
+def test_find_by_id_with_uuid_column(app_context: Session) -> None:
+ """Test find_by_id with custom uuid column."""
+ # Create a dashboard with uuid
+ dashboard = Dashboard(
+ dashboard_title="Test UUID Dashboard",
+ slug="test-uuid-dashboard",
+ published=True,
+ )
+ db.session.add(dashboard)
+ db.session.commit()
+
+ # Find by uuid string using the uuid column
+ found = DashboardDAO.find_by_id(
+ str(dashboard.uuid), id_column="uuid", skip_base_filter=True
+ )
+ assert found is not None
+ assert found.uuid == dashboard.uuid
+ assert found.dashboard_title == "Test UUID Dashboard"
+
+ # Find by numeric id (should still work)
+ found_by_id = DashboardDAO.find_by_id(dashboard.id, skip_base_filter=True)
+ assert found_by_id is not None
+ assert found_by_id.id == dashboard.id
+
+ # Test with non-existent uuid
+ not_found = DashboardDAO.find_by_id(str(uuid.uuid4()), skip_base_filter=True)
+ assert not_found is None
+
+
+def test_find_by_id_with_slug_column(app_context: Session) -> None:
+ """Test find_by_id with slug column fallback."""
+ # Create a dashboard with slug
+ dashboard = Dashboard(
+ dashboard_title="Test Slug Dashboard",
+ slug="test-slug-dashboard",
+ published=True,
+ )
+ db.session.add(dashboard)
+ db.session.commit()
+
+ # Find by slug using the slug column
+ found = DashboardDAO.find_by_id(
+ "test-slug-dashboard", id_column="slug", skip_base_filter=True
+ )
+ assert found is not None
+ assert found.slug == "test-slug-dashboard"
+ assert found.dashboard_title == "Test Slug Dashboard"
+
+ # Test with non-existent slug
+ not_found = DashboardDAO.find_by_id("non-existent-slug", skip_base_filter=True)
+ assert not_found is None
+
+
+def test_find_by_id_with_invalid_column(app_context: Session) -> None:
+ """Test find_by_id returns None when column doesn't exist."""
+ # This should return None gracefully
+ result = UserDAO.find_by_id("not_a_valid_id", skip_base_filter=True)
+ assert result is None
+
+
+def test_find_by_id_skip_base_filter(app_context: Session) -> None:
+ """Test find_by_id with skip_base_filter parameter."""
+ # Create users with different active states
+ active_user = User(
+ username="active_user",
+ first_name="Active",
+ last_name="User",
+ email="active@example.com",
+ active=True,
+ )
+ inactive_user = User(
+ username="inactive_user",
+ first_name="Inactive",
+ last_name="User",
+ email="inactive@example.com",
+ active=False,
+ )
+ db.session.add_all([active_user, inactive_user])
+ db.session.commit()
+
+ # Without skipping base filter (if one exists)
+ found_active = UserDAO.find_by_id(active_user.id, skip_base_filter=False)
+ assert found_active is not None
+
+ # With skipping base filter
+ found_active_skip = UserDAO.find_by_id(active_user.id, skip_base_filter=True)
+ assert found_active_skip is not None
+
+ # Both should find the user since UserDAO might not have a base filter
+ assert found_active.id == active_user.id
+ assert found_active_skip.id == active_user.id
+
+
+def test_find_by_ids_with_default_column(app_context: Session) -> None:
+ """Test find_by_ids with default 'id' column."""
+ # Create multiple users
+ users = []
+ for i in range(3):
+ user = User(
+ username=f"test_find_by_ids_{i}",
+ first_name=f"Test{i}",
+ last_name="User",
+ email=f"test{i}@example.com",
+ active=True,
+ )
+ users.append(user)
+ db.session.add(user)
+ db.session.commit()
+
+ # Find by multiple ids
+ ids = [user.id for user in users]
+ found = UserDAO.find_by_ids(ids, skip_base_filter=True)
+ assert len(found) == 3
+ found_ids = [u.id for u in found]
+ assert set(found_ids) == set(ids)
+
+ # Test with mix of existent and non-existent ids
+ mixed_ids = [users[0].id, 999999, users[1].id]
+ found_mixed = UserDAO.find_by_ids(mixed_ids, skip_base_filter=True)
+ assert len(found_mixed) == 2
+
+ # Test with empty list
+ found_empty = UserDAO.find_by_ids([], skip_base_filter=True)
+ assert found_empty == []
+
+
+def test_find_by_ids_with_uuid_column(app_context: Session) -> None:
+ """Test find_by_ids with uuid column."""
+ # Create multiple dashboards
+ dashboards = []
+ for i in range(3):
+ dashboard = Dashboard(
+ dashboard_title=f"Test UUID Dashboard {i}",
+ slug=f"test-uuid-dashboard-{i}",
+ published=True,
+ )
+ dashboards.append(dashboard)
+ db.session.add(dashboard)
+ db.session.commit()
+
+ # Find by multiple uuids
+ uuids = [str(dashboard.uuid) for dashboard in dashboards]
+ found = DashboardDAO.find_by_ids(uuids, id_column="uuid", skip_base_filter=True)
+ assert len(found) == 3
+ found_uuids = [str(d.uuid) for d in found]
+ assert set(found_uuids) == set(uuids)
+
+ # Test with mix of ids and uuids - search separately by column
+ found_by_id = DashboardDAO.find_by_ids([dashboards[0].id], skip_base_filter=True)
+ found_by_uuid = DashboardDAO.find_by_ids(
+ [str(dashboards[1].uuid)], id_column="uuid", skip_base_filter=True
+ )
+ assert len(found_by_id) == 1
+ assert len(found_by_uuid) == 1
+
+
+def test_find_by_ids_with_slug_column(app_context: Session) -> None:
+ """Test find_by_ids with slug column."""
+ # Create multiple dashboards
+ dashboards = []
+ for i in range(3):
+ dashboard = Dashboard(
+ dashboard_title=f"Test Slug Dashboard {i}",
+ slug=f"test-slug-dashboard-{i}",
+ published=True,
+ )
+ dashboards.append(dashboard)
+ db.session.add(dashboard)
+ db.session.commit()
+
+ # Find by multiple slugs
+ slugs = [dashboard.slug for dashboard in dashboards]
+ found = DashboardDAO.find_by_ids(slugs, id_column="slug", skip_base_filter=True)
+ assert len(found) == 3
+ found_slugs = [d.slug for d in found]
+ assert set(found_slugs) == set(slugs)
+
+
+def test_find_by_ids_with_invalid_column(app_context: Session) -> None:
+ """Test find_by_ids returns empty list when column doesn't exist."""
+ # This should return empty list gracefully
+ result = UserDAO.find_by_ids(["not_a_valid_id"], skip_base_filter=True)
+ assert result == []
+
+
+def test_find_by_ids_skip_base_filter(app_context: Session) -> None:
+ """Test find_by_ids with skip_base_filter parameter."""
+ # Create users
+ users = []
+ for i in range(3):
+ user = User(
+ username=f"test_skip_filter_{i}",
+ first_name=f"Test{i}",
+ last_name="User",
+ email=f"test{i}@example.com",
+ active=True,
+ )
+ users.append(user)
+ db.session.add(user)
+ db.session.commit()
+
+ ids = [user.id for user in users]
+
+ # Without skipping base filter
+ found_no_skip = UserDAO.find_by_ids(ids, skip_base_filter=False)
+ assert len(found_no_skip) == 3
+
+ # With skipping base filter
+ found_skip = UserDAO.find_by_ids(ids, skip_base_filter=True)
+ assert len(found_skip) == 3
+
+
+def test_base_dao_create_with_item(app_context: Session) -> None:
+ """Test BaseDAO.create with an item parameter."""
+ # Create a user item
+ user = User(
+ username="created_with_item",
+ first_name="Created",
+ last_name="Item",
+ email="created@example.com",
+ active=True,
+ )
+
+ # Create using the item
+ created = UserDAO.create(item=user)
+ assert created is not None
+ assert created.username == "created_with_item"
+ assert created.first_name == "Created"
+
+ # Verify it's in the session
+ assert created in db.session
+
+ # Commit and verify it persists
+ db.session.commit()
+
+ # Find it again to ensure it was saved
+ found = UserDAO.find_by_id(created.id, skip_base_filter=True)
+ assert found is not None
+ assert found.username == "created_with_item"
+
+
+def test_base_dao_create_with_attributes(app_context: Session) -> None:
+ """Test BaseDAO.create with attributes parameter."""
+ # Create using attributes dict
+ attributes = {
+ "username": "created_with_attrs",
+ "first_name": "Created",
+ "last_name": "Attrs",
+ "email": "attrs@example.com",
+ "active": True,
+ }
+
+ created = UserDAO.create(attributes=attributes)
+ assert created is not None
+ assert created.username == "created_with_attrs"
+ assert created.email == "attrs@example.com"
+
+ # Commit and verify
+ db.session.commit()
+ found = UserDAO.find_by_id(created.id, skip_base_filter=True)
+ assert found is not None
+ assert found.username == "created_with_attrs"
+
+
+def test_base_dao_create_with_both_item_and_attributes(app_context: Session) -> None:
+ """Test BaseDAO.create with both item and attributes (override behavior)."""
+ # Create a user item
+ user = User(
+ username="item_username",
+ first_name="Item",
+ last_name="User",
+ email="item@example.com",
+ active=False,
+ )
+
+ # Override some attributes
+ attributes = {
+ "username": "override_username",
+ "active": True,
+ }
+
+ created = UserDAO.create(item=user, attributes=attributes)
+ assert created is not None
+ assert created.username == "override_username" # Should be overridden
+ assert created.active is True # Should be overridden
+ assert created.first_name == "Item" # Should keep original
+ assert created.last_name == "User" # Should keep original
+
+ db.session.commit()
+
+
+def test_base_dao_update_with_item(app_context: Session) -> None:
+ """Test BaseDAO.update with an item parameter."""
+ # Create a user first
+ user = User(
+ username="update_test",
+ first_name="Original",
+ last_name="User",
+ email="original@example.com",
+ active=True,
+ )
+ db.session.add(user)
+ db.session.commit()
+
+ # Update the user
+ user.first_name = "Updated"
+ updated = UserDAO.update(item=user)
+ assert updated is not None
+ assert updated.first_name == "Updated"
+
+ db.session.commit()
+
+ # Verify the update persisted
+ found = UserDAO.find_by_id(user.id, skip_base_filter=True)
+ assert found is not None
+ assert found.first_name == "Updated"
+
+
+def test_base_dao_update_with_attributes(app_context: Session) -> None:
+ """Test BaseDAO.update with attributes parameter."""
+ # Create a user first
+ user = User(
+ username="update_attrs_test",
+ first_name="Original",
+ last_name="User",
+ email="original@example.com",
+ active=True,
+ )
+ db.session.add(user)
+ db.session.commit()
+
+ # Update using attributes
+ attributes = {"first_name": "Updated", "last_name": "Attr User"}
+ updated = UserDAO.update(item=user, attributes=attributes)
+ assert updated is not None
+ assert updated.first_name == "Updated"
+ assert updated.last_name == "Attr User"
+
+ db.session.commit()
+
+
+def test_base_dao_update_detached_item(app_context: Session) -> None:
+ """Test BaseDAO.update with a detached item."""
+ # Create a user first
+ user = User(
+ username="detached_test",
+ first_name="Original",
+ last_name="User",
+ email="detached@example.com",
+ active=True,
+ )
+ db.session.add(user)
+ db.session.commit()
+
+ user_id = user.id
+
+ # Expunge to detach from session
+ db.session.expunge(user)
+
+ # Update the detached user
+ user.first_name = "Updated Detached"
+ updated = UserDAO.update(item=user)
+ assert updated is not None
+ assert updated.first_name == "Updated Detached"
+
+ db.session.commit()
+
+ # Verify the update persisted
+ found = UserDAO.find_by_id(user_id, skip_base_filter=True)
+ assert found is not None
+ assert found.first_name == "Updated Detached"
+
+
+def test_base_dao_delete_single_item(app_context: Session) -> None:
+ """Test BaseDAO.delete with a single item."""
+ # Create a dashboard instead of user to avoid circular dependencies
+ dashboard = Dashboard(
+ dashboard_title="Delete Test",
+ slug="delete-test",
+ published=True,
+ )
+ db.session.add(dashboard)
+ db.session.commit()
+
+ dashboard_id = dashboard.id
+
+ DashboardDAO.delete([dashboard])
+ db.session.commit()
+
+ # Verify it's gone
+ found = DashboardDAO.find_by_id(dashboard_id, skip_base_filter=True)
+ assert found is None
+
+
+def test_base_dao_delete_multiple_items(app_context: Session) -> None:
+ """Test BaseDAO.delete with multiple items."""
+ # Create multiple dashboards instead of users to avoid circular dependencies
+ dashboards = []
+ for i in range(3):
+ dashboard = Dashboard(
+ dashboard_title=f"Delete Multi {i}",
+ slug=f"delete-multi-{i}",
+ published=True,
+ )
+ dashboards.append(dashboard)
+ db.session.add(dashboard)
+ db.session.commit()
+
+ dashboard_ids = [dashboard.id for dashboard in dashboards]
+
+ DashboardDAO.delete(dashboards)
+ db.session.commit()
+
+ # Verify they're all gone
+ for dashboard_id in dashboard_ids:
+ found = DashboardDAO.find_by_id(dashboard_id, skip_base_filter=True)
+ assert found is None
+
+
+def test_base_dao_delete_empty_list(app_context: Session) -> None:
+ """Test BaseDAO.delete with empty list."""
+ # Should not raise any errors
+ UserDAO.delete([])
+ db.session.commit()
+ # Just ensuring no exception is raised
+
+
+def test_base_dao_find_all(app_context: Session) -> None:
+ """Test BaseDAO.find_all method."""
+ # Create some users
+ users = []
+ for i in range(3):
+ user = User(
+ username=f"find_all_{i}",
+ first_name=f"Find{i}",
+ last_name="All",
+ email=f"findall{i}@example.com",
+ active=True,
+ )
+ users.append(user)
+ db.session.add(user)
+ db.session.commit()
+
+ # Find all users
+ all_users = UserDAO.find_all()
+ assert len(all_users) >= 3 # At least our 3 users
+
+ # Check our users are in the results
+ usernames = [u.username for u in all_users]
+ for user in users:
+ assert user.username in usernames
+
+
+def test_base_dao_find_one_or_none(app_context: Session) -> None:
+ """Test BaseDAO.find_one_or_none method."""
+ # Create users with specific criteria
+ user1 = User(
+ username="unique_username_123",
+ first_name="Unique",
+ last_name="User",
+ email="unique@example.com",
+ active=True,
+ )
+ user2 = User(
+ username="another_user",
+ first_name="Another",
+ last_name="User",
+ email="another@example.com",
+ active=True,
+ )
+ user3 = User(
+ username="third_user",
+ first_name="Another", # Same first name as user2
+ last_name="User",
+ email="third@example.com",
+ active=True,
+ )
+ db.session.add_all([user1, user2, user3])
+ db.session.commit()
+
+ # Find one with unique criteria
+ found = UserDAO.find_one_or_none(username="unique_username_123")
+ assert found is not None
+ assert found.username == "unique_username_123"
+
+ # Find none with non-existent criteria
+ not_found = UserDAO.find_one_or_none(username="non_existent_user")
+ assert not_found is None
+
+ db.session.commit()
+
+
+def test_base_dao_list_returns_results(user_with_data: Session) -> None:
+ """Test that BaseDAO.list returns results and total."""
+ results, total = UserDAO.list()
+ assert isinstance(results, list)
+ assert isinstance(total, int)
+ assert total >= 1 # At least the fixture user
+
+
+def test_base_dao_list_with_column_operators(user_with_data: Session) -> None:
+ """Test BaseDAO.list with column operators."""
+ column_operators = [
+ ColumnOperator(col="username", opr=ColumnOperatorEnum.eq, value="testuser")
+ ]
+ results, total = UserDAO.list(column_operators=column_operators)
+ assert total == 1
+ assert results[0].username == "testuser"
+
+
+def test_base_dao_list_with_non_matching_column_operator(
+ user_with_data: Session,
+) -> None:
+ """Test BaseDAO.list with non-matching column operator."""
+ column_operators = [
+ ColumnOperator(
+ col="username", opr=ColumnOperatorEnum.eq, value="nonexistentuser"
+ )
+ ]
+ results, total = UserDAO.list(column_operators=column_operators)
+ assert total == 0
+ assert results == []
+
+
+def test_base_dao_count_returns_value(user_with_data: Session) -> None:
+ """Test that BaseDAO.count returns correct count."""
+ count = UserDAO.count()
+ assert isinstance(count, int)
+ assert count >= 1 # At least the fixture user
+
+
+def test_base_dao_count_with_column_operators(user_with_data: Session) -> None:
+ """Test BaseDAO.count with column operators."""
+ # Count with matching operator
+ column_operators = [
+ ColumnOperator(col="username", opr=ColumnOperatorEnum.eq, value="testuser")
+ ]
+ count = UserDAO.count(column_operators=column_operators)
+ assert count == 1
+
+ # Count with non-matching operator
+ column_operators = [
+ ColumnOperator(
+ col="username", opr=ColumnOperatorEnum.eq, value="nonexistentuser"
+ )
+ ]
+ count = UserDAO.count(column_operators=column_operators)
+ assert count == 0
+
+
+def test_base_dao_list_ordering(user_with_data: Session) -> None:
+ """Test BaseDAO.list with ordering."""
+ # Create additional users with predictable names
+ users = []
+ for i in range(3):
+ user = User(
+ # Let database auto-generate IDs to avoid conflicts
+ username=f"order_test_{i}",
+ first_name=f"Order{i}",
+ last_name="Test",
+ email=f"order{i}@example.com",
+ active=True,
+ )
+ users.append(user)
+ user_with_data.add(user)
+ user_with_data.commit()
+
+ # Test ascending order
+ results_asc, _ = UserDAO.list(
+ order_column="username", order_direction="asc", page_size=100
+ )
+ usernames_asc = [r.username for r in results_asc]
+ # Check that our test users are in order
+ assert usernames_asc.index("order_test_0") < usernames_asc.index("order_test_1")
+ assert usernames_asc.index("order_test_1") < usernames_asc.index("order_test_2")
+
+ # Test descending order
+ results_desc, _ = UserDAO.list(
+ order_column="username", order_direction="desc", page_size=100
+ )
+ usernames_desc = [r.username for r in results_desc]
+ # Check that our test users are in reverse order
+ assert usernames_desc.index("order_test_2") < usernames_desc.index("order_test_1")
+ assert usernames_desc.index("order_test_1") < usernames_desc.index("order_test_0")
+
+ for user in users:
+ user_with_data.delete(user)
+ user_with_data.commit()
+
+
+def test_base_dao_list_paging(user_with_data: Session) -> None:
+ """Test BaseDAO.list with paging."""
+ # Create additional users for paging
+ users = []
+ for i in range(10):
+ user = User(
+ id=300 + i,
+ username=f"page_test_{i}",
+ first_name=f"Page{i}",
+ last_name="Test",
+ email=f"page{i}@example.com",
+ active=True,
+ )
+ users.append(user)
+ user_with_data.add(user)
+ user_with_data.commit()
+
+ # Test first page
+ page1_results, page1_total = UserDAO.list(page=0, page_size=5, order_column="id")
+ assert len(page1_results) <= 5
+ assert page1_total >= 10 # At least our 10 users
+
+ # Test second page
+ page2_results, page2_total = UserDAO.list(page=1, page_size=5, order_column="id")
+ assert len(page2_results) <= 5
+ assert page2_total >= 10
+
+ # Results should be different
+ page1_ids = [r.id for r in page1_results]
+ page2_ids = [r.id for r in page2_results]
+ assert set(page1_ids).isdisjoint(set(page2_ids)) # No overlap
+
+ for user in users:
+ user_with_data.delete(user)
+ user_with_data.commit()
+
+
+def test_base_dao_list_search(user_with_data: Session) -> None:
+ """Test BaseDAO.list with search."""
+ # Create users with searchable names
+ users = []
+ for i in range(3):
+ user = User(
+ id=400 + i,
+ username=f"searchable_{i}",
+ first_name=f"Searchable{i}",
+ last_name="User",
+ email=f"search{i}@example.com",
+ active=True,
+ )
+ users.append(user)
+ user_with_data.add(user)
+
+ # Create some non-matching users
+ for i in range(2):
+ user = User(
+ id=410 + i,
+ username=f"other_{i}",
+ first_name=f"Other{i}",
+ last_name="Person",
+ email=f"other{i}@example.com",
+ active=True,
+ )
+ users.append(user)
+ user_with_data.add(user)
+
+ user_with_data.commit()
+
+ # Search for "searchable"
+ results, total = UserDAO.list(
+ search="searchable", search_columns=["username", "first_name"]
+ )
+ assert total >= 3 # At least our 3 searchable users
+
+ # Verify all results contain "searchable"
+ for result in results:
+ assert (
+ "searchable" in result.username.lower()
+ or "searchable" in result.first_name.lower()
+ )
Review Comment:
<div>
<div id="suggestion">
<div id="issue"><b>Search test too brittle</b></div>
<div id="fix">
The search test assertions are too brittle when existing database users match the search criteria. The test should verify that the created searchable users are found rather than asserting all results contain the search term. This prevents false failures in production environments with existing data.
</div>
<details>
<summary>
<b>Code suggestion</b>
</summary>
<blockquote>Check the AI-generated fix before applying</blockquote>
<div id="code">
```suggestion
assert total >= 3 # At least our 3 searchable users
# Verify search functionality works - count how many of our test users are found
searchable_test_users = [r for r in results if "searchable" in (r.username.lower() + " " + r.first_name.lower())]
assert len(searchable_test_users) >= 3 # Our created test users should be found
# Verify the search actually filtered results (total should be reasonable)
assert total > 0 # Search returned some results
```
</div>
</details>
</div>
<small><i>Code Review Run <a href=https://github.com/apache/superset/pull/35018#issuecomment-3299702064>#b918c4</a></i></small>
</div>
---
Should Bito avoid suggestions like this for future reviews? (<a href=https://alpha.bito.ai/home/ai-agents/review-rules>Manage Rules</a>)
- [ ] Yes, avoid them
##########
superset/daos/dataset.py:
##########
@@ -35,8 +35,20 @@
logger = logging.getLogger(__name__)
+# Custom filterable fields for datasets
+DATASET_CUSTOM_FIELDS = {
+ "schema": ["eq", "in_"],
+}
Review Comment:
<div>
<div id="suggestion">
<div id="issue"><b>Duplicate schema field definition</b></div>
<div id="fix">
The custom field "schema" in DATASET_CUSTOM_FIELDS conflicts with the existing database column "schema" in SqlaTable. Since SqlaTable already has a schema column (line 1141 in SqlaTable), this creates a duplicate key that will override the actual database column definition. The base get_filterable_columns_and_operators() method will automatically include the schema field with proper type detection and operator mapping. Remove "schema" from DATASET_CUSTOM_FIELDS to prevent this conflict.
</div>
<details>
<summary>
<b>Code suggestion</b>
</summary>
<blockquote>Check the AI-generated fix before applying</blockquote>
<div id="code">
```suggestion
# Custom filterable fields for datasets
DATASET_CUSTOM_FIELDS = {}
```
</div>
</details>
</div>
<small><i>Code Review Run <a href=https://github.com/apache/superset/pull/35018#issuecomment-3299702064>#b918c4</a></i></small>
</div>
---
Should Bito avoid suggestions like this for future reviews? (<a href=https://alpha.bito.ai/home/ai-agents/review-rules>Manage Rules</a>)
- [ ] Yes, avoid them
##########
tests/integration_tests/dao/base_dao_test.py:
##########
@@ -0,0 +1,1610 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""
+Integration tests for BaseDAO functionality.
+
+This module contains comprehensive integration tests for the BaseDAO class and its
+subclasses, covering database operations, CRUD methods, flexible column support,
+column operators, and error handling.
+
+Tests use an in-memory SQLite database for isolation and to replicate the unit test
+environment behavior. User model deletions are avoided due to circular dependency
+constraints with self-referential foreign keys.
+"""
+
+import datetime
+import time
+import uuid
+
+import pytest
+from flask_appbuilder.models.filters import BaseFilter
+from flask_appbuilder.security.sqla.models import User
+from sqlalchemy import Column, DateTime, Integer, String
+from sqlalchemy.ext.declarative import declarative_base
+from sqlalchemy.orm.session import Session
+
+from superset.daos.base import BaseDAO, ColumnOperator, ColumnOperatorEnum
+from superset.daos.chart import ChartDAO
+from superset.daos.dashboard import DashboardDAO
+from superset.daos.database import DatabaseDAO
+from superset.daos.user import UserDAO
+from superset.extensions import db
+from superset.models.core import Database
+from superset.models.dashboard import Dashboard
+from superset.models.slice import Slice
+
+# Create a test model for comprehensive testing
+Base = declarative_base()
+
+
+class ExampleModel(Base): # type: ignore
+ __tablename__ = "example_model"
+ id = Column(Integer, primary_key=True)
+ uuid = Column(String(36), unique=True, nullable=False)
+ slug = Column(String(100), unique=True)
+ name = Column(String(100))
+ code = Column(String(50), unique=True)
+ created_on = Column(DateTime, default=datetime.datetime.utcnow)
+
+
+class ExampleModelDAO(BaseDAO[ExampleModel]):
+ model_cls = ExampleModel
+ id_column_name = "id"
+ base_filter = None
+
+
+@pytest.fixture(autouse=True)
+def mock_g_user(app_context):
+ """Mock the flask g.user for security context."""
+ # Within app context, we can safely mock g
+ from flask import g
+
+ mock_user = User()
+ mock_user.id = 1
+ mock_user.username = "test_user"
+
+ # Set g.user directly instead of patching
+ g.user = mock_user
+ yield
+
+ # Clean up
+ if hasattr(g, "user"):
+ delattr(g, "user")
+
+
+# =============================================================================
+# Integration Tests - These tests use the actual database
+# =============================================================================
+
+
+def test_column_operator_enum_complete_coverage(user_with_data: Session) -> None:
+ """
+ Test that every single ColumnOperatorEnum operator is covered by tests.
+ This ensures we have comprehensive test coverage for all operators.
+ """
+ # Simply verify that we can create queries with all operators
+ for operator in ColumnOperatorEnum:
+ column_operator = ColumnOperator(
+ col="username", opr=operator, value="test_value"
+ )
+ # Just check it doesn't raise an error
+ assert column_operator.opr == operator
+
+
+def test_find_by_id_with_default_column(app_context: Session) -> None:
+ """Test find_by_id with default 'id' column."""
+ # Create a user to test with
+ user = User(
+ username="test_find_by_id",
+ first_name="Test",
+ last_name="User",
+ email="test@example.com",
+ active=True,
+ )
+ db.session.add(user)
+ db.session.commit()
+
+ # Find by numeric id
+ found = UserDAO.find_by_id(user.id, skip_base_filter=True)
+ assert found is not None
+ assert found.id == user.id
+ assert found.username == "test_find_by_id"
+
+ # Test with non-existent id
+ not_found = UserDAO.find_by_id(999999, skip_base_filter=True)
+ assert not_found is None
+
+
+def test_find_by_id_with_uuid_column(app_context: Session) -> None:
+ """Test find_by_id with custom uuid column."""
+ # Create a dashboard with uuid
+ dashboard = Dashboard(
+ dashboard_title="Test UUID Dashboard",
+ slug="test-uuid-dashboard",
+ published=True,
+ )
+ db.session.add(dashboard)
+ db.session.commit()
+
+ # Find by uuid string using the uuid column
+ found = DashboardDAO.find_by_id(
+ str(dashboard.uuid), id_column="uuid", skip_base_filter=True
+ )
+ assert found is not None
+ assert found.uuid == dashboard.uuid
+ assert found.dashboard_title == "Test UUID Dashboard"
+
+ # Find by numeric id (should still work)
+ found_by_id = DashboardDAO.find_by_id(dashboard.id, skip_base_filter=True)
+ assert found_by_id is not None
+ assert found_by_id.id == dashboard.id
+
+ # Test with non-existent uuid
+ not_found = DashboardDAO.find_by_id(str(uuid.uuid4()), skip_base_filter=True)
+ assert not_found is None
+
+
+def test_find_by_id_with_slug_column(app_context: Session) -> None:
+ """Test find_by_id with slug column fallback."""
+ # Create a dashboard with slug
+ dashboard = Dashboard(
+ dashboard_title="Test Slug Dashboard",
+ slug="test-slug-dashboard",
+ published=True,
+ )
+ db.session.add(dashboard)
+ db.session.commit()
+
+ # Find by slug using the slug column
+ found = DashboardDAO.find_by_id(
+ "test-slug-dashboard", id_column="slug", skip_base_filter=True
+ )
+ assert found is not None
+ assert found.slug == "test-slug-dashboard"
+ assert found.dashboard_title == "Test Slug Dashboard"
+
+ # Test with non-existent slug
+ not_found = DashboardDAO.find_by_id("non-existent-slug", skip_base_filter=True)
+ assert not_found is None
+
+
+def test_find_by_id_with_invalid_column(app_context: Session) -> None:
+ """Test find_by_id returns None when column doesn't exist."""
+ # This should return None gracefully
+ result = UserDAO.find_by_id("not_a_valid_id", skip_base_filter=True)
+ assert result is None
+
+
+def test_find_by_id_skip_base_filter(app_context: Session) -> None:
+ """Test find_by_id with skip_base_filter parameter."""
+ # Create users with different active states
+ active_user = User(
+ username="active_user",
+ first_name="Active",
+ last_name="User",
+ email="active@example.com",
+ active=True,
+ )
+ inactive_user = User(
+ username="inactive_user",
+ first_name="Inactive",
+ last_name="User",
+ email="inactive@example.com",
+ active=False,
+ )
+ db.session.add_all([active_user, inactive_user])
+ db.session.commit()
+
+ # Without skipping base filter (if one exists)
+ found_active = UserDAO.find_by_id(active_user.id, skip_base_filter=False)
+ assert found_active is not None
+
+ # With skipping base filter
+ found_active_skip = UserDAO.find_by_id(active_user.id, skip_base_filter=True)
+ assert found_active_skip is not None
+
+ # Both should find the user since UserDAO might not have a base filter
+ assert found_active.id == active_user.id
+ assert found_active_skip.id == active_user.id
+
+
+def test_find_by_ids_with_default_column(app_context: Session) -> None:
+ """Test find_by_ids with default 'id' column."""
+ # Create multiple users
+ users = []
+ for i in range(3):
+ user = User(
+ username=f"test_find_by_ids_{i}",
+ first_name=f"Test{i}",
+ last_name="User",
+ email=f"test{i}@example.com",
+ active=True,
+ )
+ users.append(user)
+ db.session.add(user)
+ db.session.commit()
+
+ # Find by multiple ids
+ ids = [user.id for user in users]
+ found = UserDAO.find_by_ids(ids, skip_base_filter=True)
+ assert len(found) == 3
+ found_ids = [u.id for u in found]
+ assert set(found_ids) == set(ids)
+
+ # Test with mix of existent and non-existent ids
+ mixed_ids = [users[0].id, 999999, users[1].id]
+ found_mixed = UserDAO.find_by_ids(mixed_ids, skip_base_filter=True)
+ assert len(found_mixed) == 2
+
+ # Test with empty list
+ found_empty = UserDAO.find_by_ids([], skip_base_filter=True)
+ assert found_empty == []
+
+
+def test_find_by_ids_with_uuid_column(app_context: Session) -> None:
+ """Test find_by_ids with uuid column."""
+ # Create multiple dashboards
+ dashboards = []
+ for i in range(3):
+ dashboard = Dashboard(
+ dashboard_title=f"Test UUID Dashboard {i}",
+ slug=f"test-uuid-dashboard-{i}",
+ published=True,
+ )
+ dashboards.append(dashboard)
+ db.session.add(dashboard)
+ db.session.commit()
+
+ # Find by multiple uuids
+ uuids = [str(dashboard.uuid) for dashboard in dashboards]
+ found = DashboardDAO.find_by_ids(uuids, id_column="uuid", skip_base_filter=True)
+ assert len(found) == 3
+ found_uuids = [str(d.uuid) for d in found]
+ assert set(found_uuids) == set(uuids)
+
+ # Test with mix of ids and uuids - search separately by column
+ found_by_id = DashboardDAO.find_by_ids([dashboards[0].id], skip_base_filter=True)
+ found_by_uuid = DashboardDAO.find_by_ids(
+ [str(dashboards[1].uuid)], id_column="uuid", skip_base_filter=True
+ )
+ assert len(found_by_id) == 1
+ assert len(found_by_uuid) == 1
+
+
+def test_find_by_ids_with_slug_column(app_context: Session) -> None:
+ """Test find_by_ids with slug column."""
+ # Create multiple dashboards
+ dashboards = []
+ for i in range(3):
+ dashboard = Dashboard(
+ dashboard_title=f"Test Slug Dashboard {i}",
+ slug=f"test-slug-dashboard-{i}",
+ published=True,
+ )
+ dashboards.append(dashboard)
+ db.session.add(dashboard)
+ db.session.commit()
+
+ # Find by multiple slugs
+ slugs = [dashboard.slug for dashboard in dashboards]
+ found = DashboardDAO.find_by_ids(slugs, id_column="slug", skip_base_filter=True)
+ assert len(found) == 3
+ found_slugs = [d.slug for d in found]
+ assert set(found_slugs) == set(slugs)
+
+
+def test_find_by_ids_with_invalid_column(app_context: Session) -> None:
+ """Test find_by_ids returns empty list when column doesn't exist."""
+ # This should return empty list gracefully
+ result = UserDAO.find_by_ids(["not_a_valid_id"], skip_base_filter=True)
+ assert result == []
+
+
+def test_find_by_ids_skip_base_filter(app_context: Session) -> None:
+ """Test find_by_ids with skip_base_filter parameter."""
+ # Create users
+ users = []
+ for i in range(3):
+ user = User(
+ username=f"test_skip_filter_{i}",
+ first_name=f"Test{i}",
+ last_name="User",
+ email=f"test{i}@example.com",
+ active=True,
+ )
+ users.append(user)
+ db.session.add(user)
+ db.session.commit()
+
+ ids = [user.id for user in users]
+
+ # Without skipping base filter
+ found_no_skip = UserDAO.find_by_ids(ids, skip_base_filter=False)
+ assert len(found_no_skip) == 3
+
+ # With skipping base filter
+ found_skip = UserDAO.find_by_ids(ids, skip_base_filter=True)
+ assert len(found_skip) == 3
+
+
+def test_base_dao_create_with_item(app_context: Session) -> None:
+ """Test BaseDAO.create with an item parameter."""
+ # Create a user item
+ user = User(
+ username="created_with_item",
+ first_name="Created",
+ last_name="Item",
+ email="created@example.com",
+ active=True,
+ )
+
+ # Create using the item
+ created = UserDAO.create(item=user)
+ assert created is not None
+ assert created.username == "created_with_item"
+ assert created.first_name == "Created"
+
+ # Verify it's in the session
+ assert created in db.session
+
+ # Commit and verify it persists
+ db.session.commit()
+
+ # Find it again to ensure it was saved
+ found = UserDAO.find_by_id(created.id, skip_base_filter=True)
+ assert found is not None
+ assert found.username == "created_with_item"
+
+
+def test_base_dao_create_with_attributes(app_context: Session) -> None:
+ """Test BaseDAO.create with attributes parameter."""
+ # Create using attributes dict
+ attributes = {
+ "username": "created_with_attrs",
+ "first_name": "Created",
+ "last_name": "Attrs",
+ "email": "attrs@example.com",
+ "active": True,
+ }
+
+ created = UserDAO.create(attributes=attributes)
+ assert created is not None
+ assert created.username == "created_with_attrs"
+ assert created.email == "attrs@example.com"
+
+ # Commit and verify
+ db.session.commit()
+ found = UserDAO.find_by_id(created.id, skip_base_filter=True)
+ assert found is not None
+ assert found.username == "created_with_attrs"
+
+
+def test_base_dao_create_with_both_item_and_attributes(app_context: Session) -> None:
+ """Test BaseDAO.create with both item and attributes (override behavior)."""
+ # Create a user item
+ user = User(
+ username="item_username",
+ first_name="Item",
+ last_name="User",
+ email="item@example.com",
+ active=False,
+ )
+
+ # Override some attributes
+ attributes = {
+ "username": "override_username",
+ "active": True,
+ }
+
+ created = UserDAO.create(item=user, attributes=attributes)
+ assert created is not None
+ assert created.username == "override_username" # Should be overridden
+ assert created.active is True # Should be overridden
+ assert created.first_name == "Item" # Should keep original
+ assert created.last_name == "User" # Should keep original
+
+ db.session.commit()
+
+
+def test_base_dao_update_with_item(app_context: Session) -> None:
+ """Test BaseDAO.update with an item parameter."""
+ # Create a user first
+ user = User(
+ username="update_test",
+ first_name="Original",
+ last_name="User",
+ email="original@example.com",
+ active=True,
+ )
+ db.session.add(user)
+ db.session.commit()
+
+ # Update the user
+ user.first_name = "Updated"
+ updated = UserDAO.update(item=user)
+ assert updated is not None
+ assert updated.first_name == "Updated"
+
+ db.session.commit()
+
+ # Verify the update persisted
+ found = UserDAO.find_by_id(user.id, skip_base_filter=True)
+ assert found is not None
+ assert found.first_name == "Updated"
+
+
+def test_base_dao_update_with_attributes(app_context: Session) -> None:
+ """Test BaseDAO.update with attributes parameter."""
+ # Create a user first
+ user = User(
+ username="update_attrs_test",
+ first_name="Original",
+ last_name="User",
+ email="original@example.com",
+ active=True,
+ )
+ db.session.add(user)
+ db.session.commit()
+
+ # Update using attributes
+ attributes = {"first_name": "Updated", "last_name": "Attr User"}
+ updated = UserDAO.update(item=user, attributes=attributes)
+ assert updated is not None
+ assert updated.first_name == "Updated"
+ assert updated.last_name == "Attr User"
+
+ db.session.commit()
+
+
+def test_base_dao_update_detached_item(app_context: Session) -> None:
+ """Test BaseDAO.update with a detached item."""
+ # Create a user first
+ user = User(
+ username="detached_test",
+ first_name="Original",
+ last_name="User",
+ email="detached@example.com",
+ active=True,
+ )
+ db.session.add(user)
+ db.session.commit()
+
+ user_id = user.id
+
+ # Expunge to detach from session
+ db.session.expunge(user)
+
+ # Update the detached user
+ user.first_name = "Updated Detached"
+ updated = UserDAO.update(item=user)
+ assert updated is not None
+ assert updated.first_name == "Updated Detached"
+
+ db.session.commit()
+
+ # Verify the update persisted
+ found = UserDAO.find_by_id(user_id, skip_base_filter=True)
+ assert found is not None
+ assert found.first_name == "Updated Detached"
+
+
+def test_base_dao_delete_single_item(app_context: Session) -> None:
+ """Test BaseDAO.delete with a single item."""
+ # Create a dashboard instead of user to avoid circular dependencies
+ dashboard = Dashboard(
+ dashboard_title="Delete Test",
+ slug="delete-test",
+ published=True,
+ )
+ db.session.add(dashboard)
+ db.session.commit()
+
+ dashboard_id = dashboard.id
+
+ DashboardDAO.delete([dashboard])
+ db.session.commit()
+
+ # Verify it's gone
+ found = DashboardDAO.find_by_id(dashboard_id, skip_base_filter=True)
+ assert found is None
+
+
+def test_base_dao_delete_multiple_items(app_context: Session) -> None:
+ """Test BaseDAO.delete with multiple items."""
+ # Create multiple dashboards instead of users to avoid circular dependencies
+ dashboards = []
+ for i in range(3):
+ dashboard = Dashboard(
+ dashboard_title=f"Delete Multi {i}",
+ slug=f"delete-multi-{i}",
+ published=True,
+ )
+ dashboards.append(dashboard)
+ db.session.add(dashboard)
+ db.session.commit()
+
+ dashboard_ids = [dashboard.id for dashboard in dashboards]
+
+ DashboardDAO.delete(dashboards)
+ db.session.commit()
+
+ # Verify they're all gone
+ for dashboard_id in dashboard_ids:
+ found = DashboardDAO.find_by_id(dashboard_id, skip_base_filter=True)
+ assert found is None
+
+
+def test_base_dao_delete_empty_list(app_context: Session) -> None:
+ """Test BaseDAO.delete with empty list."""
+ # Should not raise any errors
+ UserDAO.delete([])
+ db.session.commit()
+ # Just ensuring no exception is raised
+
+
+def test_base_dao_find_all(app_context: Session) -> None:
+ """Test BaseDAO.find_all method."""
+ # Create some users
+ users = []
+ for i in range(3):
+ user = User(
+ username=f"find_all_{i}",
+ first_name=f"Find{i}",
+ last_name="All",
+ email=f"findall{i}@example.com",
+ active=True,
+ )
+ users.append(user)
+ db.session.add(user)
+ db.session.commit()
+
+ # Find all users
+ all_users = UserDAO.find_all()
+ assert len(all_users) >= 3 # At least our 3 users
+
+ # Check our users are in the results
+ usernames = [u.username for u in all_users]
+ for user in users:
+ assert user.username in usernames
+
+
+def test_base_dao_find_one_or_none(app_context: Session) -> None:
+ """Test BaseDAO.find_one_or_none method."""
+ # Create users with specific criteria
+ user1 = User(
+ username="unique_username_123",
+ first_name="Unique",
+ last_name="User",
+ email="unique@example.com",
+ active=True,
+ )
+ user2 = User(
+ username="another_user",
+ first_name="Another",
+ last_name="User",
+ email="another@example.com",
+ active=True,
+ )
+ user3 = User(
+ username="third_user",
+ first_name="Another", # Same first name as user2
+ last_name="User",
+ email="third@example.com",
+ active=True,
+ )
+ db.session.add_all([user1, user2, user3])
+ db.session.commit()
+
+ # Find one with unique criteria
+ found = UserDAO.find_one_or_none(username="unique_username_123")
+ assert found is not None
+ assert found.username == "unique_username_123"
+
+ # Find none with non-existent criteria
+ not_found = UserDAO.find_one_or_none(username="non_existent_user")
+ assert not_found is None
+
+ db.session.commit()
+
+
+def test_base_dao_list_returns_results(user_with_data: Session) -> None:
+ """Test that BaseDAO.list returns results and total."""
+ results, total = UserDAO.list()
+ assert isinstance(results, list)
+ assert isinstance(total, int)
+ assert total >= 1 # At least the fixture user
+
+
+def test_base_dao_list_with_column_operators(user_with_data: Session) -> None:
+ """Test BaseDAO.list with column operators."""
+ column_operators = [
+ ColumnOperator(col="username", opr=ColumnOperatorEnum.eq, value="testuser")
+ ]
+ results, total = UserDAO.list(column_operators=column_operators)
+ assert total == 1
+ assert results[0].username == "testuser"
+
+
+def test_base_dao_list_with_non_matching_column_operator(
+ user_with_data: Session,
+) -> None:
+ """Test BaseDAO.list with non-matching column operator."""
+ column_operators = [
+ ColumnOperator(
+ col="username", opr=ColumnOperatorEnum.eq, value="nonexistentuser"
+ )
+ ]
+ results, total = UserDAO.list(column_operators=column_operators)
+ assert total == 0
+ assert results == []
+
+
+def test_base_dao_count_returns_value(user_with_data: Session) -> None:
+ """Test that BaseDAO.count returns correct count."""
+ count = UserDAO.count()
+ assert isinstance(count, int)
+ assert count >= 1 # At least the fixture user
+
+
+def test_base_dao_count_with_column_operators(user_with_data: Session) -> None:
+ """Test BaseDAO.count with column operators."""
+ # Count with matching operator
+ column_operators = [
+ ColumnOperator(col="username", opr=ColumnOperatorEnum.eq, value="testuser")
+ ]
+ count = UserDAO.count(column_operators=column_operators)
+ assert count == 1
+
+ # Count with non-matching operator
+ column_operators = [
+ ColumnOperator(
+ col="username", opr=ColumnOperatorEnum.eq, value="nonexistentuser"
+ )
+ ]
+ count = UserDAO.count(column_operators=column_operators)
+ assert count == 0
+
+
+def test_base_dao_list_ordering(user_with_data: Session) -> None:
+ """Test BaseDAO.list with ordering."""
+ # Create additional users with predictable names
+ users = []
+ for i in range(3):
+ user = User(
+ # Let database auto-generate IDs to avoid conflicts
+ username=f"order_test_{i}",
+ first_name=f"Order{i}",
+ last_name="Test",
+ email=f"order{i}@example.com",
+ active=True,
+ )
+ users.append(user)
+ user_with_data.add(user)
+ user_with_data.commit()
+
+ # Test ascending order
+ results_asc, _ = UserDAO.list(
+ order_column="username", order_direction="asc", page_size=100
+ )
+ usernames_asc = [r.username for r in results_asc]
+ # Check that our test users are in order
+ assert usernames_asc.index("order_test_0") < usernames_asc.index("order_test_1")
+ assert usernames_asc.index("order_test_1") < usernames_asc.index("order_test_2")
+
+ # Test descending order
+ results_desc, _ = UserDAO.list(
+ order_column="username", order_direction="desc", page_size=100
+ )
+ usernames_desc = [r.username for r in results_desc]
+ # Check that our test users are in reverse order
+ assert usernames_desc.index("order_test_2") < usernames_desc.index("order_test_1")
+ assert usernames_desc.index("order_test_1") < usernames_desc.index("order_test_0")
+
+ for user in users:
+ user_with_data.delete(user)
+ user_with_data.commit()
+
+
+def test_base_dao_list_paging(user_with_data: Session) -> None:
+ """Test BaseDAO.list with paging."""
+ # Create additional users for paging
+ users = []
+ for i in range(10):
+ user = User(
+ id=300 + i,
Review Comment:
<div>
<div id="suggestion">
<div id="issue"><b>Primary key violation risk</b></div>
<div id="fix">
The test is manually assigning IDs (300+i) to User objects, which conflicts with the database's auto-increment primary key mechanism. This will cause primary key violations when the database already contains users in this ID range. Remove the manual ID assignment and let the database auto-generate unique IDs.
</div>
<details>
<summary>
<b>Code suggestion</b>
</summary>
<blockquote>Check the AI-generated fix before applying</blockquote>
<div id="code">
```suggestion
user = User(
```
</div>
</details>
</div>
<small><i>Code Review Run <a href=https://github.com/apache/superset/pull/35018#issuecomment-3299702064>#b918c4</a></i></small>
</div>
---
Should Bito avoid suggestions like this for future reviews? (<a href=https://alpha.bito.ai/home/ai-agents/review-rules>Manage Rules</a>)
- [ ] Yes, avoid them
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat: Add BaseDAO improvements and test reorganization [superset]
Posted by "aminghadersohi (via GitHub)" <gi...@apache.org>.
aminghadersohi commented on code in PR #35018:
URL: https://github.com/apache/superset/pull/35018#discussion_r2334555965
##########
superset/daos/base.py:
##########
@@ -251,3 +455,205 @@ def filter_by(cls, **filter_by: Any) -> list[T]:
cls.id_column_name, data_model
).apply(query, None)
return query.filter_by(**filter_by).all()
+
+ @classmethod
+ def apply_column_operators(
+ cls, query: Any, column_operators: Optional[List[ColumnOperator]] = None
+ ) -> Any:
+ """
+ Apply column operators (list of ColumnOperator) to the query using
+ ColumnOperatorEnum logic. Raises ValueError if a filter references a
+ non-existent column.
+ """
+ if not column_operators:
+ return query
+ for c in column_operators:
+ if not isinstance(c, ColumnOperator):
+ continue
+ col = c.col
+ opr = c.opr
+ value = c.value
+ if not col or not hasattr(cls.model_cls, col):
+ model_name = cls.model_cls.__name__ if cls.model_cls else "Unknown"
+ logging.error(
+ f"Invalid filter: column '{col}' does not exist on {model_name}"
+ )
+ raise ValueError(
+ f"Invalid filter: column '{col}' does not exist on {model_name}"
+ )
+ column = getattr(cls.model_cls, col)
+ try:
+ # Always use ColumnOperatorEnum's apply method
+ operator_enum = ColumnOperatorEnum(opr)
+ query = query.filter(operator_enum.apply(column, value))
+ except Exception as e:
+ logging.error(f"Error applying filter on column '{col}': {e}")
+ raise
+ return query
+
+ @classmethod
+ def get_filterable_columns_and_operators(cls) -> Dict[str, List[str]]:
+ """
+ Returns a dict mapping filterable columns (including hybrid/computed fields if
+ present) to their supported operators. Used by MCP tools to dynamically expose
+ filter options. Custom fields supported by the DAO but not present on the model
+ should be documented here.
+ """
+
+ mapper = inspect(cls.model_cls)
+ columns = {c.key: c for c in mapper.columns}
+ # Add hybrid properties
+ hybrids = {
+ name: attr
+ for name, attr in vars(cls.model_cls).items()
+ if isinstance(attr, hybrid_property)
+ }
+ # You may add custom fields here, e.g.:
+ # custom_fields = {"tags": ["eq", "in_", "like"], ...}
+ custom_fields: Dict[str, List[str]] = {}
+
+ filterable = {}
+ for name, col in columns.items():
+ if isinstance(col.type, (sa.String, sa.Text)):
+ filterable[name] = TYPE_OPERATOR_MAP["string"]
+ elif isinstance(col.type, (sa.Boolean,)):
+ filterable[name] = TYPE_OPERATOR_MAP["boolean"]
+ elif isinstance(col.type, (sa.Integer, sa.Float, sa.Numeric)):
+ filterable[name] = TYPE_OPERATOR_MAP["number"]
+ elif isinstance(col.type, (sa.DateTime, sa.Date, sa.Time)):
+ filterable[name] = TYPE_OPERATOR_MAP["datetime"]
+ else:
+ # Fallback to eq/ne/null
+ filterable[name] = ["eq", "ne", "is_null", "is_not_null"]
+ # Add hybrid properties as string fields by default
+ for name in hybrids:
+ filterable[name] = TYPE_OPERATOR_MAP["string"]
+ # Add custom fields
+ filterable.update(custom_fields)
+ return filterable
+
+ @classmethod
+ def _build_query(
+ cls,
+ column_operators: Optional[List[ColumnOperator]] = None,
+ search: Optional[str] = None,
+ search_columns: Optional[List[str]] = None,
+ custom_filters: Optional[Dict[str, BaseFilter]] = None,
+ skip_base_filter: bool = False,
+ data_model: Optional[SQLAInterface] = None,
+ ) -> Any:
+ """
+ Build a SQLAlchemy query with base filter, column operators, search, and
+ custom filters.
+ """
+ if data_model is None:
+ data_model = SQLAInterface(cls.model_cls, db.session)
+ query = data_model.session.query(cls.model_cls)
+ query = cls._apply_base_filter(
+ query, skip_base_filter=skip_base_filter, data_model=data_model
+ )
+ if search and search_columns:
+ search_filters = []
+ for column_name in search_columns:
+ if hasattr(cls.model_cls, column_name):
+ column = getattr(cls.model_cls, column_name)
+ search_filters.append(cast(column, Text).ilike(f"%{search}%"))
+ if search_filters:
+ query = query.filter(or_(*search_filters))
+ if custom_filters:
+ for filter_class in custom_filters.values():
+ query = filter_class.apply(query, None)
+ if column_operators:
+ query = cls.apply_column_operators(query, column_operators)
+ return query
+
+ @classmethod
+ def list( # noqa: C901
+ cls,
+ column_operators: Optional[List[ColumnOperator]] = None,
+ order_column: str = "changed_on",
+ order_direction: str = "desc",
+ page: int = 0,
+ page_size: int = 100,
+ search: Optional[str] = None,
+ search_columns: Optional[List[str]] = None,
+ custom_filters: Optional[Dict[str, BaseFilter]] = None,
+ columns: Optional[List[str]] = None,
+ ) -> Tuple[List[Any], int]:
+ """
+ Generic list method for filtered, sorted, and paginated results.
+ If columns is specified, returns a list of tuples (one per row),
+ otherwise returns model instances.
+ """
+ data_model = SQLAInterface(cls.model_cls, db.session)
+
+ column_attrs = []
+ relationship_loads = []
+ if columns is None:
+ columns = []
+ for name in columns:
+ attr = getattr(cls.model_cls, name, None)
+ if attr is None:
+ continue
+ prop = getattr(attr, "property", None)
+ if isinstance(prop, ColumnProperty):
+ column_attrs.append(attr)
+ elif isinstance(prop, RelationshipProperty):
+ relationship_loads.append(joinedload(attr))
Review Comment:
i had some initial trouble with using relations, so they aren't used right now by the mcp service in its discovery tools. but thats a good flag to revisit. i will write back my findings.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat: Add BaseDAO improvements and test reorganization [superset]
Posted by "aminghadersohi (via GitHub)" <gi...@apache.org>.
aminghadersohi commented on code in PR #35018:
URL: https://github.com/apache/superset/pull/35018#discussion_r2334772030
##########
tests/integration_tests/dao/base_dao_test.py:
##########
@@ -0,0 +1,1397 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""
+Integration tests for BaseDAO functionality.
+
+This module contains comprehensive integration tests for the BaseDAO class and its
+subclasses, covering database operations, CRUD methods, flexible column support,
+column operators, and error handling.
+
+Tests use an in-memory SQLite database for isolation and to replicate the unit test
+environment behavior. User model deletions are avoided due to circular dependency
+constraints with self-referential foreign keys.
+"""
+
+import datetime
+import time
+import uuid
+
+import pytest
+from flask_appbuilder.models.filters import BaseFilter
+from flask_appbuilder.security.sqla.models import User
+from sqlalchemy import Column, DateTime, Integer, String
+from sqlalchemy.ext.declarative import declarative_base
+from sqlalchemy.orm.session import Session
+
+from superset.daos.base import BaseDAO, ColumnOperator, ColumnOperatorEnum
+from superset.daos.chart import ChartDAO
+from superset.daos.dashboard import DashboardDAO
+from superset.daos.database import DatabaseDAO
+from superset.daos.user import UserDAO
+from superset.extensions import db
+from superset.models.core import Database
+from superset.models.dashboard import Dashboard
+from superset.models.slice import Slice
+
+# Create a test model for comprehensive testing
+Base = declarative_base()
+
+
+class ExampleModel(Base): # type: ignore
+ __tablename__ = "example_model"
+ id = Column(Integer, primary_key=True)
+ uuid = Column(String(36), unique=True, nullable=False)
+ slug = Column(String(100), unique=True)
+ name = Column(String(100))
+ code = Column(String(50), unique=True)
+ created_on = Column(DateTime, default=datetime.datetime.utcnow)
+
+
+class ExampleModelDAO(BaseDAO[ExampleModel]):
+ model_cls = ExampleModel
+ id_column_name = "id"
+ base_filter = None
+
+
+class MockModel:
+ def __init__(self, id=1, name="test"):
+ self.id = id
+ self.name = name
+
+
+class TestDAO(BaseDAO[MockModel]):
+ model_cls = MockModel
+
+
+@pytest.fixture(autouse=True)
+def mock_g_user(app_context):
+ """Mock the flask g.user for security context."""
+ # Within app context, we can safely mock g
+ from flask import g
+
+ mock_user = User()
+ mock_user.id = 1
+ mock_user.username = "test_user"
+
+ # Set g.user directly instead of patching
+ g.user = mock_user
+ yield
+
+ # Clean up
+ if hasattr(g, "user"):
+ delattr(g, "user")
+
+
+# =============================================================================
+# Integration Tests - These tests use the actual database
+# =============================================================================
+
+
+def test_column_operator_enum_complete_coverage(user_with_data: Session) -> None:
+ """
+ Test that every single ColumnOperatorEnum operator is covered by tests.
+ This ensures we have comprehensive test coverage for all operators.
+ """
+ # Simply verify that we can create queries with all operators
+ for operator in ColumnOperatorEnum:
+ column_operator = ColumnOperator(
+ col="username", opr=operator, value="test_value"
+ )
+ # Just check it doesn't raise an error
+ assert column_operator.opr == operator
+
+
+def test_find_by_id_with_default_column(app_context: Session) -> None:
+ """Test find_by_id with default 'id' column."""
+ # Create a user to test with
+ user = User(
+ username="test_find_by_id",
+ first_name="Test",
+ last_name="User",
+ email="test@example.com",
+ active=True,
+ )
+ db.session.add(user)
+ db.session.commit()
+
+ # Find by numeric id
+ found = UserDAO.find_by_id(user.id, skip_base_filter=True)
+ assert found is not None
+ assert found.id == user.id
+ assert found.username == "test_find_by_id"
+
+ # Test with non-existent id
+ not_found = UserDAO.find_by_id(999999, skip_base_filter=True)
+ assert not_found is None
+
+
+def test_find_by_id_with_uuid_column(app_context: Session) -> None:
+ """Test find_by_id with custom uuid column."""
+ # Create a dashboard with uuid
+ dashboard = Dashboard(
+ dashboard_title="Test UUID Dashboard",
+ slug="test-uuid-dashboard",
+ published=True,
+ )
+ db.session.add(dashboard)
+ db.session.commit()
+
+ # Find by uuid string using the uuid column
+ found = DashboardDAO.find_by_id(
+ str(dashboard.uuid), id_column="uuid", skip_base_filter=True
+ )
+ assert found is not None
+ assert found.uuid == dashboard.uuid
+ assert found.dashboard_title == "Test UUID Dashboard"
+
+ # Find by numeric id (should still work)
+ found_by_id = DashboardDAO.find_by_id(dashboard.id, skip_base_filter=True)
+ assert found_by_id is not None
+ assert found_by_id.id == dashboard.id
+
+ # Test with non-existent uuid
+ not_found = DashboardDAO.find_by_id(str(uuid.uuid4()), skip_base_filter=True)
+ assert not_found is None
+
+
+def test_find_by_id_with_slug_column(app_context: Session) -> None:
+ """Test find_by_id with slug column fallback."""
+ # Create a dashboard with slug
+ dashboard = Dashboard(
+ dashboard_title="Test Slug Dashboard",
+ slug="test-slug-dashboard",
+ published=True,
+ )
+ db.session.add(dashboard)
+ db.session.commit()
+
+ # Find by slug using the slug column
+ found = DashboardDAO.find_by_id(
+ "test-slug-dashboard", id_column="slug", skip_base_filter=True
+ )
+ assert found is not None
+ assert found.slug == "test-slug-dashboard"
+ assert found.dashboard_title == "Test Slug Dashboard"
+
+ # Test with non-existent slug
+ not_found = DashboardDAO.find_by_id("non-existent-slug", skip_base_filter=True)
+ assert not_found is None
+
+
+def test_find_by_id_with_invalid_column(app_context: Session) -> None:
+ """Test find_by_id returns None when column doesn't exist."""
+ # This should return None gracefully
+ result = UserDAO.find_by_id("not_a_valid_id", skip_base_filter=True)
+ assert result is None
+
+
+def test_find_by_id_skip_base_filter(app_context: Session) -> None:
+ """Test find_by_id with skip_base_filter parameter."""
+ # Create users with different active states
+ active_user = User(
+ username="active_user",
+ first_name="Active",
+ last_name="User",
+ email="active@example.com",
+ active=True,
+ )
+ inactive_user = User(
+ username="inactive_user",
+ first_name="Inactive",
+ last_name="User",
+ email="inactive@example.com",
+ active=False,
+ )
+ db.session.add_all([active_user, inactive_user])
+ db.session.commit()
+
+ # Without skipping base filter (if one exists)
+ found_active = UserDAO.find_by_id(active_user.id, skip_base_filter=False)
+ assert found_active is not None
+
+ # With skipping base filter
+ found_active_skip = UserDAO.find_by_id(active_user.id, skip_base_filter=True)
+ assert found_active_skip is not None
+
+ # Both should find the user since UserDAO might not have a base filter
+ assert found_active.id == active_user.id
+ assert found_active_skip.id == active_user.id
+
+
+def test_find_by_ids_with_default_column(app_context: Session) -> None:
+ """Test find_by_ids with default 'id' column."""
+ # Create multiple users
+ users = []
+ for i in range(3):
+ user = User(
+ username=f"test_find_by_ids_{i}",
+ first_name=f"Test{i}",
+ last_name="User",
+ email=f"test{i}@example.com",
+ active=True,
+ )
+ users.append(user)
+ db.session.add(user)
+ db.session.commit()
+
+ # Find by multiple ids
+ ids = [user.id for user in users]
+ found = UserDAO.find_by_ids(ids, skip_base_filter=True)
+ assert len(found) == 3
+ found_ids = [u.id for u in found]
+ assert set(found_ids) == set(ids)
+
+ # Test with mix of existent and non-existent ids
+ mixed_ids = [users[0].id, 999999, users[1].id]
+ found_mixed = UserDAO.find_by_ids(mixed_ids, skip_base_filter=True)
+ assert len(found_mixed) == 2
+
+ # Test with empty list
+ found_empty = UserDAO.find_by_ids([], skip_base_filter=True)
+ assert found_empty == []
+
+
+def test_find_by_ids_with_uuid_column(app_context: Session) -> None:
+ """Test find_by_ids with uuid column."""
+ # Create multiple dashboards
+ dashboards = []
+ for i in range(3):
+ dashboard = Dashboard(
+ dashboard_title=f"Test UUID Dashboard {i}",
+ slug=f"test-uuid-dashboard-{i}",
+ published=True,
+ )
+ dashboards.append(dashboard)
+ db.session.add(dashboard)
+ db.session.commit()
+
+ # Find by multiple uuids
+ uuids = [str(dashboard.uuid) for dashboard in dashboards]
+ found = DashboardDAO.find_by_ids(uuids, id_column="uuid", skip_base_filter=True)
+ assert len(found) == 3
+ found_uuids = [str(d.uuid) for d in found]
+ assert set(found_uuids) == set(uuids)
+
+ # Test with mix of ids and uuids - search separately by column
+ found_by_id = DashboardDAO.find_by_ids([dashboards[0].id], skip_base_filter=True)
+ found_by_uuid = DashboardDAO.find_by_ids(
+ [str(dashboards[1].uuid)], id_column="uuid", skip_base_filter=True
+ )
+ assert len(found_by_id) == 1
+ assert len(found_by_uuid) == 1
+
+
+def test_find_by_ids_with_slug_column(app_context: Session) -> None:
+ """Test find_by_ids with slug column."""
+ # Create multiple dashboards
+ dashboards = []
+ for i in range(3):
+ dashboard = Dashboard(
+ dashboard_title=f"Test Slug Dashboard {i}",
+ slug=f"test-slug-dashboard-{i}",
+ published=True,
+ )
+ dashboards.append(dashboard)
+ db.session.add(dashboard)
+ db.session.commit()
+
+ # Find by multiple slugs
+ slugs = [dashboard.slug for dashboard in dashboards]
+ found = DashboardDAO.find_by_ids(slugs, id_column="slug", skip_base_filter=True)
+ assert len(found) == 3
+ found_slugs = [d.slug for d in found]
+ assert set(found_slugs) == set(slugs)
+
+
+def test_find_by_ids_with_invalid_column(app_context: Session) -> None:
+ """Test find_by_ids returns empty list when column doesn't exist."""
+ # This should return empty list gracefully
+ result = UserDAO.find_by_ids(["not_a_valid_id"], skip_base_filter=True)
+ assert result == []
+
+
+def test_find_by_ids_skip_base_filter(app_context: Session) -> None:
+ """Test find_by_ids with skip_base_filter parameter."""
+ # Create users
+ users = []
+ for i in range(3):
+ user = User(
+ username=f"test_skip_filter_{i}",
+ first_name=f"Test{i}",
+ last_name="User",
+ email=f"test{i}@example.com",
+ active=True,
+ )
+ users.append(user)
+ db.session.add(user)
+ db.session.commit()
+
+ ids = [user.id for user in users]
+
+ # Without skipping base filter
+ found_no_skip = UserDAO.find_by_ids(ids, skip_base_filter=False)
+ assert len(found_no_skip) == 3
+
+ # With skipping base filter
+ found_skip = UserDAO.find_by_ids(ids, skip_base_filter=True)
+ assert len(found_skip) == 3
+
+
+def test_base_dao_create_with_item(app_context: Session) -> None:
+ """Test BaseDAO.create with an item parameter."""
+ # Create a user item
+ user = User(
+ username="created_with_item",
+ first_name="Created",
+ last_name="Item",
+ email="created@example.com",
+ active=True,
+ )
+
+ # Create using the item
+ created = UserDAO.create(item=user)
+ assert created is not None
+ assert created.username == "created_with_item"
+ assert created.first_name == "Created"
+
+ # Verify it's in the session
+ assert created in db.session
+
+ # Commit and verify it persists
+ db.session.commit()
+
+ # Find it again to ensure it was saved
+ found = UserDAO.find_by_id(created.id, skip_base_filter=True)
+ assert found is not None
+ assert found.username == "created_with_item"
+
+
+def test_base_dao_create_with_attributes(app_context: Session) -> None:
+ """Test BaseDAO.create with attributes parameter."""
+ # Create using attributes dict
+ attributes = {
+ "username": "created_with_attrs",
+ "first_name": "Created",
+ "last_name": "Attrs",
+ "email": "attrs@example.com",
+ "active": True,
+ }
+
+ created = UserDAO.create(attributes=attributes)
+ assert created is not None
+ assert created.username == "created_with_attrs"
+ assert created.email == "attrs@example.com"
+
+ # Commit and verify
+ db.session.commit()
+ found = UserDAO.find_by_id(created.id, skip_base_filter=True)
+ assert found is not None
+ assert found.username == "created_with_attrs"
+
+
+def test_base_dao_create_with_both_item_and_attributes(app_context: Session) -> None:
+ """Test BaseDAO.create with both item and attributes (override behavior)."""
+ # Create a user item
+ user = User(
+ username="item_username",
+ first_name="Item",
+ last_name="User",
+ email="item@example.com",
+ active=False,
+ )
+
+ # Override some attributes
+ attributes = {
+ "username": "override_username",
+ "active": True,
+ }
+
+ created = UserDAO.create(item=user, attributes=attributes)
+ assert created is not None
+ assert created.username == "override_username" # Should be overridden
+ assert created.active is True # Should be overridden
+ assert created.first_name == "Item" # Should keep original
+ assert created.last_name == "User" # Should keep original
+
+ db.session.commit()
+
+
+def test_base_dao_update_with_item(app_context: Session) -> None:
+ """Test BaseDAO.update with an item parameter."""
+ # Create a user first
+ user = User(
+ username="update_test",
+ first_name="Original",
+ last_name="User",
+ email="original@example.com",
+ active=True,
+ )
+ db.session.add(user)
+ db.session.commit()
+
+ # Update the user
+ user.first_name = "Updated"
+ updated = UserDAO.update(item=user)
+ assert updated is not None
+ assert updated.first_name == "Updated"
+
+ db.session.commit()
+
+ # Verify the update persisted
+ found = UserDAO.find_by_id(user.id, skip_base_filter=True)
+ assert found is not None
+ assert found.first_name == "Updated"
+
+
+def test_base_dao_update_with_attributes(app_context: Session) -> None:
+ """Test BaseDAO.update with attributes parameter."""
+ # Create a user first
+ user = User(
+ username="update_attrs_test",
+ first_name="Original",
+ last_name="User",
+ email="original@example.com",
+ active=True,
+ )
+ db.session.add(user)
+ db.session.commit()
+
+ # Update using attributes
+ attributes = {"first_name": "Updated", "last_name": "Attr User"}
+ updated = UserDAO.update(item=user, attributes=attributes)
+ assert updated is not None
+ assert updated.first_name == "Updated"
+ assert updated.last_name == "Attr User"
+
+ db.session.commit()
+
+
+def test_base_dao_update_detached_item(app_context: Session) -> None:
+ """Test BaseDAO.update with a detached item."""
+ # Create a user first
+ user = User(
+ username="detached_test",
+ first_name="Original",
+ last_name="User",
+ email="detached@example.com",
+ active=True,
+ )
+ db.session.add(user)
+ db.session.commit()
+
+ user_id = user.id
+
+ # Expunge to detach from session
+ db.session.expunge(user)
Review Comment:
1. Real-world scenario: In web applications, objects often become
detached between requests. For example:
- Object retrieved in one request, modified in another
- Objects stored in cache/session and later updated
- Background tasks that operate on objects outside the request
context
2. SQLAlchemy complexity: Detached objects are a common source of bugs
with SQLAlchemy. Without proper handling, you get errors like:
- DetachedInstanceError
- InvalidRequestError: Object is not bound to a Session
3. API contract: This test documents that BaseDAO.update() explicitly
supports detached objects. Removing the test would mean we're no longer
guaranteeing this behavior, which could break existing code that
relies on it.
4. Defensive programming: The fact that someone wrote this test
suggests they either:
- Hit this issue in production and fixed it
- Proactively ensured the DAO layer is robust
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat: Add BaseDAO improvements and test reorganization [superset]
Posted by "aminghadersohi (via GitHub)" <gi...@apache.org>.
aminghadersohi commented on code in PR #35018:
URL: https://github.com/apache/superset/pull/35018#discussion_r2334764245
##########
tests/integration_tests/dao/base_dao_test.py:
##########
@@ -0,0 +1,1397 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""
+Integration tests for BaseDAO functionality.
+
+This module contains comprehensive integration tests for the BaseDAO class and its
+subclasses, covering database operations, CRUD methods, flexible column support,
+column operators, and error handling.
+
+Tests use an in-memory SQLite database for isolation and to replicate the unit test
+environment behavior. User model deletions are avoided due to circular dependency
+constraints with self-referential foreign keys.
+"""
+
+import datetime
+import time
+import uuid
+
+import pytest
+from flask_appbuilder.models.filters import BaseFilter
+from flask_appbuilder.security.sqla.models import User
+from sqlalchemy import Column, DateTime, Integer, String
+from sqlalchemy.ext.declarative import declarative_base
+from sqlalchemy.orm.session import Session
+
+from superset.daos.base import BaseDAO, ColumnOperator, ColumnOperatorEnum
+from superset.daos.chart import ChartDAO
+from superset.daos.dashboard import DashboardDAO
+from superset.daos.database import DatabaseDAO
+from superset.daos.user import UserDAO
+from superset.extensions import db
+from superset.models.core import Database
+from superset.models.dashboard import Dashboard
+from superset.models.slice import Slice
+
+# Create a test model for comprehensive testing
+Base = declarative_base()
+
+
+class ExampleModel(Base): # type: ignore
+ __tablename__ = "example_model"
+ id = Column(Integer, primary_key=True)
+ uuid = Column(String(36), unique=True, nullable=False)
+ slug = Column(String(100), unique=True)
+ name = Column(String(100))
+ code = Column(String(50), unique=True)
+ created_on = Column(DateTime, default=datetime.datetime.utcnow)
+
+
+class ExampleModelDAO(BaseDAO[ExampleModel]):
+ model_cls = ExampleModel
+ id_column_name = "id"
+ base_filter = None
+
+
+class MockModel:
+ def __init__(self, id=1, name="test"):
+ self.id = id
+ self.name = name
+
+
+class TestDAO(BaseDAO[MockModel]):
+ model_cls = MockModel
+
+
+@pytest.fixture(autouse=True)
+def mock_g_user(app_context):
+ """Mock the flask g.user for security context."""
+ # Within app context, we can safely mock g
+ from flask import g
+
+ mock_user = User()
+ mock_user.id = 1
+ mock_user.username = "test_user"
+
+ # Set g.user directly instead of patching
+ g.user = mock_user
+ yield
+
+ # Clean up
+ if hasattr(g, "user"):
+ delattr(g, "user")
+
+
+# =============================================================================
+# Integration Tests - These tests use the actual database
+# =============================================================================
+
+
+def test_column_operator_enum_complete_coverage(user_with_data: Session) -> None:
+ """
+ Test that every single ColumnOperatorEnum operator is covered by tests.
+ This ensures we have comprehensive test coverage for all operators.
+ """
+ # Simply verify that we can create queries with all operators
+ for operator in ColumnOperatorEnum:
+ column_operator = ColumnOperator(
+ col="username", opr=operator, value="test_value"
+ )
+ # Just check it doesn't raise an error
+ assert column_operator.opr == operator
+
+
+def test_find_by_id_with_default_column(app_context: Session) -> None:
+ """Test find_by_id with default 'id' column."""
+ # Create a user to test with
+ user = User(
+ username="test_find_by_id",
+ first_name="Test",
+ last_name="User",
+ email="test@example.com",
+ active=True,
+ )
+ db.session.add(user)
+ db.session.commit()
+
+ # Find by numeric id
+ found = UserDAO.find_by_id(user.id, skip_base_filter=True)
+ assert found is not None
+ assert found.id == user.id
+ assert found.username == "test_find_by_id"
+
+ # Test with non-existent id
+ not_found = UserDAO.find_by_id(999999, skip_base_filter=True)
+ assert not_found is None
+
+
+def test_find_by_id_with_uuid_column(app_context: Session) -> None:
+ """Test find_by_id with custom uuid column."""
+ # Create a dashboard with uuid
+ dashboard = Dashboard(
+ dashboard_title="Test UUID Dashboard",
+ slug="test-uuid-dashboard",
+ published=True,
+ )
+ db.session.add(dashboard)
+ db.session.commit()
+
+ # Find by uuid string using the uuid column
+ found = DashboardDAO.find_by_id(
+ str(dashboard.uuid), id_column="uuid", skip_base_filter=True
+ )
+ assert found is not None
+ assert found.uuid == dashboard.uuid
+ assert found.dashboard_title == "Test UUID Dashboard"
+
+ # Find by numeric id (should still work)
+ found_by_id = DashboardDAO.find_by_id(dashboard.id, skip_base_filter=True)
+ assert found_by_id is not None
+ assert found_by_id.id == dashboard.id
+
+ # Test with non-existent uuid
+ not_found = DashboardDAO.find_by_id(str(uuid.uuid4()), skip_base_filter=True)
+ assert not_found is None
+
+
+def test_find_by_id_with_slug_column(app_context: Session) -> None:
+ """Test find_by_id with slug column fallback."""
+ # Create a dashboard with slug
+ dashboard = Dashboard(
+ dashboard_title="Test Slug Dashboard",
+ slug="test-slug-dashboard",
+ published=True,
+ )
+ db.session.add(dashboard)
+ db.session.commit()
+
+ # Find by slug using the slug column
+ found = DashboardDAO.find_by_id(
+ "test-slug-dashboard", id_column="slug", skip_base_filter=True
+ )
+ assert found is not None
+ assert found.slug == "test-slug-dashboard"
+ assert found.dashboard_title == "Test Slug Dashboard"
+
+ # Test with non-existent slug
+ not_found = DashboardDAO.find_by_id("non-existent-slug", skip_base_filter=True)
+ assert not_found is None
+
+
+def test_find_by_id_with_invalid_column(app_context: Session) -> None:
+ """Test find_by_id returns None when column doesn't exist."""
+ # This should return None gracefully
+ result = UserDAO.find_by_id("not_a_valid_id", skip_base_filter=True)
+ assert result is None
+
+
+def test_find_by_id_skip_base_filter(app_context: Session) -> None:
+ """Test find_by_id with skip_base_filter parameter."""
+ # Create users with different active states
+ active_user = User(
+ username="active_user",
+ first_name="Active",
+ last_name="User",
+ email="active@example.com",
+ active=True,
+ )
+ inactive_user = User(
+ username="inactive_user",
+ first_name="Inactive",
+ last_name="User",
+ email="inactive@example.com",
+ active=False,
+ )
+ db.session.add_all([active_user, inactive_user])
+ db.session.commit()
+
+ # Without skipping base filter (if one exists)
+ found_active = UserDAO.find_by_id(active_user.id, skip_base_filter=False)
+ assert found_active is not None
+
+ # With skipping base filter
+ found_active_skip = UserDAO.find_by_id(active_user.id, skip_base_filter=True)
+ assert found_active_skip is not None
+
+ # Both should find the user since UserDAO might not have a base filter
+ assert found_active.id == active_user.id
+ assert found_active_skip.id == active_user.id
+
+
+def test_find_by_ids_with_default_column(app_context: Session) -> None:
+ """Test find_by_ids with default 'id' column."""
+ # Create multiple users
+ users = []
+ for i in range(3):
+ user = User(
+ username=f"test_find_by_ids_{i}",
+ first_name=f"Test{i}",
+ last_name="User",
+ email=f"test{i}@example.com",
+ active=True,
+ )
+ users.append(user)
+ db.session.add(user)
+ db.session.commit()
+
+ # Find by multiple ids
+ ids = [user.id for user in users]
+ found = UserDAO.find_by_ids(ids, skip_base_filter=True)
+ assert len(found) == 3
+ found_ids = [u.id for u in found]
+ assert set(found_ids) == set(ids)
+
+ # Test with mix of existent and non-existent ids
+ mixed_ids = [users[0].id, 999999, users[1].id]
+ found_mixed = UserDAO.find_by_ids(mixed_ids, skip_base_filter=True)
+ assert len(found_mixed) == 2
+
+ # Test with empty list
+ found_empty = UserDAO.find_by_ids([], skip_base_filter=True)
+ assert found_empty == []
+
+
+def test_find_by_ids_with_uuid_column(app_context: Session) -> None:
+ """Test find_by_ids with uuid column."""
+ # Create multiple dashboards
+ dashboards = []
+ for i in range(3):
+ dashboard = Dashboard(
+ dashboard_title=f"Test UUID Dashboard {i}",
+ slug=f"test-uuid-dashboard-{i}",
+ published=True,
+ )
+ dashboards.append(dashboard)
+ db.session.add(dashboard)
+ db.session.commit()
+
+ # Find by multiple uuids
+ uuids = [str(dashboard.uuid) for dashboard in dashboards]
+ found = DashboardDAO.find_by_ids(uuids, id_column="uuid", skip_base_filter=True)
+ assert len(found) == 3
+ found_uuids = [str(d.uuid) for d in found]
+ assert set(found_uuids) == set(uuids)
+
+ # Test with mix of ids and uuids - search separately by column
+ found_by_id = DashboardDAO.find_by_ids([dashboards[0].id], skip_base_filter=True)
+ found_by_uuid = DashboardDAO.find_by_ids(
+ [str(dashboards[1].uuid)], id_column="uuid", skip_base_filter=True
+ )
+ assert len(found_by_id) == 1
+ assert len(found_by_uuid) == 1
+
+
+def test_find_by_ids_with_slug_column(app_context: Session) -> None:
+ """Test find_by_ids with slug column."""
+ # Create multiple dashboards
+ dashboards = []
+ for i in range(3):
+ dashboard = Dashboard(
+ dashboard_title=f"Test Slug Dashboard {i}",
+ slug=f"test-slug-dashboard-{i}",
+ published=True,
+ )
+ dashboards.append(dashboard)
+ db.session.add(dashboard)
+ db.session.commit()
+
+ # Find by multiple slugs
+ slugs = [dashboard.slug for dashboard in dashboards]
+ found = DashboardDAO.find_by_ids(slugs, id_column="slug", skip_base_filter=True)
+ assert len(found) == 3
+ found_slugs = [d.slug for d in found]
+ assert set(found_slugs) == set(slugs)
+
+
+def test_find_by_ids_with_invalid_column(app_context: Session) -> None:
+ """Test find_by_ids returns empty list when column doesn't exist."""
+ # This should return empty list gracefully
+ result = UserDAO.find_by_ids(["not_a_valid_id"], skip_base_filter=True)
+ assert result == []
+
+
+def test_find_by_ids_skip_base_filter(app_context: Session) -> None:
+ """Test find_by_ids with skip_base_filter parameter."""
+ # Create users
+ users = []
+ for i in range(3):
+ user = User(
+ username=f"test_skip_filter_{i}",
+ first_name=f"Test{i}",
+ last_name="User",
+ email=f"test{i}@example.com",
+ active=True,
+ )
+ users.append(user)
+ db.session.add(user)
+ db.session.commit()
+
+ ids = [user.id for user in users]
+
+ # Without skipping base filter
+ found_no_skip = UserDAO.find_by_ids(ids, skip_base_filter=False)
+ assert len(found_no_skip) == 3
+
+ # With skipping base filter
+ found_skip = UserDAO.find_by_ids(ids, skip_base_filter=True)
+ assert len(found_skip) == 3
+
+
+def test_base_dao_create_with_item(app_context: Session) -> None:
+ """Test BaseDAO.create with an item parameter."""
+ # Create a user item
+ user = User(
+ username="created_with_item",
+ first_name="Created",
+ last_name="Item",
+ email="created@example.com",
+ active=True,
+ )
+
+ # Create using the item
+ created = UserDAO.create(item=user)
+ assert created is not None
+ assert created.username == "created_with_item"
+ assert created.first_name == "Created"
+
+ # Verify it's in the session
+ assert created in db.session
+
+ # Commit and verify it persists
+ db.session.commit()
+
+ # Find it again to ensure it was saved
+ found = UserDAO.find_by_id(created.id, skip_base_filter=True)
+ assert found is not None
+ assert found.username == "created_with_item"
+
+
+def test_base_dao_create_with_attributes(app_context: Session) -> None:
+ """Test BaseDAO.create with attributes parameter."""
+ # Create using attributes dict
+ attributes = {
+ "username": "created_with_attrs",
+ "first_name": "Created",
+ "last_name": "Attrs",
+ "email": "attrs@example.com",
+ "active": True,
+ }
+
+ created = UserDAO.create(attributes=attributes)
+ assert created is not None
+ assert created.username == "created_with_attrs"
+ assert created.email == "attrs@example.com"
+
+ # Commit and verify
+ db.session.commit()
+ found = UserDAO.find_by_id(created.id, skip_base_filter=True)
+ assert found is not None
+ assert found.username == "created_with_attrs"
+
+
+def test_base_dao_create_with_both_item_and_attributes(app_context: Session) -> None:
+ """Test BaseDAO.create with both item and attributes (override behavior)."""
+ # Create a user item
+ user = User(
+ username="item_username",
+ first_name="Item",
+ last_name="User",
+ email="item@example.com",
+ active=False,
+ )
+
+ # Override some attributes
+ attributes = {
+ "username": "override_username",
+ "active": True,
+ }
+
+ created = UserDAO.create(item=user, attributes=attributes)
+ assert created is not None
+ assert created.username == "override_username" # Should be overridden
+ assert created.active is True # Should be overridden
+ assert created.first_name == "Item" # Should keep original
+ assert created.last_name == "User" # Should keep original
+
+ db.session.commit()
+
+
+def test_base_dao_update_with_item(app_context: Session) -> None:
+ """Test BaseDAO.update with an item parameter."""
+ # Create a user first
+ user = User(
+ username="update_test",
+ first_name="Original",
+ last_name="User",
+ email="original@example.com",
+ active=True,
+ )
+ db.session.add(user)
+ db.session.commit()
+
+ # Update the user
+ user.first_name = "Updated"
+ updated = UserDAO.update(item=user)
+ assert updated is not None
+ assert updated.first_name == "Updated"
+
+ db.session.commit()
+
+ # Verify the update persisted
+ found = UserDAO.find_by_id(user.id, skip_base_filter=True)
+ assert found is not None
+ assert found.first_name == "Updated"
+
+
+def test_base_dao_update_with_attributes(app_context: Session) -> None:
+ """Test BaseDAO.update with attributes parameter."""
+ # Create a user first
+ user = User(
+ username="update_attrs_test",
+ first_name="Original",
+ last_name="User",
+ email="original@example.com",
+ active=True,
+ )
+ db.session.add(user)
+ db.session.commit()
+
+ # Update using attributes
+ attributes = {"first_name": "Updated", "last_name": "Attr User"}
+ updated = UserDAO.update(item=user, attributes=attributes)
+ assert updated is not None
+ assert updated.first_name == "Updated"
+ assert updated.last_name == "Attr User"
+
+ db.session.commit()
+
+
+def test_base_dao_update_detached_item(app_context: Session) -> None:
+ """Test BaseDAO.update with a detached item."""
+ # Create a user first
+ user = User(
+ username="detached_test",
+ first_name="Original",
+ last_name="User",
+ email="detached@example.com",
+ active=True,
+ )
+ db.session.add(user)
+ db.session.commit()
+
+ user_id = user.id
+
+ # Expunge to detach from session
+ db.session.expunge(user)
Review Comment:
if i recall correctly, i ran into some issues with detached objects while working on the mcp service and got claude code to write this test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat: Add BaseDAO improvements and test reorganization [superset]
Posted by "aminghadersohi (via GitHub)" <gi...@apache.org>.
aminghadersohi commented on code in PR #35018:
URL: https://github.com/apache/superset/pull/35018#discussion_r2334689726
##########
superset/daos/base.py:
##########
@@ -251,3 +455,205 @@ def filter_by(cls, **filter_by: Any) -> list[T]:
cls.id_column_name, data_model
).apply(query, None)
return query.filter_by(**filter_by).all()
+
+ @classmethod
+ def apply_column_operators(
+ cls, query: Any, column_operators: Optional[List[ColumnOperator]] = None
+ ) -> Any:
+ """
+ Apply column operators (list of ColumnOperator) to the query using
+ ColumnOperatorEnum logic. Raises ValueError if a filter references a
+ non-existent column.
+ """
+ if not column_operators:
+ return query
+ for c in column_operators:
+ if not isinstance(c, ColumnOperator):
+ continue
+ col = c.col
+ opr = c.opr
+ value = c.value
+ if not col or not hasattr(cls.model_cls, col):
+ model_name = cls.model_cls.__name__ if cls.model_cls else "Unknown"
+ logging.error(
+ f"Invalid filter: column '{col}' does not exist on {model_name}"
+ )
+ raise ValueError(
+ f"Invalid filter: column '{col}' does not exist on {model_name}"
+ )
+ column = getattr(cls.model_cls, col)
+ try:
+ # Always use ColumnOperatorEnum's apply method
+ operator_enum = ColumnOperatorEnum(opr)
+ query = query.filter(operator_enum.apply(column, value))
+ except Exception as e:
+ logging.error(f"Error applying filter on column '{col}': {e}")
+ raise
+ return query
+
+ @classmethod
+ def get_filterable_columns_and_operators(cls) -> Dict[str, List[str]]:
+ """
+ Returns a dict mapping filterable columns (including hybrid/computed fields if
+ present) to their supported operators. Used by MCP tools to dynamically expose
+ filter options. Custom fields supported by the DAO but not present on the model
+ should be documented here.
+ """
+
+ mapper = inspect(cls.model_cls)
+ columns = {c.key: c for c in mapper.columns}
+ # Add hybrid properties
+ hybrids = {
+ name: attr
+ for name, attr in vars(cls.model_cls).items()
+ if isinstance(attr, hybrid_property)
+ }
+ # You may add custom fields here, e.g.:
+ # custom_fields = {"tags": ["eq", "in_", "like"], ...}
+ custom_fields: Dict[str, List[str]] = {}
+
+ filterable = {}
+ for name, col in columns.items():
+ if isinstance(col.type, (sa.String, sa.Text)):
+ filterable[name] = TYPE_OPERATOR_MAP["string"]
+ elif isinstance(col.type, (sa.Boolean,)):
+ filterable[name] = TYPE_OPERATOR_MAP["boolean"]
+ elif isinstance(col.type, (sa.Integer, sa.Float, sa.Numeric)):
+ filterable[name] = TYPE_OPERATOR_MAP["number"]
+ elif isinstance(col.type, (sa.DateTime, sa.Date, sa.Time)):
+ filterable[name] = TYPE_OPERATOR_MAP["datetime"]
+ else:
+ # Fallback to eq/ne/null
+ filterable[name] = ["eq", "ne", "is_null", "is_not_null"]
+ # Add hybrid properties as string fields by default
+ for name in hybrids:
+ filterable[name] = TYPE_OPERATOR_MAP["string"]
+ # Add custom fields
+ filterable.update(custom_fields)
+ return filterable
+
+ @classmethod
+ def _build_query(
+ cls,
+ column_operators: Optional[List[ColumnOperator]] = None,
+ search: Optional[str] = None,
+ search_columns: Optional[List[str]] = None,
+ custom_filters: Optional[Dict[str, BaseFilter]] = None,
+ skip_base_filter: bool = False,
+ data_model: Optional[SQLAInterface] = None,
+ ) -> Any:
+ """
+ Build a SQLAlchemy query with base filter, column operators, search, and
+ custom filters.
+ """
+ if data_model is None:
+ data_model = SQLAInterface(cls.model_cls, db.session)
+ query = data_model.session.query(cls.model_cls)
+ query = cls._apply_base_filter(
+ query, skip_base_filter=skip_base_filter, data_model=data_model
+ )
+ if search and search_columns:
+ search_filters = []
+ for column_name in search_columns:
+ if hasattr(cls.model_cls, column_name):
+ column = getattr(cls.model_cls, column_name)
+ search_filters.append(cast(column, Text).ilike(f"%{search}%"))
+ if search_filters:
+ query = query.filter(or_(*search_filters))
+ if custom_filters:
+ for filter_class in custom_filters.values():
+ query = filter_class.apply(query, None)
+ if column_operators:
+ query = cls.apply_column_operators(query, column_operators)
+ return query
+
+ @classmethod
+ def list( # noqa: C901
+ cls,
+ column_operators: Optional[List[ColumnOperator]] = None,
+ order_column: str = "changed_on",
+ order_direction: str = "desc",
+ page: int = 0,
+ page_size: int = 100,
+ search: Optional[str] = None,
+ search_columns: Optional[List[str]] = None,
+ custom_filters: Optional[Dict[str, BaseFilter]] = None,
+ columns: Optional[List[str]] = None,
+ ) -> Tuple[List[Any], int]:
+ """
+ Generic list method for filtered, sorted, and paginated results.
+ If columns is specified, returns a list of tuples (one per row),
+ otherwise returns model instances.
+ """
+ data_model = SQLAInterface(cls.model_cls, db.session)
+
+ column_attrs = []
+ relationship_loads = []
+ if columns is None:
+ columns = []
+ for name in columns:
+ attr = getattr(cls.model_cls, name, None)
+ if attr is None:
+ continue
+ prop = getattr(attr, "property", None)
+ if isinstance(prop, ColumnProperty):
+ column_attrs.append(attr)
+ elif isinstance(prop, RelationshipProperty):
+ relationship_loads.append(joinedload(attr))
Review Comment:
Good catch! You're right - `joinedload` with many-to-many relationships
would break pagination. I've fixed it by counting before adding the
relationship joins. This ensures:
- Accurate counts (not inflated by joins)
- Correct pagination (LIMIT/OFFSET on entities, not joined rows)
We don't have tests for relationship loading in the list method - that
should be added to prevent regression.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] Feat/dao updates [superset]
Posted by "mistercrunch (via GitHub)" <gi...@apache.org>.
mistercrunch commented on code in PR #35018:
URL: https://github.com/apache/superset/pull/35018#discussion_r2326356095
##########
superset/daos/chart.py:
##########
@@ -39,6 +39,20 @@
class ChartDAO(BaseDAO[Slice]):
base_filter = ChartFilter
+ @classmethod
+ def get_filterable_columns_and_operators(cls) -> Dict[str, List[str]]:
+ filterable = super().get_filterable_columns_and_operators()
+ # Add custom fields for charts
+ filterable.update(
+ {
Review Comment:
NIT: also could fit as a module constant, this way it's not regenerated at every call
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] Feat/dao updates [superset]
Posted by "bito-code-review[bot] (via GitHub)" <gi...@apache.org>.
bito-code-review[bot] commented on PR #35018:
URL: https://github.com/apache/superset/pull/35018#issuecomment-3259456168
**Bito Automatic Review Skipped - Draft PR**<br/><br/>Bito didn't auto-review because this pull request is in draft status.<br/>No action is needed if you didn't intend for the agent to review it. Otherwise, to manually trigger a review, type `/review` in a comment and save.<br/>You can change draft PR review settings [here](https://alpha.bito.ai/home/ai-agents/code-review-agent), or contact your Bito workspace admin at [evan@preset.io](mailto:evan@preset.io).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat: Add BaseDAO improvements and test reorganization [superset]
Posted by "aminghadersohi (via GitHub)" <gi...@apache.org>.
aminghadersohi commented on code in PR #35018:
URL: https://github.com/apache/superset/pull/35018#discussion_r2334351225
##########
superset/daos/base.py:
##########
@@ -83,39 +196,138 @@ def find_by_id_or_uuid(
return None
@classmethod
- def find_by_id(
- cls,
- model_id: str | int,
- skip_base_filter: bool = False,
- ) -> T | None:
+ def _apply_base_filter(
+ cls, query: Any, skip_base_filter: bool = False, data_model: Any = None
+ ) -> Any:
"""
- Find a model by id, if defined applies `base_filter`
+ Apply the base_filter to the query if it exists and skip_base_filter is False.
"""
- query = db.session.query(cls.model_cls)
if cls.base_filter and not skip_base_filter:
- data_model = SQLAInterface(cls.model_cls, db.session)
+ if data_model is None:
+ data_model = SQLAInterface(cls.model_cls, db.session)
query = cls.base_filter( # pylint: disable=not-callable
cls.id_column_name, data_model
).apply(query, None)
- id_column = getattr(cls.model_cls, cls.id_column_name)
+ return query
+
+ @classmethod
+ def _convert_value_for_column(cls, column: Any, value: Any) -> Any:
+ """
+ Convert a value to the appropriate type for a given SQLAlchemy column.
+
+ Args:
+ column: SQLAlchemy column object
+ value: Value to convert
+
+ Returns:
+ Converted value or None if conversion fails
+ """
+ if (
+ hasattr(column.type, "python_type")
+ and column.type.python_type == uuid_lib.UUID
+ ):
+ if isinstance(value, str):
+ try:
+ return uuid_lib.UUID(value)
+ except (ValueError, AttributeError):
+ return None
+ return value
+
+ @classmethod
+ def _find_by_column(
+ cls,
+ column_name: str,
+ value: str | int,
+ skip_base_filter: bool = False,
+ ) -> T | None:
+ """
+ Private method to find a model by any column value.
+
+ Args:
+ column_name: Name of the column to search by
+ value: Value to search for
+ skip_base_filter: Whether to skip base filtering
+
+ Returns:
+ Model instance or None if not found
+ """
+ query = db.session.query(cls.model_cls)
+ query = cls._apply_base_filter(query, skip_base_filter)
+
+ if not hasattr(cls.model_cls, column_name):
+ return None
+
+ column = getattr(cls.model_cls, column_name)
+ converted_value = cls._convert_value_for_column(column, value)
+ if converted_value is None:
+ return None
+
try:
- return query.filter(id_column == model_id).one_or_none()
+ return query.filter(column == converted_value).one_or_none()
except StatementError:
# can happen if int is passed instead of a string or similar
return None
+ @classmethod
+ def find_by_id(
+ cls,
+ model_id: str | int,
+ skip_base_filter: bool = False,
+ id_column: str | None = None,
+ ) -> T | None:
+ """
+ Find a model by ID using specified or default ID column.
+
+ Args:
+ model_id: ID value to search for
+ skip_base_filter: Whether to skip base filtering
+ id_column: Column name to use (defaults to cls.id_column_name)
+
+ Returns:
+ Model instance or None if not found
+ """
+ column = id_column or cls.id_column_name
+ return cls._find_by_column(column, model_id, skip_base_filter)
+
@classmethod
def find_by_ids(
cls,
- model_ids: list[str] | list[int],
+ model_ids: Sequence[str | int],
skip_base_filter: bool = False,
+ id_column: str | None = None,
) -> list[T]:
"""
Find a List of models by a list of ids, if defined applies `base_filter`
+
+ :param model_ids: List of IDs to find
+ :param skip_base_filter: If true, skip applying the base filter
+ :param id_column: Optional column name to use for ID lookup
+ (defaults to id_column_name)
"""
- id_col = getattr(cls.model_cls, cls.id_column_name, None)
+ column = id_column or cls.id_column_name
+ id_col = getattr(cls.model_cls, column, None)
if id_col is None or not model_ids:
return []
+
+ # Convert IDs to appropriate types based on column type
+ converted_ids: list[str | int | uuid_lib.UUID] = []
+ for id_val in model_ids:
+ converted_value = cls._convert_value_for_column(id_col, id_val)
+ if converted_value is not None:
+ converted_ids.append(converted_value)
Review Comment:
Good catch! You're absolutely right - we should validate type consistency. I'll add a check to ensure all converted IDs are of the same type, with a warning logged for mixed types. This will help catch bugs early while maintaining backward compatibility.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat: Add BaseDAO improvements and test reorganization [superset]
Posted by "mistercrunch (via GitHub)" <gi...@apache.org>.
mistercrunch commented on code in PR #35018:
URL: https://github.com/apache/superset/pull/35018#discussion_r2331702213
##########
requirements/development.in:
##########
@@ -16,5 +16,5 @@
# specific language governing permissions and limitations
# under the License.
#
--e .[development,bigquery,druid,duckdb,gevent,gsheets,mysql,postgres,presto,prophet,trino,thumbnails]
+-e .[development,bigquery,druid,duckdb,gevent,gsheets,mysql,postgres,presto,prophet,pydantic,trino,thumbnails]
Review Comment:
The bot is right here, `pydantic` is not required in this file since it's part of the core dependencies in `pyproject.toml`. The references here are pointing to a set of `project.optional-dependencies` (same as `extra_requires` in old setup.py). Can safely remove `pydantic` and re-run `./scripts/uv-pip-compile.sh` which should have no effect since pydantic is kind of a no-op here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat: Add BaseDAO improvements and test reorganization [superset]
Posted by "korbit-ai[bot] (via GitHub)" <gi...@apache.org>.
korbit-ai[bot] commented on code in PR #35018:
URL: https://github.com/apache/superset/pull/35018#discussion_r2330607807
##########
superset/daos/base.py:
##########
@@ -251,3 +455,205 @@ def filter_by(cls, **filter_by: Any) -> list[T]:
cls.id_column_name, data_model
).apply(query, None)
return query.filter_by(**filter_by).all()
+
+ @classmethod
+ def apply_column_operators(
+ cls, query: Any, column_operators: Optional[List[ColumnOperator]] = None
+ ) -> Any:
+ """
+ Apply column operators (list of ColumnOperator) to the query using
+ ColumnOperatorEnum logic. Raises ValueError if a filter references a
+ non-existent column.
+ """
+ if not column_operators:
+ return query
+ for c in column_operators:
+ if not isinstance(c, ColumnOperator):
+ continue
+ col = c.col
+ opr = c.opr
+ value = c.value
+ if not col or not hasattr(cls.model_cls, col):
+ model_name = cls.model_cls.__name__ if cls.model_cls else "Unknown"
+ logging.error(
+ f"Invalid filter: column '{col}' does not exist on {model_name}"
+ )
+ raise ValueError(
+ f"Invalid filter: column '{col}' does not exist on {model_name}"
+ )
+ column = getattr(cls.model_cls, col)
+ try:
+ # Always use ColumnOperatorEnum's apply method
+ operator_enum = ColumnOperatorEnum(opr)
+ query = query.filter(operator_enum.apply(column, value))
+ except Exception as e:
+ logging.error(f"Error applying filter on column '{col}': {e}")
+ raise
+ return query
+
+ @classmethod
+ def get_filterable_columns_and_operators(cls) -> Dict[str, List[str]]:
+ """
+ Returns a dict mapping filterable columns (including hybrid/computed fields if
+ present) to their supported operators. Used by MCP tools to dynamically expose
+ filter options. Custom fields supported by the DAO but not present on the model
+ should be documented here.
+ """
+
+ mapper = inspect(cls.model_cls)
+ columns = {c.key: c for c in mapper.columns}
+ # Add hybrid properties
+ hybrids = {
+ name: attr
+ for name, attr in vars(cls.model_cls).items()
+ if isinstance(attr, hybrid_property)
+ }
+ # You may add custom fields here, e.g.:
+ # custom_fields = {"tags": ["eq", "in_", "like"], ...}
+ custom_fields: Dict[str, List[str]] = {}
+
+ filterable = {}
+ for name, col in columns.items():
+ if isinstance(col.type, (sa.String, sa.Text)):
+ filterable[name] = TYPE_OPERATOR_MAP["string"]
+ elif isinstance(col.type, (sa.Boolean,)):
+ filterable[name] = TYPE_OPERATOR_MAP["boolean"]
+ elif isinstance(col.type, (sa.Integer, sa.Float, sa.Numeric)):
+ filterable[name] = TYPE_OPERATOR_MAP["number"]
+ elif isinstance(col.type, (sa.DateTime, sa.Date, sa.Time)):
+ filterable[name] = TYPE_OPERATOR_MAP["datetime"]
+ else:
+ # Fallback to eq/ne/null
+ filterable[name] = ["eq", "ne", "is_null", "is_not_null"]
+ # Add hybrid properties as string fields by default
+ for name in hybrids:
+ filterable[name] = TYPE_OPERATOR_MAP["string"]
+ # Add custom fields
+ filterable.update(custom_fields)
+ return filterable
+
+ @classmethod
+ def _build_query(
+ cls,
+ column_operators: Optional[List[ColumnOperator]] = None,
+ search: Optional[str] = None,
+ search_columns: Optional[List[str]] = None,
+ custom_filters: Optional[Dict[str, BaseFilter]] = None,
+ skip_base_filter: bool = False,
+ data_model: Optional[SQLAInterface] = None,
+ ) -> Any:
+ """
+ Build a SQLAlchemy query with base filter, column operators, search, and
+ custom filters.
+ """
+ if data_model is None:
+ data_model = SQLAInterface(cls.model_cls, db.session)
+ query = data_model.session.query(cls.model_cls)
+ query = cls._apply_base_filter(
+ query, skip_base_filter=skip_base_filter, data_model=data_model
+ )
+ if search and search_columns:
+ search_filters = []
+ for column_name in search_columns:
+ if hasattr(cls.model_cls, column_name):
+ column = getattr(cls.model_cls, column_name)
+ search_filters.append(cast(column, Text).ilike(f"%{search}%"))
+ if search_filters:
+ query = query.filter(or_(*search_filters))
+ if custom_filters:
+ for filter_class in custom_filters.values():
+ query = filter_class.apply(query, None)
+ if column_operators:
+ query = cls.apply_column_operators(query, column_operators)
+ return query
+
+ @classmethod
+ def list( # noqa: C901
+ cls,
+ column_operators: Optional[List[ColumnOperator]] = None,
+ order_column: str = "changed_on",
Review Comment:
### Invalid default order column assumption <sub></sub>
<details>
<summary>Tell me more</summary>
###### What is the issue?
The list method assumes 'changed_on' column exists in all models by using it as default order_column
###### Why this matters
If a model doesn't have a 'changed_on' column, the query will raise an AttributeError when trying to order results
###### Suggested change ∙ *Feature Preview*
Change the default order column to use the model's primary key or make it optional:
```python
@classmethod
def list( # noqa: C901
cls,
column_operators: Optional[List[ColumnOperator]] = None,
order_column: Optional[str] = None,
```
###### Provide feedback to improve future suggestions
[](https://app.korbit.ai/feedback/aa91ff46-6083-4491-9416-b83dd1994b51/609016ae-df79-49eb-affc-b50551844803/upvote) [](https://app.korbit.ai/feedback/aa91ff46-6083-4491-9416-b83dd1994b51/609016ae-df79-49eb-affc-b50551844803?what_not_true=true) [](https://app.korbit.ai/feedback/aa91ff46-6083-4491-9416-b83dd1994b51/609016ae-df79-49eb-affc-b50551844803?what_out_of_scope=true) [](https://app.korbit.ai/feedback/aa91ff46-6083-4491-9416-b83dd1994b51/609016ae-df79-49eb-affc-b50551844803?what_not_in_standard=true) [](https://app.korbit.ai/feedback/aa91ff46-6083-4491-9416-b83dd1994b51/609016ae-df79-49eb-affc-b50551844803)
</details>
<sub>
💬 Looking for more details? Reply to this comment to chat with Korbit.
</sub>
<!--- korbi internal id:13f6e983-0064-412b-a140-7f642e23e36a -->
[](13f6e983-0064-412b-a140-7f642e23e36a)
##########
superset/daos/base.py:
##########
@@ -251,3 +455,205 @@ def filter_by(cls, **filter_by: Any) -> list[T]:
cls.id_column_name, data_model
).apply(query, None)
return query.filter_by(**filter_by).all()
+
+ @classmethod
+ def apply_column_operators(
+ cls, query: Any, column_operators: Optional[List[ColumnOperator]] = None
+ ) -> Any:
+ """
+ Apply column operators (list of ColumnOperator) to the query using
+ ColumnOperatorEnum logic. Raises ValueError if a filter references a
+ non-existent column.
+ """
+ if not column_operators:
+ return query
+ for c in column_operators:
+ if not isinstance(c, ColumnOperator):
+ continue
+ col = c.col
+ opr = c.opr
+ value = c.value
+ if not col or not hasattr(cls.model_cls, col):
+ model_name = cls.model_cls.__name__ if cls.model_cls else "Unknown"
+ logging.error(
+ f"Invalid filter: column '{col}' does not exist on {model_name}"
+ )
+ raise ValueError(
+ f"Invalid filter: column '{col}' does not exist on {model_name}"
+ )
+ column = getattr(cls.model_cls, col)
+ try:
+ # Always use ColumnOperatorEnum's apply method
+ operator_enum = ColumnOperatorEnum(opr)
+ query = query.filter(operator_enum.apply(column, value))
+ except Exception as e:
+ logging.error(f"Error applying filter on column '{col}': {e}")
+ raise
+ return query
+
+ @classmethod
+ def get_filterable_columns_and_operators(cls) -> Dict[str, List[str]]:
+ """
+ Returns a dict mapping filterable columns (including hybrid/computed fields if
+ present) to their supported operators. Used by MCP tools to dynamically expose
+ filter options. Custom fields supported by the DAO but not present on the model
+ should be documented here.
+ """
+
+ mapper = inspect(cls.model_cls)
+ columns = {c.key: c for c in mapper.columns}
+ # Add hybrid properties
+ hybrids = {
+ name: attr
+ for name, attr in vars(cls.model_cls).items()
+ if isinstance(attr, hybrid_property)
+ }
+ # You may add custom fields here, e.g.:
+ # custom_fields = {"tags": ["eq", "in_", "like"], ...}
+ custom_fields: Dict[str, List[str]] = {}
+
+ filterable = {}
+ for name, col in columns.items():
+ if isinstance(col.type, (sa.String, sa.Text)):
+ filterable[name] = TYPE_OPERATOR_MAP["string"]
+ elif isinstance(col.type, (sa.Boolean,)):
+ filterable[name] = TYPE_OPERATOR_MAP["boolean"]
+ elif isinstance(col.type, (sa.Integer, sa.Float, sa.Numeric)):
+ filterable[name] = TYPE_OPERATOR_MAP["number"]
+ elif isinstance(col.type, (sa.DateTime, sa.Date, sa.Time)):
+ filterable[name] = TYPE_OPERATOR_MAP["datetime"]
+ else:
+ # Fallback to eq/ne/null
+ filterable[name] = ["eq", "ne", "is_null", "is_not_null"]
+ # Add hybrid properties as string fields by default
+ for name in hybrids:
+ filterable[name] = TYPE_OPERATOR_MAP["string"]
+ # Add custom fields
+ filterable.update(custom_fields)
+ return filterable
+
+ @classmethod
+ def _build_query(
+ cls,
+ column_operators: Optional[List[ColumnOperator]] = None,
+ search: Optional[str] = None,
+ search_columns: Optional[List[str]] = None,
+ custom_filters: Optional[Dict[str, BaseFilter]] = None,
+ skip_base_filter: bool = False,
+ data_model: Optional[SQLAInterface] = None,
+ ) -> Any:
+ """
+ Build a SQLAlchemy query with base filter, column operators, search, and
+ custom filters.
+ """
+ if data_model is None:
+ data_model = SQLAInterface(cls.model_cls, db.session)
+ query = data_model.session.query(cls.model_cls)
+ query = cls._apply_base_filter(
+ query, skip_base_filter=skip_base_filter, data_model=data_model
+ )
+ if search and search_columns:
+ search_filters = []
+ for column_name in search_columns:
+ if hasattr(cls.model_cls, column_name):
+ column = getattr(cls.model_cls, column_name)
+ search_filters.append(cast(column, Text).ilike(f"%{search}%"))
Review Comment:
### Unsanitized Search Input in LIKE Queries <sub></sub>
<details>
<summary>Tell me more</summary>
###### What is the issue?
User-provided search input is directly interpolated into SQL LIKE queries without proper sanitization, making it potentially vulnerable to SQL injection attacks.
###### Why this matters
Malicious users could inject SQL code through the search parameter that could alter the query's behavior or access unauthorized data.
###### Suggested change ∙ *Feature Preview*
Use SQLAlchemy's bind parameters to safely handle the search input:
```python
search_filters.append(cast(column, Text).ilike('%' + search + '%'))
```
###### Provide feedback to improve future suggestions
[](https://app.korbit.ai/feedback/aa91ff46-6083-4491-9416-b83dd1994b51/d8bce617-60bb-43e9-a999-a65eb7b707ee/upvote) [](https://app.korbit.ai/feedback/aa91ff46-6083-4491-9416-b83dd1994b51/d8bce617-60bb-43e9-a999-a65eb7b707ee?what_not_true=true) [](https://app.korbit.ai/feedback/aa91ff46-6083-4491-9416-b83dd1994b51/d8bce617-60bb-43e9-a999-a65eb7b707ee?what_out_of_scope=true) [](https://app.korbit.ai/feedback/aa91ff46-6083-4491-9416-b83dd1994b51/d8bce617-60bb-43e9-a999-a65eb7b707ee?what_not_in_standard=true) [](https://app.korbit.ai/feedback/aa91ff46-6083-4491-9416-b83dd1994b51/d8bce617-60bb-43e9-a999-a65eb7b707ee)
</details>
<sub>
💬 Looking for more details? Reply to this comment to chat with Korbit.
</sub>
<!--- korbi internal id:968f87f8-8059-4293-98d2-fd1bb05195ad -->
[](968f87f8-8059-4293-98d2-fd1bb05195ad)
##########
superset/daos/base.py:
##########
@@ -32,6 +51,100 @@
T = TypeVar("T", bound=Model)
+class ColumnOperatorEnum(str, Enum):
+ eq = "eq"
+ ne = "ne"
+ sw = "sw"
+ ew = "ew"
+ in_ = "in"
+ nin = "nin"
+ gt = "gt"
+ gte = "gte"
+ lt = "lt"
+ lte = "lte"
+ like = "like"
+ ilike = "ilike"
+ is_null = "is_null"
+ is_not_null = "is_not_null"
+
+ @classmethod
+ def operator_map(cls) -> Dict[ColumnOperatorEnum, Any]:
+ return {
+ cls.eq: lambda col, val: col == val,
+ cls.ne: lambda col, val: col != val,
+ cls.sw: lambda col, val: col.like(f"{val}%"),
+ cls.ew: lambda col, val: col.like(f"%{val}"),
+ cls.in_: lambda col, val: col.in_(
+ val if isinstance(val, (list, tuple)) else [val]
+ ),
+ cls.nin: lambda col, val: ~col.in_(
+ val if isinstance(val, (list, tuple)) else [val]
+ ),
+ cls.gt: lambda col, val: col > val,
+ cls.gte: lambda col, val: col >= val,
+ cls.lt: lambda col, val: col < val,
+ cls.lte: lambda col, val: col <= val,
+ cls.like: lambda col, val: col.like(f"%{val}%"),
+ cls.ilike: lambda col, val: col.ilike(f"%{val}%"),
+ cls.is_null: lambda col, _: col.is_(None),
+ cls.is_not_null: lambda col, _: col.isnot(None),
+ }
+
+ def apply(self, column: Any, value: Any) -> Any:
+ op_func = self.operator_map().get(self)
+ if not op_func:
+ raise ValueError(f"Unsupported operator: {self}")
+ return op_func(column, value)
Review Comment:
### Missing value type validation <sub></sub>
<details>
<summary>Tell me more</summary>
###### What is the issue?
The apply method in ColumnOperatorEnum doesn't validate the value type before applying the operator
###### Why this matters
Invalid value types could lead to runtime errors or incorrect SQL queries, especially for operators like 'in_' that expect specific value types
###### Suggested change ∙ *Feature Preview*
Add value type validation before applying the operator:
```python
def apply(self, column: Any, value: Any) -> Any:
op_func = self.operator_map().get(self)
if not op_func:
raise ValueError(f"Unsupported operator: {self}")
if self in (self.in_, self.nin) and not isinstance(value, (list, tuple)):
raise ValueError(f"Operator {self} requires a list or tuple value")
if value is None and self not in (self.is_null, self.is_not_null):
raise ValueError(f"None value not allowed for operator {self}")
return op_func(column, value)
```
###### Provide feedback to improve future suggestions
[](https://app.korbit.ai/feedback/aa91ff46-6083-4491-9416-b83dd1994b51/056ffcbd-a2f7-4005-98ff-a64b4f08abb7/upvote) [](https://app.korbit.ai/feedback/aa91ff46-6083-4491-9416-b83dd1994b51/056ffcbd-a2f7-4005-98ff-a64b4f08abb7?what_not_true=true) [](https://app.korbit.ai/feedback/aa91ff46-6083-4491-9416-b83dd1994b51/056ffcbd-a2f7-4005-98ff-a64b4f08abb7?what_out_of_scope=true) [](https://app.korbit.ai/feedback/aa91ff46-6083-4491-9416-b83dd1994b51/056ffcbd-a2f7-4005-98ff-a64b4f08abb7?what_not_in_standard=true) [](https://app.korbit.ai/feedback/aa91ff46-6083-4491-9416-b83dd1994b51/056ffcbd-a2f7-4005-98ff-a64b4f08abb7)
</details>
<sub>
💬 Looking for more details? Reply to this comment to chat with Korbit.
</sub>
<!--- korbi internal id:d2cf643a-8e34-48c5-826c-152e044b14a0 -->
[](d2cf643a-8e34-48c5-826c-152e044b14a0)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] Feat/dao updates [superset]
Posted by "mistercrunch (via GitHub)" <gi...@apache.org>.
mistercrunch commented on code in PR #35018:
URL: https://github.com/apache/superset/pull/35018#discussion_r2326355683
##########
superset/daos/base.py:
##########
@@ -251,3 +411,247 @@ def filter_by(cls, **filter_by: Any) -> list[T]:
cls.id_column_name, data_model
).apply(query, None)
return query.filter_by(**filter_by).all()
+
+ @classmethod
+ def apply_column_operators(
+ cls, query: Any, column_operators: Optional[List[ColumnOperator]] = None
+ ) -> Any:
+ """
+ Apply column operators (list of ColumnOperator) to the query using
+ ColumnOperatorEnum logic. Raises ValueError if a filter references a
+ non-existent column.
+ """
+ if not column_operators:
+ return query
+ for c in column_operators:
+ if not isinstance(c, ColumnOperator):
+ continue
+ col = c.col
+ opr = c.opr
+ value = c.value
+ if not col or not hasattr(cls.model_cls, col):
+ model_name = cls.model_cls.__name__ if cls.model_cls else "Unknown"
+ logging.error(
+ f"Invalid filter: column '{col}' does not exist on {model_name}"
+ )
+ raise ValueError(
+ f"Invalid filter: column '{col}' does not exist on {model_name}"
+ )
+ column = getattr(cls.model_cls, col)
+ try:
+ # Always use ColumnOperatorEnum's apply method
+ operator_enum = ColumnOperatorEnum(opr)
+ query = query.filter(operator_enum.apply(column, value))
+ except Exception as e:
+ logging.error(f"Error applying filter on column '{col}': {e}")
+ raise
+ return query
+
+ @classmethod
+ def get_filterable_columns_and_operators(cls) -> Dict[str, List[str]]:
+ """
+ Returns a dict mapping filterable columns (including hybrid/computed fields if
+ present) to their supported operators. Used by MCP tools to dynamically expose
+ filter options. Custom fields supported by the DAO but not present on the model
+ should be documented here.
+ """
+ from sqlalchemy.ext.hybrid import hybrid_property
+
+ mapper = inspect(cls.model_cls)
+ columns = {c.key: c for c in mapper.columns}
+ # Add hybrid properties
+ hybrids = {
+ name: attr
+ for name, attr in vars(cls.model_cls).items()
+ if isinstance(attr, hybrid_property)
+ }
+ # You may add custom fields here, e.g.:
+ # custom_fields = {"tags": ["eq", "in_", "like"], ...}
+ custom_fields: Dict[str, List[str]] = {}
+ # Map SQLAlchemy types to supported operators
+ type_operator_map = {
Review Comment:
NIT: could be great as a module constant - or class-level prop if there's a need to override in subclasses, but i think it's static module constant probably best
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat: Add BaseDAO improvements and test reorganization [superset]
Posted by "dpgaspar (via GitHub)" <gi...@apache.org>.
dpgaspar commented on code in PR #35018:
URL: https://github.com/apache/superset/pull/35018#discussion_r2352431921
##########
superset/daos/base.py:
##########
@@ -251,3 +457,231 @@ def filter_by(cls, **filter_by: Any) -> list[T]:
cls.id_column_name, data_model
).apply(query, None)
return query.filter_by(**filter_by).all()
+
+ @classmethod
+ def apply_column_operators(
+ cls, query: Any, column_operators: Optional[List[ColumnOperator]] = None
+ ) -> Any:
+ """
+ Apply column operators (list of ColumnOperator) to the query using
+ ColumnOperatorEnum logic. Raises ValueError if a filter references a
+ non-existent column.
+ """
+ if not column_operators:
+ return query
+ for c in column_operators:
+ if not isinstance(c, ColumnOperator):
+ continue
+ col = c.col
+ opr = c.opr
+ value = c.value
Review Comment:
nit(optional): col, opr, value = c.col, c.opr, c.value
##########
superset/daos/base.py:
##########
@@ -83,52 +197,145 @@ def find_by_id_or_uuid(
return None
@classmethod
- def find_by_id(
- cls,
- model_id: str | int,
- skip_base_filter: bool = False,
- ) -> T | None:
+ def _apply_base_filter(
+ cls, query: Any, skip_base_filter: bool = False, data_model: Any = None
+ ) -> Any:
"""
- Find a model by id, if defined applies `base_filter`
+ Apply the base_filter to the query if it exists and skip_base_filter is False.
"""
- query = db.session.query(cls.model_cls)
if cls.base_filter and not skip_base_filter:
- data_model = SQLAInterface(cls.model_cls, db.session)
+ if data_model is None:
+ data_model = SQLAInterface(cls.model_cls, db.session)
query = cls.base_filter( # pylint: disable=not-callable
cls.id_column_name, data_model
).apply(query, None)
- id_column = getattr(cls.model_cls, cls.id_column_name)
+ return query
+
+ @classmethod
+ def _convert_value_for_column(cls, column: Any, value: Any) -> Any:
+ """
+ Convert a value to the appropriate type for a given SQLAlchemy column.
+
+ Args:
+ column: SQLAlchemy column object
+ value: Value to convert
+
+ Returns:
+ Converted value or None if conversion fails
+ """
+ if (
+ hasattr(column.type, "python_type")
+ and column.type.python_type == uuid_lib.UUID
+ ):
+ if isinstance(value, str):
+ try:
+ return uuid_lib.UUID(value)
+ except (ValueError, AttributeError):
+ return None
+ return value
+
+ @classmethod
+ def _find_by_column(
+ cls,
+ column_name: str,
+ value: str | int,
+ skip_base_filter: bool = False,
+ ) -> T | None:
+ """
+ Private method to find a model by any column value.
+
+ Args:
+ column_name: Name of the column to search by
+ value: Value to search for
+ skip_base_filter: Whether to skip base filtering
+
+ Returns:
+ Model instance or None if not found
+ """
+ query = db.session.query(cls.model_cls)
+ query = cls._apply_base_filter(query, skip_base_filter)
+
+ if not hasattr(cls.model_cls, column_name):
+ return None
+
+ column = getattr(cls.model_cls, column_name)
+ converted_value = cls._convert_value_for_column(column, value)
+ if converted_value is None:
+ return None
+
try:
- return query.filter(id_column == model_id).one_or_none()
+ return query.filter(column == converted_value).one_or_none()
except StatementError:
# can happen if int is passed instead of a string or similar
return None
+ @classmethod
+ def find_by_id(
+ cls,
+ model_id: str | int,
+ skip_base_filter: bool = False,
+ id_column: str | None = None,
+ ) -> T | None:
+ """
+ Find a model by ID using specified or default ID column.
+
+ Args:
+ model_id: ID value to search for
+ skip_base_filter: Whether to skip base filtering
+ id_column: Column name to use (defaults to cls.id_column_name)
+
+ Returns:
+ Model instance or None if not found
+ """
+ column = id_column or cls.id_column_name
+ return cls._find_by_column(column, model_id, skip_base_filter)
+
@classmethod
def find_by_ids(
cls,
- model_ids: list[str] | list[int],
+ model_ids: Sequence[str | int],
skip_base_filter: bool = False,
+ id_column: str | None = None,
) -> list[T]:
"""
Find a List of models by a list of ids, if defined applies `base_filter`
+
+ :param model_ids: List of IDs to find
+ :param skip_base_filter: If true, skip applying the base filter
+ :param id_column: Optional column name to use for ID lookup
+ (defaults to id_column_name)
"""
- id_col = getattr(cls.model_cls, cls.id_column_name, None)
+ column = id_column or cls.id_column_name
+ id_col = getattr(cls.model_cls, column, None)
if id_col is None or not model_ids:
return []
- query = db.session.query(cls.model_cls).filter(id_col.in_(model_ids))
- if cls.base_filter and not skip_base_filter:
- data_model = SQLAInterface(cls.model_cls, db.session)
- query = cls.base_filter( # pylint: disable=not-callable
- cls.id_column_name, data_model
- ).apply(query, None)
+
+ # Convert IDs to appropriate types based on column type
+ converted_ids: list[str | int | uuid_lib.UUID] = []
+ for id_val in model_ids:
+ converted_value = cls._convert_value_for_column(id_col, id_val)
+ if converted_value is not None:
+ converted_ids.append(converted_value)
+
+ # Validate type consistency for better error handling
+ if len(converted_ids) > 1:
+ types_found = set(map(type, converted_ids))
+ if len(types_found) > 1:
+ logger.warning(
+ "Mixed ID types detected for %s: %s",
+ cls.model_cls.__name__ if cls.model_cls else "Unknown",
+ [t.__name__ for t in types_found],
+ )
Review Comment:
Looks good, but should it fail with ValueError?
##########
superset/daos/base.py:
##########
@@ -251,3 +455,205 @@ def filter_by(cls, **filter_by: Any) -> list[T]:
cls.id_column_name, data_model
).apply(query, None)
return query.filter_by(**filter_by).all()
+
+ @classmethod
+ def apply_column_operators(
+ cls, query: Any, column_operators: Optional[List[ColumnOperator]] = None
+ ) -> Any:
+ """
+ Apply column operators (list of ColumnOperator) to the query using
+ ColumnOperatorEnum logic. Raises ValueError if a filter references a
+ non-existent column.
+ """
+ if not column_operators:
+ return query
+ for c in column_operators:
+ if not isinstance(c, ColumnOperator):
+ continue
+ col = c.col
+ opr = c.opr
+ value = c.value
+ if not col or not hasattr(cls.model_cls, col):
+ model_name = cls.model_cls.__name__ if cls.model_cls else "Unknown"
+ logging.error(
+ f"Invalid filter: column '{col}' does not exist on {model_name}"
+ )
+ raise ValueError(
+ f"Invalid filter: column '{col}' does not exist on {model_name}"
+ )
+ column = getattr(cls.model_cls, col)
+ try:
+ # Always use ColumnOperatorEnum's apply method
+ operator_enum = ColumnOperatorEnum(opr)
+ query = query.filter(operator_enum.apply(column, value))
+ except Exception as e:
+ logging.error(f"Error applying filter on column '{col}': {e}")
+ raise
+ return query
+
+ @classmethod
+ def get_filterable_columns_and_operators(cls) -> Dict[str, List[str]]:
+ """
+ Returns a dict mapping filterable columns (including hybrid/computed fields if
+ present) to their supported operators. Used by MCP tools to dynamically expose
+ filter options. Custom fields supported by the DAO but not present on the model
+ should be documented here.
+ """
+
+ mapper = inspect(cls.model_cls)
+ columns = {c.key: c for c in mapper.columns}
+ # Add hybrid properties
+ hybrids = {
+ name: attr
+ for name, attr in vars(cls.model_cls).items()
+ if isinstance(attr, hybrid_property)
+ }
+ # You may add custom fields here, e.g.:
+ # custom_fields = {"tags": ["eq", "in_", "like"], ...}
+ custom_fields: Dict[str, List[str]] = {}
+
+ filterable = {}
+ for name, col in columns.items():
+ if isinstance(col.type, (sa.String, sa.Text)):
+ filterable[name] = TYPE_OPERATOR_MAP["string"]
+ elif isinstance(col.type, (sa.Boolean,)):
+ filterable[name] = TYPE_OPERATOR_MAP["boolean"]
+ elif isinstance(col.type, (sa.Integer, sa.Float, sa.Numeric)):
+ filterable[name] = TYPE_OPERATOR_MAP["number"]
+ elif isinstance(col.type, (sa.DateTime, sa.Date, sa.Time)):
+ filterable[name] = TYPE_OPERATOR_MAP["datetime"]
+ else:
+ # Fallback to eq/ne/null
+ filterable[name] = ["eq", "ne", "is_null", "is_not_null"]
+ # Add hybrid properties as string fields by default
+ for name in hybrids:
+ filterable[name] = TYPE_OPERATOR_MAP["string"]
+ # Add custom fields
+ filterable.update(custom_fields)
+ return filterable
+
+ @classmethod
+ def _build_query(
+ cls,
+ column_operators: Optional[List[ColumnOperator]] = None,
+ search: Optional[str] = None,
+ search_columns: Optional[List[str]] = None,
+ custom_filters: Optional[Dict[str, BaseFilter]] = None,
+ skip_base_filter: bool = False,
+ data_model: Optional[SQLAInterface] = None,
+ ) -> Any:
+ """
+ Build a SQLAlchemy query with base filter, column operators, search, and
+ custom filters.
+ """
+ if data_model is None:
+ data_model = SQLAInterface(cls.model_cls, db.session)
+ query = data_model.session.query(cls.model_cls)
+ query = cls._apply_base_filter(
+ query, skip_base_filter=skip_base_filter, data_model=data_model
+ )
+ if search and search_columns:
+ search_filters = []
+ for column_name in search_columns:
+ if hasattr(cls.model_cls, column_name):
+ column = getattr(cls.model_cls, column_name)
+ search_filters.append(cast(column, Text).ilike(f"%{search}%"))
Review Comment:
This is safe because we are using SQLAlchemy ORM
SQLAlchemy doesn't directly concatenate this into the SQL string. Instead, it:
Creates a parameterized query with placeholders
Passes the value '%' + search + '%' as a parameter separately from the SQL structure
The actual SQL sent to the database looks something like:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat: Add BaseDAO improvements and test reorganization [superset]
Posted by "dpgaspar (via GitHub)" <gi...@apache.org>.
dpgaspar commented on PR #35018:
URL: https://github.com/apache/superset/pull/35018#issuecomment-3299443644
/resolve
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat: Add BaseDAO improvements and test reorganization [superset]
Posted by "aminghadersohi (via GitHub)" <gi...@apache.org>.
aminghadersohi commented on code in PR #35018:
URL: https://github.com/apache/superset/pull/35018#discussion_r2334821000
##########
superset/daos/base.py:
##########
@@ -251,3 +450,213 @@ def filter_by(cls, **filter_by: Any) -> list[T]:
cls.id_column_name, data_model
).apply(query, None)
return query.filter_by(**filter_by).all()
+
+ @classmethod
+ def apply_column_operators(
+ cls, query: Any, column_operators: Optional[List[ColumnOperator]] = None
+ ) -> Any:
+ """
+ Apply column operators (list of ColumnOperator) to the query using
+ ColumnOperatorEnum logic. Raises ValueError if a filter references a
+ non-existent column.
+ """
+ if not column_operators:
+ return query
+ for c in column_operators:
+ if not isinstance(c, ColumnOperator):
+ continue
+ col = c.col
+ opr = c.opr
+ value = c.value
+ if not col or not hasattr(cls.model_cls, col):
+ model_name = cls.model_cls.__name__ if cls.model_cls else "Unknown"
+ logging.error(
+ "Invalid filter: column '%s' does not exist on %s", col, model_name
+ )
+ raise ValueError(
+ "Invalid filter: column '%s' does not exist on %s"
+ % (col, model_name)
+ )
+ column = getattr(cls.model_cls, col)
+ try:
+ # Always use ColumnOperatorEnum's apply method
+ operator_enum = ColumnOperatorEnum(opr)
+ query = query.filter(operator_enum.apply(column, value))
+ except Exception as e:
+ logging.error("Error applying filter on column '%s': %s", col, e)
+ raise
+ return query
+
+ @classmethod
+ def get_filterable_columns_and_operators(cls) -> Dict[str, List[str]]:
+ """
+ Returns a dict mapping filterable columns (including hybrid/computed fields if
+ present) to their supported operators. Used by MCP tools to dynamically expose
+ filter options. Custom fields supported by the DAO but not present on the model
+ should be documented here.
+ """
+
+ mapper = inspect(cls.model_cls)
+ columns = {c.key: c for c in mapper.columns}
+ # Add hybrid properties
+ hybrids = {
+ name: attr
+ for name, attr in vars(cls.model_cls).items()
+ if isinstance(attr, hybrid_property)
+ }
+ # You may add custom fields here, e.g.:
+ # custom_fields = {"tags": ["eq", "in_", "like"], ...}
+ custom_fields: Dict[str, List[str]] = {}
+
+ filterable = {}
+ for name, col in columns.items():
+ if isinstance(col.type, (sa.String, sa.Text)):
+ filterable[name] = TYPE_OPERATOR_MAP["string"]
+ elif isinstance(col.type, (sa.Boolean,)):
+ filterable[name] = TYPE_OPERATOR_MAP["boolean"]
+ elif isinstance(col.type, (sa.Integer, sa.Float, sa.Numeric)):
+ filterable[name] = TYPE_OPERATOR_MAP["number"]
+ elif isinstance(col.type, (sa.DateTime, sa.Date, sa.Time)):
+ filterable[name] = TYPE_OPERATOR_MAP["datetime"]
+ else:
+ # Fallback to eq/ne/null
+ filterable[name] = ["eq", "ne", "is_null", "is_not_null"]
+ # Add hybrid properties as string fields by default
+ for name in hybrids:
+ filterable[name] = TYPE_OPERATOR_MAP["string"]
+ # Add custom fields
+ filterable.update(custom_fields)
+ return filterable
+
+ @classmethod
+ def _build_query(
+ cls,
+ column_operators: Optional[List[ColumnOperator]] = None,
+ search: Optional[str] = None,
+ search_columns: Optional[List[str]] = None,
+ custom_filters: Optional[Dict[str, BaseFilter]] = None,
+ skip_base_filter: bool = False,
+ data_model: Optional[SQLAInterface] = None,
+ ) -> Any:
+ """
+ Build a SQLAlchemy query with base filter, column operators, search, and
+ custom filters.
+ """
+ if data_model is None:
+ data_model = SQLAInterface(cls.model_cls, db.session)
+ query = data_model.session.query(cls.model_cls)
+ query = cls._apply_base_filter(
+ query, skip_base_filter=skip_base_filter, data_model=data_model
+ )
+ if search and search_columns:
+ search_filters = []
+ for column_name in search_columns:
+ if hasattr(cls.model_cls, column_name):
+ column = getattr(cls.model_cls, column_name)
+ search_filters.append(cast(column, Text).ilike(f"%{search}%"))
+ if search_filters:
+ query = query.filter(or_(*search_filters))
+ if custom_filters:
+ for filter_class in custom_filters.values():
+ query = filter_class.apply(query, None)
+ if column_operators:
+ query = cls.apply_column_operators(query, column_operators)
+ return query
+
+ @classmethod
+ def list( # noqa: C901
+ cls,
+ column_operators: Optional[List[ColumnOperator]] = None,
+ order_column: str = "changed_on",
+ order_direction: str = "desc",
+ page: int = 0,
+ page_size: int = 100,
+ search: Optional[str] = None,
+ search_columns: Optional[List[str]] = None,
+ custom_filters: Optional[Dict[str, BaseFilter]] = None,
+ columns: Optional[List[str]] = None,
+ ) -> Tuple[List[Any], int]:
+ """
+ Generic list method for filtered, sorted, and paginated results.
+ If columns is specified, returns a list of tuples (one per row),
+ otherwise returns model instances.
+ """
+ data_model = SQLAInterface(cls.model_cls, db.session)
+
+ column_attrs = []
+ relationship_loads = []
+ if columns is None:
+ columns = []
+ for name in columns:
+ attr = getattr(cls.model_cls, name, None)
+ if attr is None:
+ continue
+ prop = getattr(attr, "property", None)
+ if isinstance(prop, ColumnProperty):
+ column_attrs.append(attr)
+ elif isinstance(prop, RelationshipProperty):
+ relationship_loads.append(joinedload(attr))
+ # Ignore properties and other non-queryable attributes
+
+ if relationship_loads:
+ # If any relationships are requested, query the full model
+ # but don't add the joins yet - we'll add them after counting
+ query = data_model.session.query(cls.model_cls)
+ elif column_attrs:
+ # Only columns requested
+ query = data_model.session.query(*column_attrs)
+ else:
+ # Fallback: query the full model
+ query = data_model.session.query(cls.model_cls)
+ query = cls._apply_base_filter(query, data_model=data_model)
+ if search and search_columns:
+ search_filters = []
+ for column_name in search_columns:
+ if hasattr(cls.model_cls, column_name):
+ column = getattr(cls.model_cls, column_name)
+ search_filters.append(cast(column, Text).ilike(f"%{search}%"))
+ if search_filters:
+ query = query.filter(or_(*search_filters))
+ if custom_filters:
+ for filter_class in custom_filters.values():
+ query = filter_class.apply(query, None)
+ if column_operators:
+ query = cls.apply_column_operators(query, column_operators)
+
+ # Count before adding relationship joins to avoid inflated counts
+ # with one-to-many or many-to-many relationships
+ total_count = query.count()
+
+ # Add relationship joins after counting
Review Comment:
@dpgaspar move the relationship joins to after the count
##########
superset/daos/base.py:
##########
@@ -251,3 +450,213 @@ def filter_by(cls, **filter_by: Any) -> list[T]:
cls.id_column_name, data_model
).apply(query, None)
return query.filter_by(**filter_by).all()
+
+ @classmethod
+ def apply_column_operators(
+ cls, query: Any, column_operators: Optional[List[ColumnOperator]] = None
+ ) -> Any:
+ """
+ Apply column operators (list of ColumnOperator) to the query using
+ ColumnOperatorEnum logic. Raises ValueError if a filter references a
+ non-existent column.
+ """
+ if not column_operators:
+ return query
+ for c in column_operators:
+ if not isinstance(c, ColumnOperator):
+ continue
+ col = c.col
+ opr = c.opr
+ value = c.value
+ if not col or not hasattr(cls.model_cls, col):
+ model_name = cls.model_cls.__name__ if cls.model_cls else "Unknown"
+ logging.error(
+ "Invalid filter: column '%s' does not exist on %s", col, model_name
+ )
+ raise ValueError(
+ "Invalid filter: column '%s' does not exist on %s"
+ % (col, model_name)
+ )
+ column = getattr(cls.model_cls, col)
+ try:
+ # Always use ColumnOperatorEnum's apply method
+ operator_enum = ColumnOperatorEnum(opr)
+ query = query.filter(operator_enum.apply(column, value))
+ except Exception as e:
+ logging.error("Error applying filter on column '%s': %s", col, e)
+ raise
+ return query
+
+ @classmethod
+ def get_filterable_columns_and_operators(cls) -> Dict[str, List[str]]:
+ """
+ Returns a dict mapping filterable columns (including hybrid/computed fields if
+ present) to their supported operators. Used by MCP tools to dynamically expose
+ filter options. Custom fields supported by the DAO but not present on the model
+ should be documented here.
+ """
+
+ mapper = inspect(cls.model_cls)
+ columns = {c.key: c for c in mapper.columns}
+ # Add hybrid properties
+ hybrids = {
+ name: attr
+ for name, attr in vars(cls.model_cls).items()
+ if isinstance(attr, hybrid_property)
+ }
+ # You may add custom fields here, e.g.:
+ # custom_fields = {"tags": ["eq", "in_", "like"], ...}
+ custom_fields: Dict[str, List[str]] = {}
+
+ filterable = {}
+ for name, col in columns.items():
+ if isinstance(col.type, (sa.String, sa.Text)):
+ filterable[name] = TYPE_OPERATOR_MAP["string"]
+ elif isinstance(col.type, (sa.Boolean,)):
+ filterable[name] = TYPE_OPERATOR_MAP["boolean"]
+ elif isinstance(col.type, (sa.Integer, sa.Float, sa.Numeric)):
+ filterable[name] = TYPE_OPERATOR_MAP["number"]
+ elif isinstance(col.type, (sa.DateTime, sa.Date, sa.Time)):
+ filterable[name] = TYPE_OPERATOR_MAP["datetime"]
+ else:
+ # Fallback to eq/ne/null
+ filterable[name] = ["eq", "ne", "is_null", "is_not_null"]
+ # Add hybrid properties as string fields by default
+ for name in hybrids:
+ filterable[name] = TYPE_OPERATOR_MAP["string"]
+ # Add custom fields
+ filterable.update(custom_fields)
+ return filterable
+
+ @classmethod
+ def _build_query(
+ cls,
+ column_operators: Optional[List[ColumnOperator]] = None,
+ search: Optional[str] = None,
+ search_columns: Optional[List[str]] = None,
+ custom_filters: Optional[Dict[str, BaseFilter]] = None,
+ skip_base_filter: bool = False,
+ data_model: Optional[SQLAInterface] = None,
+ ) -> Any:
+ """
+ Build a SQLAlchemy query with base filter, column operators, search, and
+ custom filters.
+ """
+ if data_model is None:
+ data_model = SQLAInterface(cls.model_cls, db.session)
+ query = data_model.session.query(cls.model_cls)
+ query = cls._apply_base_filter(
+ query, skip_base_filter=skip_base_filter, data_model=data_model
+ )
+ if search and search_columns:
+ search_filters = []
+ for column_name in search_columns:
+ if hasattr(cls.model_cls, column_name):
+ column = getattr(cls.model_cls, column_name)
+ search_filters.append(cast(column, Text).ilike(f"%{search}%"))
+ if search_filters:
+ query = query.filter(or_(*search_filters))
+ if custom_filters:
+ for filter_class in custom_filters.values():
+ query = filter_class.apply(query, None)
+ if column_operators:
+ query = cls.apply_column_operators(query, column_operators)
+ return query
+
+ @classmethod
+ def list( # noqa: C901
+ cls,
+ column_operators: Optional[List[ColumnOperator]] = None,
+ order_column: str = "changed_on",
+ order_direction: str = "desc",
+ page: int = 0,
+ page_size: int = 100,
+ search: Optional[str] = None,
+ search_columns: Optional[List[str]] = None,
+ custom_filters: Optional[Dict[str, BaseFilter]] = None,
+ columns: Optional[List[str]] = None,
+ ) -> Tuple[List[Any], int]:
+ """
+ Generic list method for filtered, sorted, and paginated results.
+ If columns is specified, returns a list of tuples (one per row),
+ otherwise returns model instances.
+ """
+ data_model = SQLAInterface(cls.model_cls, db.session)
+
+ column_attrs = []
+ relationship_loads = []
+ if columns is None:
+ columns = []
+ for name in columns:
+ attr = getattr(cls.model_cls, name, None)
+ if attr is None:
+ continue
+ prop = getattr(attr, "property", None)
+ if isinstance(prop, ColumnProperty):
+ column_attrs.append(attr)
+ elif isinstance(prop, RelationshipProperty):
+ relationship_loads.append(joinedload(attr))
+ # Ignore properties and other non-queryable attributes
+
+ if relationship_loads:
+ # If any relationships are requested, query the full model
+ # but don't add the joins yet - we'll add them after counting
+ query = data_model.session.query(cls.model_cls)
+ elif column_attrs:
+ # Only columns requested
+ query = data_model.session.query(*column_attrs)
+ else:
+ # Fallback: query the full model
+ query = data_model.session.query(cls.model_cls)
+ query = cls._apply_base_filter(query, data_model=data_model)
+ if search and search_columns:
+ search_filters = []
+ for column_name in search_columns:
+ if hasattr(cls.model_cls, column_name):
+ column = getattr(cls.model_cls, column_name)
+ search_filters.append(cast(column, Text).ilike(f"%{search}%"))
+ if search_filters:
+ query = query.filter(or_(*search_filters))
+ if custom_filters:
+ for filter_class in custom_filters.values():
+ query = filter_class.apply(query, None)
+ if column_operators:
+ query = cls.apply_column_operators(query, column_operators)
+
+ # Count before adding relationship joins to avoid inflated counts
+ # with one-to-many or many-to-many relationships
+ total_count = query.count()
+
+ # Add relationship joins after counting
Review Comment:
@dpgaspar moved the relationship joins to after the count
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat: Add BaseDAO improvements and test reorganization [superset]
Posted by "bito-code-review[bot] (via GitHub)" <gi...@apache.org>.
bito-code-review[bot] commented on PR #35018:
URL: https://github.com/apache/superset/pull/35018#issuecomment-3267072028
AI Code Review is in progress (usually takes 3 to 15 minutes unless it's a very large PR).
<details>
<summary>
Bito Usage Guide
</summary>
**Commands**
Type the following command in the pull request comment and save the comment.
- `/review` - Manually triggers a full AI review.
- `/pause` - Pauses automatic reviews on this pull request.
- `/resume` - Resumes automatic reviews.
- `/resolve` - Marks all Bito-posted review comments as resolved.
- `/abort` - Cancels all in-progress reviews.
Refer to the <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/available-commands" target="_blank" rel="noopener noreferrer">documentation</a> for additional commands.
**Configuration**
This repository uses `Default Agent` You can customize the agent settings <a href="https://alpha.bito.ai/home/ai-agents/code-review-agent" target="_blank" rel="noopener noreferrer">here</a> or contact your Bito workspace admin at evan@preset.io.
**Documentation & Help**
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/getting-started/install-run-using-bito-cloud/create-or-customize-an-agent-instance" target="_blank" rel="noopener noreferrer">Customize agent settings</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/implementing-custom-code-review-rules" target="_blank" rel="noopener noreferrer">Review rules</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent" target="_blank" rel="noopener noreferrer">General documentation</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/faqs" target="_blank" rel="noopener noreferrer">FAQ</a>
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat: Add BaseDAO improvements and test reorganization [superset]
Posted by "aminghadersohi (via GitHub)" <gi...@apache.org>.
aminghadersohi commented on code in PR #35018:
URL: https://github.com/apache/superset/pull/35018#discussion_r2334798195
##########
tests/integration_tests/dao/base_dao_test.py:
##########
@@ -0,0 +1,1397 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""
+Integration tests for BaseDAO functionality.
+
+This module contains comprehensive integration tests for the BaseDAO class and its
+subclasses, covering database operations, CRUD methods, flexible column support,
+column operators, and error handling.
+
+Tests use an in-memory SQLite database for isolation and to replicate the unit test
+environment behavior. User model deletions are avoided due to circular dependency
+constraints with self-referential foreign keys.
+"""
+
+import datetime
+import time
+import uuid
+
+import pytest
+from flask_appbuilder.models.filters import BaseFilter
+from flask_appbuilder.security.sqla.models import User
+from sqlalchemy import Column, DateTime, Integer, String
+from sqlalchemy.ext.declarative import declarative_base
+from sqlalchemy.orm.session import Session
+
+from superset.daos.base import BaseDAO, ColumnOperator, ColumnOperatorEnum
+from superset.daos.chart import ChartDAO
+from superset.daos.dashboard import DashboardDAO
+from superset.daos.database import DatabaseDAO
+from superset.daos.user import UserDAO
+from superset.extensions import db
+from superset.models.core import Database
+from superset.models.dashboard import Dashboard
+from superset.models.slice import Slice
+
+# Create a test model for comprehensive testing
+Base = declarative_base()
+
+
+class ExampleModel(Base): # type: ignore
+ __tablename__ = "example_model"
+ id = Column(Integer, primary_key=True)
+ uuid = Column(String(36), unique=True, nullable=False)
+ slug = Column(String(100), unique=True)
+ name = Column(String(100))
+ code = Column(String(50), unique=True)
+ created_on = Column(DateTime, default=datetime.datetime.utcnow)
+
+
+class ExampleModelDAO(BaseDAO[ExampleModel]):
+ model_cls = ExampleModel
+ id_column_name = "id"
+ base_filter = None
+
+
+class MockModel:
+ def __init__(self, id=1, name="test"):
+ self.id = id
+ self.name = name
+
+
+class TestDAO(BaseDAO[MockModel]):
+ model_cls = MockModel
+
+
+@pytest.fixture(autouse=True)
+def mock_g_user(app_context):
+ """Mock the flask g.user for security context."""
+ # Within app context, we can safely mock g
+ from flask import g
+
+ mock_user = User()
+ mock_user.id = 1
+ mock_user.username = "test_user"
+
+ # Set g.user directly instead of patching
+ g.user = mock_user
+ yield
+
+ # Clean up
+ if hasattr(g, "user"):
+ delattr(g, "user")
+
+
+# =============================================================================
+# Integration Tests - These tests use the actual database
+# =============================================================================
+
+
+def test_column_operator_enum_complete_coverage(user_with_data: Session) -> None:
+ """
+ Test that every single ColumnOperatorEnum operator is covered by tests.
+ This ensures we have comprehensive test coverage for all operators.
+ """
+ # Simply verify that we can create queries with all operators
+ for operator in ColumnOperatorEnum:
+ column_operator = ColumnOperator(
+ col="username", opr=operator, value="test_value"
+ )
+ # Just check it doesn't raise an error
+ assert column_operator.opr == operator
+
+
+def test_find_by_id_with_default_column(app_context: Session) -> None:
+ """Test find_by_id with default 'id' column."""
+ # Create a user to test with
+ user = User(
+ username="test_find_by_id",
+ first_name="Test",
+ last_name="User",
+ email="test@example.com",
+ active=True,
+ )
+ db.session.add(user)
+ db.session.commit()
+
+ # Find by numeric id
+ found = UserDAO.find_by_id(user.id, skip_base_filter=True)
+ assert found is not None
+ assert found.id == user.id
+ assert found.username == "test_find_by_id"
+
+ # Test with non-existent id
+ not_found = UserDAO.find_by_id(999999, skip_base_filter=True)
+ assert not_found is None
+
+
+def test_find_by_id_with_uuid_column(app_context: Session) -> None:
+ """Test find_by_id with custom uuid column."""
+ # Create a dashboard with uuid
+ dashboard = Dashboard(
+ dashboard_title="Test UUID Dashboard",
+ slug="test-uuid-dashboard",
+ published=True,
+ )
+ db.session.add(dashboard)
+ db.session.commit()
+
+ # Find by uuid string using the uuid column
+ found = DashboardDAO.find_by_id(
+ str(dashboard.uuid), id_column="uuid", skip_base_filter=True
+ )
+ assert found is not None
+ assert found.uuid == dashboard.uuid
+ assert found.dashboard_title == "Test UUID Dashboard"
+
+ # Find by numeric id (should still work)
+ found_by_id = DashboardDAO.find_by_id(dashboard.id, skip_base_filter=True)
+ assert found_by_id is not None
+ assert found_by_id.id == dashboard.id
+
+ # Test with non-existent uuid
+ not_found = DashboardDAO.find_by_id(str(uuid.uuid4()), skip_base_filter=True)
+ assert not_found is None
+
+
+def test_find_by_id_with_slug_column(app_context: Session) -> None:
+ """Test find_by_id with slug column fallback."""
+ # Create a dashboard with slug
+ dashboard = Dashboard(
+ dashboard_title="Test Slug Dashboard",
+ slug="test-slug-dashboard",
+ published=True,
+ )
+ db.session.add(dashboard)
+ db.session.commit()
+
+ # Find by slug using the slug column
+ found = DashboardDAO.find_by_id(
+ "test-slug-dashboard", id_column="slug", skip_base_filter=True
+ )
+ assert found is not None
+ assert found.slug == "test-slug-dashboard"
+ assert found.dashboard_title == "Test Slug Dashboard"
+
+ # Test with non-existent slug
+ not_found = DashboardDAO.find_by_id("non-existent-slug", skip_base_filter=True)
+ assert not_found is None
+
+
+def test_find_by_id_with_invalid_column(app_context: Session) -> None:
+ """Test find_by_id returns None when column doesn't exist."""
+ # This should return None gracefully
+ result = UserDAO.find_by_id("not_a_valid_id", skip_base_filter=True)
+ assert result is None
+
+
+def test_find_by_id_skip_base_filter(app_context: Session) -> None:
+ """Test find_by_id with skip_base_filter parameter."""
+ # Create users with different active states
+ active_user = User(
+ username="active_user",
+ first_name="Active",
+ last_name="User",
+ email="active@example.com",
+ active=True,
+ )
+ inactive_user = User(
+ username="inactive_user",
+ first_name="Inactive",
+ last_name="User",
+ email="inactive@example.com",
+ active=False,
+ )
+ db.session.add_all([active_user, inactive_user])
+ db.session.commit()
+
+ # Without skipping base filter (if one exists)
+ found_active = UserDAO.find_by_id(active_user.id, skip_base_filter=False)
+ assert found_active is not None
+
+ # With skipping base filter
+ found_active_skip = UserDAO.find_by_id(active_user.id, skip_base_filter=True)
+ assert found_active_skip is not None
+
+ # Both should find the user since UserDAO might not have a base filter
+ assert found_active.id == active_user.id
+ assert found_active_skip.id == active_user.id
+
+
+def test_find_by_ids_with_default_column(app_context: Session) -> None:
+ """Test find_by_ids with default 'id' column."""
+ # Create multiple users
+ users = []
+ for i in range(3):
+ user = User(
+ username=f"test_find_by_ids_{i}",
+ first_name=f"Test{i}",
+ last_name="User",
+ email=f"test{i}@example.com",
+ active=True,
+ )
+ users.append(user)
+ db.session.add(user)
+ db.session.commit()
+
+ # Find by multiple ids
+ ids = [user.id for user in users]
+ found = UserDAO.find_by_ids(ids, skip_base_filter=True)
+ assert len(found) == 3
+ found_ids = [u.id for u in found]
+ assert set(found_ids) == set(ids)
+
+ # Test with mix of existent and non-existent ids
+ mixed_ids = [users[0].id, 999999, users[1].id]
+ found_mixed = UserDAO.find_by_ids(mixed_ids, skip_base_filter=True)
+ assert len(found_mixed) == 2
+
+ # Test with empty list
+ found_empty = UserDAO.find_by_ids([], skip_base_filter=True)
+ assert found_empty == []
+
+
+def test_find_by_ids_with_uuid_column(app_context: Session) -> None:
+ """Test find_by_ids with uuid column."""
+ # Create multiple dashboards
+ dashboards = []
+ for i in range(3):
+ dashboard = Dashboard(
+ dashboard_title=f"Test UUID Dashboard {i}",
+ slug=f"test-uuid-dashboard-{i}",
+ published=True,
+ )
+ dashboards.append(dashboard)
+ db.session.add(dashboard)
+ db.session.commit()
+
+ # Find by multiple uuids
+ uuids = [str(dashboard.uuid) for dashboard in dashboards]
+ found = DashboardDAO.find_by_ids(uuids, id_column="uuid", skip_base_filter=True)
+ assert len(found) == 3
+ found_uuids = [str(d.uuid) for d in found]
+ assert set(found_uuids) == set(uuids)
+
+ # Test with mix of ids and uuids - search separately by column
+ found_by_id = DashboardDAO.find_by_ids([dashboards[0].id], skip_base_filter=True)
+ found_by_uuid = DashboardDAO.find_by_ids(
+ [str(dashboards[1].uuid)], id_column="uuid", skip_base_filter=True
+ )
+ assert len(found_by_id) == 1
+ assert len(found_by_uuid) == 1
+
+
+def test_find_by_ids_with_slug_column(app_context: Session) -> None:
+ """Test find_by_ids with slug column."""
+ # Create multiple dashboards
+ dashboards = []
+ for i in range(3):
+ dashboard = Dashboard(
+ dashboard_title=f"Test Slug Dashboard {i}",
+ slug=f"test-slug-dashboard-{i}",
+ published=True,
+ )
+ dashboards.append(dashboard)
+ db.session.add(dashboard)
+ db.session.commit()
+
+ # Find by multiple slugs
+ slugs = [dashboard.slug for dashboard in dashboards]
+ found = DashboardDAO.find_by_ids(slugs, id_column="slug", skip_base_filter=True)
+ assert len(found) == 3
+ found_slugs = [d.slug for d in found]
+ assert set(found_slugs) == set(slugs)
+
+
+def test_find_by_ids_with_invalid_column(app_context: Session) -> None:
+ """Test find_by_ids returns empty list when column doesn't exist."""
+ # This should return empty list gracefully
+ result = UserDAO.find_by_ids(["not_a_valid_id"], skip_base_filter=True)
+ assert result == []
+
+
+def test_find_by_ids_skip_base_filter(app_context: Session) -> None:
+ """Test find_by_ids with skip_base_filter parameter."""
+ # Create users
+ users = []
+ for i in range(3):
+ user = User(
+ username=f"test_skip_filter_{i}",
+ first_name=f"Test{i}",
+ last_name="User",
+ email=f"test{i}@example.com",
+ active=True,
+ )
+ users.append(user)
+ db.session.add(user)
+ db.session.commit()
+
+ ids = [user.id for user in users]
+
+ # Without skipping base filter
+ found_no_skip = UserDAO.find_by_ids(ids, skip_base_filter=False)
+ assert len(found_no_skip) == 3
+
+ # With skipping base filter
+ found_skip = UserDAO.find_by_ids(ids, skip_base_filter=True)
+ assert len(found_skip) == 3
+
+
+def test_base_dao_create_with_item(app_context: Session) -> None:
+ """Test BaseDAO.create with an item parameter."""
+ # Create a user item
+ user = User(
+ username="created_with_item",
+ first_name="Created",
+ last_name="Item",
+ email="created@example.com",
+ active=True,
+ )
+
+ # Create using the item
+ created = UserDAO.create(item=user)
+ assert created is not None
+ assert created.username == "created_with_item"
+ assert created.first_name == "Created"
+
+ # Verify it's in the session
+ assert created in db.session
+
+ # Commit and verify it persists
+ db.session.commit()
+
+ # Find it again to ensure it was saved
+ found = UserDAO.find_by_id(created.id, skip_base_filter=True)
+ assert found is not None
+ assert found.username == "created_with_item"
+
+
+def test_base_dao_create_with_attributes(app_context: Session) -> None:
+ """Test BaseDAO.create with attributes parameter."""
+ # Create using attributes dict
+ attributes = {
+ "username": "created_with_attrs",
+ "first_name": "Created",
+ "last_name": "Attrs",
+ "email": "attrs@example.com",
+ "active": True,
+ }
+
+ created = UserDAO.create(attributes=attributes)
+ assert created is not None
+ assert created.username == "created_with_attrs"
+ assert created.email == "attrs@example.com"
+
+ # Commit and verify
+ db.session.commit()
+ found = UserDAO.find_by_id(created.id, skip_base_filter=True)
+ assert found is not None
+ assert found.username == "created_with_attrs"
+
+
+def test_base_dao_create_with_both_item_and_attributes(app_context: Session) -> None:
+ """Test BaseDAO.create with both item and attributes (override behavior)."""
+ # Create a user item
+ user = User(
+ username="item_username",
+ first_name="Item",
+ last_name="User",
+ email="item@example.com",
+ active=False,
+ )
+
+ # Override some attributes
+ attributes = {
+ "username": "override_username",
+ "active": True,
+ }
+
+ created = UserDAO.create(item=user, attributes=attributes)
+ assert created is not None
+ assert created.username == "override_username" # Should be overridden
+ assert created.active is True # Should be overridden
+ assert created.first_name == "Item" # Should keep original
+ assert created.last_name == "User" # Should keep original
+
+ db.session.commit()
+
+
+def test_base_dao_update_with_item(app_context: Session) -> None:
+ """Test BaseDAO.update with an item parameter."""
+ # Create a user first
+ user = User(
+ username="update_test",
+ first_name="Original",
+ last_name="User",
+ email="original@example.com",
+ active=True,
+ )
+ db.session.add(user)
+ db.session.commit()
+
+ # Update the user
+ user.first_name = "Updated"
+ updated = UserDAO.update(item=user)
+ assert updated is not None
+ assert updated.first_name == "Updated"
+
+ db.session.commit()
+
+ # Verify the update persisted
+ found = UserDAO.find_by_id(user.id, skip_base_filter=True)
+ assert found is not None
+ assert found.first_name == "Updated"
+
+
+def test_base_dao_update_with_attributes(app_context: Session) -> None:
+ """Test BaseDAO.update with attributes parameter."""
+ # Create a user first
+ user = User(
+ username="update_attrs_test",
+ first_name="Original",
+ last_name="User",
+ email="original@example.com",
+ active=True,
+ )
+ db.session.add(user)
+ db.session.commit()
+
+ # Update using attributes
+ attributes = {"first_name": "Updated", "last_name": "Attr User"}
+ updated = UserDAO.update(item=user, attributes=attributes)
+ assert updated is not None
+ assert updated.first_name == "Updated"
+ assert updated.last_name == "Attr User"
+
+ db.session.commit()
+
+
+def test_base_dao_update_detached_item(app_context: Session) -> None:
+ """Test BaseDAO.update with a detached item."""
+ # Create a user first
+ user = User(
+ username="detached_test",
+ first_name="Original",
+ last_name="User",
+ email="detached@example.com",
+ active=True,
+ )
+ db.session.add(user)
+ db.session.commit()
+
+ user_id = user.id
+
+ # Expunge to detach from session
+ db.session.expunge(user)
+
+ # Update the detached user
+ user.first_name = "Updated Detached"
+ updated = UserDAO.update(item=user)
+ assert updated is not None
+ assert updated.first_name == "Updated Detached"
+
+ db.session.commit()
+
+ # Verify the update persisted
+ found = UserDAO.find_by_id(user_id, skip_base_filter=True)
+ assert found is not None
+ assert found.first_name == "Updated Detached"
+
+
+def test_base_dao_delete_single_item(app_context: Session) -> None:
+ """Test BaseDAO.delete with a single item."""
+ # Create a dashboard instead of user to avoid circular dependencies
+ dashboard = Dashboard(
+ dashboard_title="Delete Test",
+ slug="delete-test",
+ published=True,
+ )
+ db.session.add(dashboard)
+ db.session.commit()
+
+ dashboard_id = dashboard.id
+
+ DashboardDAO.delete([dashboard])
+ db.session.commit()
+
+ # Verify it's gone
+ found = DashboardDAO.find_by_id(dashboard_id, skip_base_filter=True)
+ assert found is None
+
+
+def test_base_dao_delete_multiple_items(app_context: Session) -> None:
+ """Test BaseDAO.delete with multiple items."""
+ # Create multiple dashboards instead of users to avoid circular dependencies
+ dashboards = []
+ for i in range(3):
+ dashboard = Dashboard(
+ dashboard_title=f"Delete Multi {i}",
+ slug=f"delete-multi-{i}",
+ published=True,
+ )
+ dashboards.append(dashboard)
+ db.session.add(dashboard)
+ db.session.commit()
+
+ dashboard_ids = [dashboard.id for dashboard in dashboards]
+
+ DashboardDAO.delete(dashboards)
+ db.session.commit()
+
+ # Verify they're all gone
+ for dashboard_id in dashboard_ids:
+ found = DashboardDAO.find_by_id(dashboard_id, skip_base_filter=True)
+ assert found is None
Review Comment:
this inside the loop.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat: Add BaseDAO improvements and test reorganization [superset]
Posted by "aminghadersohi (via GitHub)" <gi...@apache.org>.
aminghadersohi commented on code in PR #35018:
URL: https://github.com/apache/superset/pull/35018#discussion_r2334772030
##########
tests/integration_tests/dao/base_dao_test.py:
##########
@@ -0,0 +1,1397 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""
+Integration tests for BaseDAO functionality.
+
+This module contains comprehensive integration tests for the BaseDAO class and its
+subclasses, covering database operations, CRUD methods, flexible column support,
+column operators, and error handling.
+
+Tests use an in-memory SQLite database for isolation and to replicate the unit test
+environment behavior. User model deletions are avoided due to circular dependency
+constraints with self-referential foreign keys.
+"""
+
+import datetime
+import time
+import uuid
+
+import pytest
+from flask_appbuilder.models.filters import BaseFilter
+from flask_appbuilder.security.sqla.models import User
+from sqlalchemy import Column, DateTime, Integer, String
+from sqlalchemy.ext.declarative import declarative_base
+from sqlalchemy.orm.session import Session
+
+from superset.daos.base import BaseDAO, ColumnOperator, ColumnOperatorEnum
+from superset.daos.chart import ChartDAO
+from superset.daos.dashboard import DashboardDAO
+from superset.daos.database import DatabaseDAO
+from superset.daos.user import UserDAO
+from superset.extensions import db
+from superset.models.core import Database
+from superset.models.dashboard import Dashboard
+from superset.models.slice import Slice
+
+# Create a test model for comprehensive testing
+Base = declarative_base()
+
+
+class ExampleModel(Base): # type: ignore
+ __tablename__ = "example_model"
+ id = Column(Integer, primary_key=True)
+ uuid = Column(String(36), unique=True, nullable=False)
+ slug = Column(String(100), unique=True)
+ name = Column(String(100))
+ code = Column(String(50), unique=True)
+ created_on = Column(DateTime, default=datetime.datetime.utcnow)
+
+
+class ExampleModelDAO(BaseDAO[ExampleModel]):
+ model_cls = ExampleModel
+ id_column_name = "id"
+ base_filter = None
+
+
+class MockModel:
+ def __init__(self, id=1, name="test"):
+ self.id = id
+ self.name = name
+
+
+class TestDAO(BaseDAO[MockModel]):
+ model_cls = MockModel
+
+
+@pytest.fixture(autouse=True)
+def mock_g_user(app_context):
+ """Mock the flask g.user for security context."""
+ # Within app context, we can safely mock g
+ from flask import g
+
+ mock_user = User()
+ mock_user.id = 1
+ mock_user.username = "test_user"
+
+ # Set g.user directly instead of patching
+ g.user = mock_user
+ yield
+
+ # Clean up
+ if hasattr(g, "user"):
+ delattr(g, "user")
+
+
+# =============================================================================
+# Integration Tests - These tests use the actual database
+# =============================================================================
+
+
+def test_column_operator_enum_complete_coverage(user_with_data: Session) -> None:
+ """
+ Test that every single ColumnOperatorEnum operator is covered by tests.
+ This ensures we have comprehensive test coverage for all operators.
+ """
+ # Simply verify that we can create queries with all operators
+ for operator in ColumnOperatorEnum:
+ column_operator = ColumnOperator(
+ col="username", opr=operator, value="test_value"
+ )
+ # Just check it doesn't raise an error
+ assert column_operator.opr == operator
+
+
+def test_find_by_id_with_default_column(app_context: Session) -> None:
+ """Test find_by_id with default 'id' column."""
+ # Create a user to test with
+ user = User(
+ username="test_find_by_id",
+ first_name="Test",
+ last_name="User",
+ email="test@example.com",
+ active=True,
+ )
+ db.session.add(user)
+ db.session.commit()
+
+ # Find by numeric id
+ found = UserDAO.find_by_id(user.id, skip_base_filter=True)
+ assert found is not None
+ assert found.id == user.id
+ assert found.username == "test_find_by_id"
+
+ # Test with non-existent id
+ not_found = UserDAO.find_by_id(999999, skip_base_filter=True)
+ assert not_found is None
+
+
+def test_find_by_id_with_uuid_column(app_context: Session) -> None:
+ """Test find_by_id with custom uuid column."""
+ # Create a dashboard with uuid
+ dashboard = Dashboard(
+ dashboard_title="Test UUID Dashboard",
+ slug="test-uuid-dashboard",
+ published=True,
+ )
+ db.session.add(dashboard)
+ db.session.commit()
+
+ # Find by uuid string using the uuid column
+ found = DashboardDAO.find_by_id(
+ str(dashboard.uuid), id_column="uuid", skip_base_filter=True
+ )
+ assert found is not None
+ assert found.uuid == dashboard.uuid
+ assert found.dashboard_title == "Test UUID Dashboard"
+
+ # Find by numeric id (should still work)
+ found_by_id = DashboardDAO.find_by_id(dashboard.id, skip_base_filter=True)
+ assert found_by_id is not None
+ assert found_by_id.id == dashboard.id
+
+ # Test with non-existent uuid
+ not_found = DashboardDAO.find_by_id(str(uuid.uuid4()), skip_base_filter=True)
+ assert not_found is None
+
+
+def test_find_by_id_with_slug_column(app_context: Session) -> None:
+ """Test find_by_id with slug column fallback."""
+ # Create a dashboard with slug
+ dashboard = Dashboard(
+ dashboard_title="Test Slug Dashboard",
+ slug="test-slug-dashboard",
+ published=True,
+ )
+ db.session.add(dashboard)
+ db.session.commit()
+
+ # Find by slug using the slug column
+ found = DashboardDAO.find_by_id(
+ "test-slug-dashboard", id_column="slug", skip_base_filter=True
+ )
+ assert found is not None
+ assert found.slug == "test-slug-dashboard"
+ assert found.dashboard_title == "Test Slug Dashboard"
+
+ # Test with non-existent slug
+ not_found = DashboardDAO.find_by_id("non-existent-slug", skip_base_filter=True)
+ assert not_found is None
+
+
+def test_find_by_id_with_invalid_column(app_context: Session) -> None:
+ """Test find_by_id returns None when column doesn't exist."""
+ # This should return None gracefully
+ result = UserDAO.find_by_id("not_a_valid_id", skip_base_filter=True)
+ assert result is None
+
+
+def test_find_by_id_skip_base_filter(app_context: Session) -> None:
+ """Test find_by_id with skip_base_filter parameter."""
+ # Create users with different active states
+ active_user = User(
+ username="active_user",
+ first_name="Active",
+ last_name="User",
+ email="active@example.com",
+ active=True,
+ )
+ inactive_user = User(
+ username="inactive_user",
+ first_name="Inactive",
+ last_name="User",
+ email="inactive@example.com",
+ active=False,
+ )
+ db.session.add_all([active_user, inactive_user])
+ db.session.commit()
+
+ # Without skipping base filter (if one exists)
+ found_active = UserDAO.find_by_id(active_user.id, skip_base_filter=False)
+ assert found_active is not None
+
+ # With skipping base filter
+ found_active_skip = UserDAO.find_by_id(active_user.id, skip_base_filter=True)
+ assert found_active_skip is not None
+
+ # Both should find the user since UserDAO might not have a base filter
+ assert found_active.id == active_user.id
+ assert found_active_skip.id == active_user.id
+
+
+def test_find_by_ids_with_default_column(app_context: Session) -> None:
+ """Test find_by_ids with default 'id' column."""
+ # Create multiple users
+ users = []
+ for i in range(3):
+ user = User(
+ username=f"test_find_by_ids_{i}",
+ first_name=f"Test{i}",
+ last_name="User",
+ email=f"test{i}@example.com",
+ active=True,
+ )
+ users.append(user)
+ db.session.add(user)
+ db.session.commit()
+
+ # Find by multiple ids
+ ids = [user.id for user in users]
+ found = UserDAO.find_by_ids(ids, skip_base_filter=True)
+ assert len(found) == 3
+ found_ids = [u.id for u in found]
+ assert set(found_ids) == set(ids)
+
+ # Test with mix of existent and non-existent ids
+ mixed_ids = [users[0].id, 999999, users[1].id]
+ found_mixed = UserDAO.find_by_ids(mixed_ids, skip_base_filter=True)
+ assert len(found_mixed) == 2
+
+ # Test with empty list
+ found_empty = UserDAO.find_by_ids([], skip_base_filter=True)
+ assert found_empty == []
+
+
+def test_find_by_ids_with_uuid_column(app_context: Session) -> None:
+ """Test find_by_ids with uuid column."""
+ # Create multiple dashboards
+ dashboards = []
+ for i in range(3):
+ dashboard = Dashboard(
+ dashboard_title=f"Test UUID Dashboard {i}",
+ slug=f"test-uuid-dashboard-{i}",
+ published=True,
+ )
+ dashboards.append(dashboard)
+ db.session.add(dashboard)
+ db.session.commit()
+
+ # Find by multiple uuids
+ uuids = [str(dashboard.uuid) for dashboard in dashboards]
+ found = DashboardDAO.find_by_ids(uuids, id_column="uuid", skip_base_filter=True)
+ assert len(found) == 3
+ found_uuids = [str(d.uuid) for d in found]
+ assert set(found_uuids) == set(uuids)
+
+ # Test with mix of ids and uuids - search separately by column
+ found_by_id = DashboardDAO.find_by_ids([dashboards[0].id], skip_base_filter=True)
+ found_by_uuid = DashboardDAO.find_by_ids(
+ [str(dashboards[1].uuid)], id_column="uuid", skip_base_filter=True
+ )
+ assert len(found_by_id) == 1
+ assert len(found_by_uuid) == 1
+
+
+def test_find_by_ids_with_slug_column(app_context: Session) -> None:
+ """Test find_by_ids with slug column."""
+ # Create multiple dashboards
+ dashboards = []
+ for i in range(3):
+ dashboard = Dashboard(
+ dashboard_title=f"Test Slug Dashboard {i}",
+ slug=f"test-slug-dashboard-{i}",
+ published=True,
+ )
+ dashboards.append(dashboard)
+ db.session.add(dashboard)
+ db.session.commit()
+
+ # Find by multiple slugs
+ slugs = [dashboard.slug for dashboard in dashboards]
+ found = DashboardDAO.find_by_ids(slugs, id_column="slug", skip_base_filter=True)
+ assert len(found) == 3
+ found_slugs = [d.slug for d in found]
+ assert set(found_slugs) == set(slugs)
+
+
+def test_find_by_ids_with_invalid_column(app_context: Session) -> None:
+ """Test find_by_ids returns empty list when column doesn't exist."""
+ # This should return empty list gracefully
+ result = UserDAO.find_by_ids(["not_a_valid_id"], skip_base_filter=True)
+ assert result == []
+
+
+def test_find_by_ids_skip_base_filter(app_context: Session) -> None:
+ """Test find_by_ids with skip_base_filter parameter."""
+ # Create users
+ users = []
+ for i in range(3):
+ user = User(
+ username=f"test_skip_filter_{i}",
+ first_name=f"Test{i}",
+ last_name="User",
+ email=f"test{i}@example.com",
+ active=True,
+ )
+ users.append(user)
+ db.session.add(user)
+ db.session.commit()
+
+ ids = [user.id for user in users]
+
+ # Without skipping base filter
+ found_no_skip = UserDAO.find_by_ids(ids, skip_base_filter=False)
+ assert len(found_no_skip) == 3
+
+ # With skipping base filter
+ found_skip = UserDAO.find_by_ids(ids, skip_base_filter=True)
+ assert len(found_skip) == 3
+
+
+def test_base_dao_create_with_item(app_context: Session) -> None:
+ """Test BaseDAO.create with an item parameter."""
+ # Create a user item
+ user = User(
+ username="created_with_item",
+ first_name="Created",
+ last_name="Item",
+ email="created@example.com",
+ active=True,
+ )
+
+ # Create using the item
+ created = UserDAO.create(item=user)
+ assert created is not None
+ assert created.username == "created_with_item"
+ assert created.first_name == "Created"
+
+ # Verify it's in the session
+ assert created in db.session
+
+ # Commit and verify it persists
+ db.session.commit()
+
+ # Find it again to ensure it was saved
+ found = UserDAO.find_by_id(created.id, skip_base_filter=True)
+ assert found is not None
+ assert found.username == "created_with_item"
+
+
+def test_base_dao_create_with_attributes(app_context: Session) -> None:
+ """Test BaseDAO.create with attributes parameter."""
+ # Create using attributes dict
+ attributes = {
+ "username": "created_with_attrs",
+ "first_name": "Created",
+ "last_name": "Attrs",
+ "email": "attrs@example.com",
+ "active": True,
+ }
+
+ created = UserDAO.create(attributes=attributes)
+ assert created is not None
+ assert created.username == "created_with_attrs"
+ assert created.email == "attrs@example.com"
+
+ # Commit and verify
+ db.session.commit()
+ found = UserDAO.find_by_id(created.id, skip_base_filter=True)
+ assert found is not None
+ assert found.username == "created_with_attrs"
+
+
+def test_base_dao_create_with_both_item_and_attributes(app_context: Session) -> None:
+ """Test BaseDAO.create with both item and attributes (override behavior)."""
+ # Create a user item
+ user = User(
+ username="item_username",
+ first_name="Item",
+ last_name="User",
+ email="item@example.com",
+ active=False,
+ )
+
+ # Override some attributes
+ attributes = {
+ "username": "override_username",
+ "active": True,
+ }
+
+ created = UserDAO.create(item=user, attributes=attributes)
+ assert created is not None
+ assert created.username == "override_username" # Should be overridden
+ assert created.active is True # Should be overridden
+ assert created.first_name == "Item" # Should keep original
+ assert created.last_name == "User" # Should keep original
+
+ db.session.commit()
+
+
+def test_base_dao_update_with_item(app_context: Session) -> None:
+ """Test BaseDAO.update with an item parameter."""
+ # Create a user first
+ user = User(
+ username="update_test",
+ first_name="Original",
+ last_name="User",
+ email="original@example.com",
+ active=True,
+ )
+ db.session.add(user)
+ db.session.commit()
+
+ # Update the user
+ user.first_name = "Updated"
+ updated = UserDAO.update(item=user)
+ assert updated is not None
+ assert updated.first_name == "Updated"
+
+ db.session.commit()
+
+ # Verify the update persisted
+ found = UserDAO.find_by_id(user.id, skip_base_filter=True)
+ assert found is not None
+ assert found.first_name == "Updated"
+
+
+def test_base_dao_update_with_attributes(app_context: Session) -> None:
+ """Test BaseDAO.update with attributes parameter."""
+ # Create a user first
+ user = User(
+ username="update_attrs_test",
+ first_name="Original",
+ last_name="User",
+ email="original@example.com",
+ active=True,
+ )
+ db.session.add(user)
+ db.session.commit()
+
+ # Update using attributes
+ attributes = {"first_name": "Updated", "last_name": "Attr User"}
+ updated = UserDAO.update(item=user, attributes=attributes)
+ assert updated is not None
+ assert updated.first_name == "Updated"
+ assert updated.last_name == "Attr User"
+
+ db.session.commit()
+
+
+def test_base_dao_update_detached_item(app_context: Session) -> None:
+ """Test BaseDAO.update with a detached item."""
+ # Create a user first
+ user = User(
+ username="detached_test",
+ first_name="Original",
+ last_name="User",
+ email="detached@example.com",
+ active=True,
+ )
+ db.session.add(user)
+ db.session.commit()
+
+ user_id = user.id
+
+ # Expunge to detach from session
+ db.session.expunge(user)
Review Comment:
1. Real-world scenario: In web applications, objects often become
detached between requests. For example:
- Object retrieved in one request, modified in another
- Objects stored in cache/session and later updated
- Background tasks that operate on objects outside the request
context
2. SQLAlchemy complexity: Detached objects are a common source of bugs
with SQLAlchemy. Without proper handling, you get errors like:
- DetachedInstanceError
- InvalidRequestError: Object is not bound to a Session
3. API contract: This test documents that BaseDAO.update() explicitly
supports detached objects. Removing the test would mean we're no longer
guaranteeing this behavior, which could break existing code that
relies on it.
4. Defensive programming: It ensures the DAO layer is robust
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat: Add BaseDAO improvements and test reorganization [superset]
Posted by "korbit-ai[bot] (via GitHub)" <gi...@apache.org>.
korbit-ai[bot] commented on code in PR #35018:
URL: https://github.com/apache/superset/pull/35018#discussion_r2352445375
##########
superset/daos/base.py:
##########
@@ -251,3 +455,205 @@ def filter_by(cls, **filter_by: Any) -> list[T]:
cls.id_column_name, data_model
).apply(query, None)
return query.filter_by(**filter_by).all()
+
+ @classmethod
+ def apply_column_operators(
+ cls, query: Any, column_operators: Optional[List[ColumnOperator]] = None
+ ) -> Any:
+ """
+ Apply column operators (list of ColumnOperator) to the query using
+ ColumnOperatorEnum logic. Raises ValueError if a filter references a
+ non-existent column.
+ """
+ if not column_operators:
+ return query
+ for c in column_operators:
+ if not isinstance(c, ColumnOperator):
+ continue
+ col = c.col
+ opr = c.opr
+ value = c.value
+ if not col or not hasattr(cls.model_cls, col):
+ model_name = cls.model_cls.__name__ if cls.model_cls else "Unknown"
+ logging.error(
+ f"Invalid filter: column '{col}' does not exist on {model_name}"
+ )
+ raise ValueError(
+ f"Invalid filter: column '{col}' does not exist on {model_name}"
+ )
+ column = getattr(cls.model_cls, col)
+ try:
+ # Always use ColumnOperatorEnum's apply method
+ operator_enum = ColumnOperatorEnum(opr)
+ query = query.filter(operator_enum.apply(column, value))
+ except Exception as e:
+ logging.error(f"Error applying filter on column '{col}': {e}")
+ raise
+ return query
+
+ @classmethod
+ def get_filterable_columns_and_operators(cls) -> Dict[str, List[str]]:
+ """
+ Returns a dict mapping filterable columns (including hybrid/computed fields if
+ present) to their supported operators. Used by MCP tools to dynamically expose
+ filter options. Custom fields supported by the DAO but not present on the model
+ should be documented here.
+ """
+
+ mapper = inspect(cls.model_cls)
+ columns = {c.key: c for c in mapper.columns}
+ # Add hybrid properties
+ hybrids = {
+ name: attr
+ for name, attr in vars(cls.model_cls).items()
+ if isinstance(attr, hybrid_property)
+ }
+ # You may add custom fields here, e.g.:
+ # custom_fields = {"tags": ["eq", "in_", "like"], ...}
+ custom_fields: Dict[str, List[str]] = {}
+
+ filterable = {}
+ for name, col in columns.items():
+ if isinstance(col.type, (sa.String, sa.Text)):
+ filterable[name] = TYPE_OPERATOR_MAP["string"]
+ elif isinstance(col.type, (sa.Boolean,)):
+ filterable[name] = TYPE_OPERATOR_MAP["boolean"]
+ elif isinstance(col.type, (sa.Integer, sa.Float, sa.Numeric)):
+ filterable[name] = TYPE_OPERATOR_MAP["number"]
+ elif isinstance(col.type, (sa.DateTime, sa.Date, sa.Time)):
+ filterable[name] = TYPE_OPERATOR_MAP["datetime"]
+ else:
+ # Fallback to eq/ne/null
+ filterable[name] = ["eq", "ne", "is_null", "is_not_null"]
+ # Add hybrid properties as string fields by default
+ for name in hybrids:
+ filterable[name] = TYPE_OPERATOR_MAP["string"]
+ # Add custom fields
+ filterable.update(custom_fields)
+ return filterable
+
+ @classmethod
+ def _build_query(
+ cls,
+ column_operators: Optional[List[ColumnOperator]] = None,
+ search: Optional[str] = None,
+ search_columns: Optional[List[str]] = None,
+ custom_filters: Optional[Dict[str, BaseFilter]] = None,
+ skip_base_filter: bool = False,
+ data_model: Optional[SQLAInterface] = None,
+ ) -> Any:
+ """
+ Build a SQLAlchemy query with base filter, column operators, search, and
+ custom filters.
+ """
+ if data_model is None:
+ data_model = SQLAInterface(cls.model_cls, db.session)
+ query = data_model.session.query(cls.model_cls)
+ query = cls._apply_base_filter(
+ query, skip_base_filter=skip_base_filter, data_model=data_model
+ )
+ if search and search_columns:
+ search_filters = []
+ for column_name in search_columns:
+ if hasattr(cls.model_cls, column_name):
+ column = getattr(cls.model_cls, column_name)
+ search_filters.append(cast(column, Text).ilike(f"%{search}%"))
Review Comment:
Thank you for the clarification. You're correct that SQLAlchemy ORM handles parameter sanitization, mitigating the SQL injection risk in this case. I appreciate you providing this important context.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat: Add BaseDAO improvements and test reorganization [superset]
Posted by "bito-code-review[bot] (via GitHub)" <gi...@apache.org>.
bito-code-review[bot] commented on PR #35018:
URL: https://github.com/apache/superset/pull/35018#issuecomment-3299702064
AI Code Review is in progress (usually takes 3 to 15 minutes unless it's a very large PR).
<details>
<summary>
Bito Usage Guide
</summary>
**Commands**
Type the following command in the pull request comment and save the comment.
- `/review` - Manually triggers a full AI review.
- `/pause` - Pauses automatic reviews on this pull request.
- `/resume` - Resumes automatic reviews.
- `/resolve` - Marks all Bito-posted review comments as resolved.
- `/abort` - Cancels all in-progress reviews.
Refer to the <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/available-commands" target="_blank" rel="noopener noreferrer">documentation</a> for additional commands.
**Configuration**
This repository uses `Default Agent` You can customize the agent settings <a href="https://alpha.bito.ai/home/ai-agents/code-review-agent" target="_blank" rel="noopener noreferrer">here</a> or contact your Bito workspace admin at evan@preset.io.
**Documentation & Help**
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/getting-started/install-run-using-bito-cloud/create-or-customize-an-agent-instance" target="_blank" rel="noopener noreferrer">Customize agent settings</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/implementing-custom-code-review-rules" target="_blank" rel="noopener noreferrer">Review rules</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent" target="_blank" rel="noopener noreferrer">General documentation</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/faqs" target="_blank" rel="noopener noreferrer">FAQ</a>
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat: Add BaseDAO improvements and test reorganization [superset]
Posted by "bito-code-review[bot] (via GitHub)" <gi...@apache.org>.
bito-code-review[bot] commented on PR #35018:
URL: https://github.com/apache/superset/pull/35018#issuecomment-3293390307
AI Code Review is in progress (usually takes 3 to 15 minutes unless it's a very large PR).
<details>
<summary>
Bito Usage Guide
</summary>
**Commands**
Type the following command in the pull request comment and save the comment.
- `/review` - Manually triggers a full AI review.
- `/pause` - Pauses automatic reviews on this pull request.
- `/resume` - Resumes automatic reviews.
- `/resolve` - Marks all Bito-posted review comments as resolved.
- `/abort` - Cancels all in-progress reviews.
Refer to the <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/available-commands" target="_blank" rel="noopener noreferrer">documentation</a> for additional commands.
**Configuration**
This repository uses `Default Agent` You can customize the agent settings <a href="https://alpha.bito.ai/home/ai-agents/code-review-agent" target="_blank" rel="noopener noreferrer">here</a> or contact your Bito workspace admin at evan@preset.io.
**Documentation & Help**
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/getting-started/install-run-using-bito-cloud/create-or-customize-an-agent-instance" target="_blank" rel="noopener noreferrer">Customize agent settings</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/implementing-custom-code-review-rules" target="_blank" rel="noopener noreferrer">Review rules</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent" target="_blank" rel="noopener noreferrer">General documentation</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/faqs" target="_blank" rel="noopener noreferrer">FAQ</a>
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat: Add BaseDAO improvements and test reorganization [superset]
Posted by "bito-code-review[bot] (via GitHub)" <gi...@apache.org>.
bito-code-review[bot] commented on code in PR #35018:
URL: https://github.com/apache/superset/pull/35018#discussion_r2349800496
##########
tests/integration_tests/dao/base_dao_test.py:
##########
@@ -0,0 +1,1619 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""
+Integration tests for BaseDAO functionality.
+
+This module contains comprehensive integration tests for the BaseDAO class and its
+subclasses, covering database operations, CRUD methods, flexible column support,
+column operators, and error handling.
+
+Tests use an in-memory SQLite database for isolation and to replicate the unit test
+environment behavior. User model deletions are avoided due to circular dependency
+constraints with self-referential foreign keys.
+"""
+
+import datetime
+import time
+import uuid
+
+import pytest
+from flask_appbuilder.models.filters import BaseFilter
+from flask_appbuilder.security.sqla.models import User
+from sqlalchemy import Column, DateTime, Integer, String
+from sqlalchemy.ext.declarative import declarative_base
+from sqlalchemy.orm.session import Session
+
+from superset.daos.base import BaseDAO, ColumnOperator, ColumnOperatorEnum
+from superset.daos.chart import ChartDAO
+from superset.daos.dashboard import DashboardDAO
+from superset.daos.database import DatabaseDAO
+from superset.daos.user import UserDAO
+from superset.extensions import db
+from superset.models.core import Database
+from superset.models.dashboard import Dashboard
+from superset.models.slice import Slice
+
+# Create a test model for comprehensive testing
+Base = declarative_base()
+
+
+class ExampleModel(Base): # type: ignore
+ __tablename__ = "example_model"
+ id = Column(Integer, primary_key=True)
+ uuid = Column(String(36), unique=True, nullable=False)
+ slug = Column(String(100), unique=True)
+ name = Column(String(100))
+ code = Column(String(50), unique=True)
+ created_on = Column(DateTime, default=datetime.datetime.utcnow)
+
+
+class ExampleModelDAO(BaseDAO[ExampleModel]):
+ model_cls = ExampleModel
+ id_column_name = "id"
+ base_filter = None
+
+
+class MockModel:
+ def __init__(self, id=1, name="test"):
+ self.id = id
+ self.name = name
+
+
+class TestDAO(BaseDAO[MockModel]):
+ model_cls = MockModel
Review Comment:
<div>
<div id="suggestion">
<div id="issue"><b>Invalid model type</b></div>
<div id="fix">
The MockModel class is a plain Python class, not a SQLAlchemy model, but TestDAO inherits from BaseDAO[MockModel] which expects a SQLAlchemy model class. This will cause runtime errors when BaseDAO tries to use SQLAlchemy operations on MockModel.
</div>
<details>
<summary>
<b>Code suggestion</b>
</summary>
<blockquote>Check the AI-generated fix before applying</blockquote>
<div id="code">
```suggestion
```
</div>
</details>
</div>
<small><i>Code Review Run <a href=https://github.com/apache/superset/pull/35018#issuecomment-3293390307>#8dadc1</a></i></small>
</div>
---
Should Bito avoid suggestions like this for future reviews? (<a href=https://alpha.bito.ai/home/ai-agents/review-rules>Manage Rules</a>)
- [ ] Yes, avoid them
##########
superset/daos/chart.py:
##########
@@ -35,10 +35,25 @@
logger = logging.getLogger(__name__)
+# Custom filterable fields for charts
+CHART_CUSTOM_FIELDS = {
+ "tags": ["eq", "in_", "like"],
+ "owner": ["eq", "in_"],
+ "viz_type": ["eq", "in_", "like"],
+ "datasource_name": ["eq", "in_", "like"],
+}
Review Comment:
<div>
<div id="suggestion">
<div id="issue"><b>Field name mismatch</b></div>
<div id="fix">
The CHART_CUSTOM_FIELDS mapping uses "owner" as a key, but the actual relationship in the Slice model is named "owners" (plural). This mismatch will cause filtering by owner to fail since the filtering system won't be able to resolve the "owner" field to the actual "owners" relationship defined in the Slice model. Change "owner" to "owners" to match the actual relationship name.
</div>
<details>
<summary>
<b>Code suggestion</b>
</summary>
<blockquote>Check the AI-generated fix before applying</blockquote>
<div id="code">
```suggestion
# Custom filterable fields for charts
CHART_CUSTOM_FIELDS = {
"tags": ["eq", "in_", "like"],
"owners": ["eq", "in_"],
"viz_type": ["eq", "in_", "like"],
"datasource_name": ["eq", "in_", "like"],
}
```
</div>
</details>
</div>
<small><i>Code Review Run <a href=https://github.com/apache/superset/pull/35018#issuecomment-3293390307>#8dadc1</a></i></small>
</div>
---
Should Bito avoid suggestions like this for future reviews? (<a href=https://alpha.bito.ai/home/ai-agents/review-rules>Manage Rules</a>)
- [ ] Yes, avoid them
##########
tests/integration_tests/dao/base_dao_test.py:
##########
@@ -0,0 +1,1619 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""
+Integration tests for BaseDAO functionality.
+
+This module contains comprehensive integration tests for the BaseDAO class and its
+subclasses, covering database operations, CRUD methods, flexible column support,
+column operators, and error handling.
+
+Tests use an in-memory SQLite database for isolation and to replicate the unit test
+environment behavior. User model deletions are avoided due to circular dependency
+constraints with self-referential foreign keys.
+"""
+
+import datetime
+import time
+import uuid
+
+import pytest
+from flask_appbuilder.models.filters import BaseFilter
+from flask_appbuilder.security.sqla.models import User
+from sqlalchemy import Column, DateTime, Integer, String
+from sqlalchemy.ext.declarative import declarative_base
+from sqlalchemy.orm.session import Session
+
+from superset.daos.base import BaseDAO, ColumnOperator, ColumnOperatorEnum
+from superset.daos.chart import ChartDAO
+from superset.daos.dashboard import DashboardDAO
+from superset.daos.database import DatabaseDAO
+from superset.daos.user import UserDAO
+from superset.extensions import db
+from superset.models.core import Database
+from superset.models.dashboard import Dashboard
+from superset.models.slice import Slice
+
+# Create a test model for comprehensive testing
+Base = declarative_base()
+
+
+class ExampleModel(Base): # type: ignore
+ __tablename__ = "example_model"
+ id = Column(Integer, primary_key=True)
+ uuid = Column(String(36), unique=True, nullable=False)
+ slug = Column(String(100), unique=True)
+ name = Column(String(100))
+ code = Column(String(50), unique=True)
+ created_on = Column(DateTime, default=datetime.datetime.utcnow)
+
+
+class ExampleModelDAO(BaseDAO[ExampleModel]):
+ model_cls = ExampleModel
+ id_column_name = "id"
+ base_filter = None
+
+
+class MockModel:
+ def __init__(self, id=1, name="test"):
+ self.id = id
+ self.name = name
+
+
+class TestDAO(BaseDAO[MockModel]):
+ model_cls = MockModel
+
+
+@pytest.fixture(autouse=True)
+def mock_g_user(app_context):
+ """Mock the flask g.user for security context."""
+ # Within app context, we can safely mock g
+ from flask import g
+
+ mock_user = User()
+ mock_user.id = 1
+ mock_user.username = "test_user"
+
+ # Set g.user directly instead of patching
+ g.user = mock_user
+ yield
+
+ # Clean up
+ if hasattr(g, "user"):
+ delattr(g, "user")
+
+
+# =============================================================================
+# Integration Tests - These tests use the actual database
+# =============================================================================
+
+
+def test_column_operator_enum_complete_coverage(user_with_data: Session) -> None:
+ """
+ Test that every single ColumnOperatorEnum operator is covered by tests.
+ This ensures we have comprehensive test coverage for all operators.
+ """
+ # Simply verify that we can create queries with all operators
+ for operator in ColumnOperatorEnum:
+ column_operator = ColumnOperator(
+ col="username", opr=operator, value="test_value"
+ )
+ # Just check it doesn't raise an error
+ assert column_operator.opr == operator
+
+
+def test_find_by_id_with_default_column(app_context: Session) -> None:
+ """Test find_by_id with default 'id' column."""
+ # Create a user to test with
+ user = User(
+ username="test_find_by_id",
+ first_name="Test",
+ last_name="User",
+ email="test@example.com",
+ active=True,
+ )
+ db.session.add(user)
+ db.session.commit()
+
+ # Find by numeric id
+ found = UserDAO.find_by_id(user.id, skip_base_filter=True)
+ assert found is not None
+ assert found.id == user.id
+ assert found.username == "test_find_by_id"
+
+ # Test with non-existent id
+ not_found = UserDAO.find_by_id(999999, skip_base_filter=True)
+ assert not_found is None
+
+
+def test_find_by_id_with_uuid_column(app_context: Session) -> None:
+ """Test find_by_id with custom uuid column."""
+ # Create a dashboard with uuid
+ dashboard = Dashboard(
+ dashboard_title="Test UUID Dashboard",
+ slug="test-uuid-dashboard",
+ published=True,
+ )
+ db.session.add(dashboard)
+ db.session.commit()
+
+ # Find by uuid string using the uuid column
+ found = DashboardDAO.find_by_id(
+ str(dashboard.uuid), id_column="uuid", skip_base_filter=True
+ )
+ assert found is not None
+ assert found.uuid == dashboard.uuid
+ assert found.dashboard_title == "Test UUID Dashboard"
+
+ # Find by numeric id (should still work)
+ found_by_id = DashboardDAO.find_by_id(dashboard.id, skip_base_filter=True)
+ assert found_by_id is not None
+ assert found_by_id.id == dashboard.id
+
+ # Test with non-existent uuid
+ not_found = DashboardDAO.find_by_id(str(uuid.uuid4()), skip_base_filter=True)
+ assert not_found is None
+
+
+def test_find_by_id_with_slug_column(app_context: Session) -> None:
+ """Test find_by_id with slug column fallback."""
+ # Create a dashboard with slug
+ dashboard = Dashboard(
+ dashboard_title="Test Slug Dashboard",
+ slug="test-slug-dashboard",
+ published=True,
+ )
+ db.session.add(dashboard)
+ db.session.commit()
+
+ # Find by slug using the slug column
+ found = DashboardDAO.find_by_id(
+ "test-slug-dashboard", id_column="slug", skip_base_filter=True
+ )
+ assert found is not None
+ assert found.slug == "test-slug-dashboard"
+ assert found.dashboard_title == "Test Slug Dashboard"
+
+ # Test with non-existent slug
+ not_found = DashboardDAO.find_by_id("non-existent-slug", skip_base_filter=True)
+ assert not_found is None
+
+
+def test_find_by_id_with_invalid_column(app_context: Session) -> None:
+ """Test find_by_id returns None when column doesn't exist."""
+ # This should return None gracefully
+ result = UserDAO.find_by_id("not_a_valid_id", skip_base_filter=True)
+ assert result is None
+
+
+def test_find_by_id_skip_base_filter(app_context: Session) -> None:
+ """Test find_by_id with skip_base_filter parameter."""
+ # Create users with different active states
+ active_user = User(
+ username="active_user",
+ first_name="Active",
+ last_name="User",
+ email="active@example.com",
+ active=True,
+ )
+ inactive_user = User(
+ username="inactive_user",
+ first_name="Inactive",
+ last_name="User",
+ email="inactive@example.com",
+ active=False,
+ )
+ db.session.add_all([active_user, inactive_user])
+ db.session.commit()
+
+ # Without skipping base filter (if one exists)
+ found_active = UserDAO.find_by_id(active_user.id, skip_base_filter=False)
+ assert found_active is not None
+
+ # With skipping base filter
+ found_active_skip = UserDAO.find_by_id(active_user.id, skip_base_filter=True)
+ assert found_active_skip is not None
+
+ # Both should find the user since UserDAO might not have a base filter
+ assert found_active.id == active_user.id
+ assert found_active_skip.id == active_user.id
+
+
+def test_find_by_ids_with_default_column(app_context: Session) -> None:
+ """Test find_by_ids with default 'id' column."""
+ # Create multiple users
+ users = []
+ for i in range(3):
+ user = User(
+ username=f"test_find_by_ids_{i}",
+ first_name=f"Test{i}",
+ last_name="User",
+ email=f"test{i}@example.com",
+ active=True,
+ )
+ users.append(user)
+ db.session.add(user)
+ db.session.commit()
+
+ # Find by multiple ids
+ ids = [user.id for user in users]
+ found = UserDAO.find_by_ids(ids, skip_base_filter=True)
+ assert len(found) == 3
+ found_ids = [u.id for u in found]
+ assert set(found_ids) == set(ids)
+
+ # Test with mix of existent and non-existent ids
+ mixed_ids = [users[0].id, 999999, users[1].id]
+ found_mixed = UserDAO.find_by_ids(mixed_ids, skip_base_filter=True)
+ assert len(found_mixed) == 2
+
+ # Test with empty list
+ found_empty = UserDAO.find_by_ids([], skip_base_filter=True)
+ assert found_empty == []
+
+
+def test_find_by_ids_with_uuid_column(app_context: Session) -> None:
+ """Test find_by_ids with uuid column."""
+ # Create multiple dashboards
+ dashboards = []
+ for i in range(3):
+ dashboard = Dashboard(
+ dashboard_title=f"Test UUID Dashboard {i}",
+ slug=f"test-uuid-dashboard-{i}",
+ published=True,
+ )
+ dashboards.append(dashboard)
+ db.session.add(dashboard)
+ db.session.commit()
+
+ # Find by multiple uuids
+ uuids = [str(dashboard.uuid) for dashboard in dashboards]
+ found = DashboardDAO.find_by_ids(uuids, id_column="uuid", skip_base_filter=True)
+ assert len(found) == 3
+ found_uuids = [str(d.uuid) for d in found]
+ assert set(found_uuids) == set(uuids)
+
+ # Test with mix of ids and uuids - search separately by column
+ found_by_id = DashboardDAO.find_by_ids([dashboards[0].id], skip_base_filter=True)
+ found_by_uuid = DashboardDAO.find_by_ids(
+ [str(dashboards[1].uuid)], id_column="uuid", skip_base_filter=True
+ )
+ assert len(found_by_id) == 1
+ assert len(found_by_uuid) == 1
+
+
+def test_find_by_ids_with_slug_column(app_context: Session) -> None:
+ """Test find_by_ids with slug column."""
+ # Create multiple dashboards
+ dashboards = []
+ for i in range(3):
+ dashboard = Dashboard(
+ dashboard_title=f"Test Slug Dashboard {i}",
+ slug=f"test-slug-dashboard-{i}",
+ published=True,
+ )
+ dashboards.append(dashboard)
+ db.session.add(dashboard)
+ db.session.commit()
+
+ # Find by multiple slugs
+ slugs = [dashboard.slug for dashboard in dashboards]
+ found = DashboardDAO.find_by_ids(slugs, id_column="slug", skip_base_filter=True)
+ assert len(found) == 3
+ found_slugs = [d.slug for d in found]
+ assert set(found_slugs) == set(slugs)
+
+
+def test_find_by_ids_with_invalid_column(app_context: Session) -> None:
+ """Test find_by_ids returns empty list when column doesn't exist."""
+ # This should return empty list gracefully
+ result = UserDAO.find_by_ids(["not_a_valid_id"], skip_base_filter=True)
+ assert result == []
+
+
+def test_find_by_ids_skip_base_filter(app_context: Session) -> None:
+ """Test find_by_ids with skip_base_filter parameter."""
+ # Create users
+ users = []
+ for i in range(3):
+ user = User(
+ username=f"test_skip_filter_{i}",
+ first_name=f"Test{i}",
+ last_name="User",
+ email=f"test{i}@example.com",
+ active=True,
+ )
+ users.append(user)
+ db.session.add(user)
+ db.session.commit()
+
+ ids = [user.id for user in users]
+
+ # Without skipping base filter
+ found_no_skip = UserDAO.find_by_ids(ids, skip_base_filter=False)
+ assert len(found_no_skip) == 3
+
+ # With skipping base filter
+ found_skip = UserDAO.find_by_ids(ids, skip_base_filter=True)
+ assert len(found_skip) == 3
+
+
+def test_base_dao_create_with_item(app_context: Session) -> None:
+ """Test BaseDAO.create with an item parameter."""
+ # Create a user item
+ user = User(
+ username="created_with_item",
+ first_name="Created",
+ last_name="Item",
+ email="created@example.com",
+ active=True,
+ )
+
+ # Create using the item
+ created = UserDAO.create(item=user)
+ assert created is not None
+ assert created.username == "created_with_item"
+ assert created.first_name == "Created"
+
+ # Verify it's in the session
+ assert created in db.session
+
+ # Commit and verify it persists
+ db.session.commit()
+
+ # Find it again to ensure it was saved
+ found = UserDAO.find_by_id(created.id, skip_base_filter=True)
+ assert found is not None
+ assert found.username == "created_with_item"
+
+
+def test_base_dao_create_with_attributes(app_context: Session) -> None:
+ """Test BaseDAO.create with attributes parameter."""
+ # Create using attributes dict
+ attributes = {
+ "username": "created_with_attrs",
+ "first_name": "Created",
+ "last_name": "Attrs",
+ "email": "attrs@example.com",
+ "active": True,
+ }
+
+ created = UserDAO.create(attributes=attributes)
+ assert created is not None
+ assert created.username == "created_with_attrs"
+ assert created.email == "attrs@example.com"
+
+ # Commit and verify
+ db.session.commit()
+ found = UserDAO.find_by_id(created.id, skip_base_filter=True)
+ assert found is not None
+ assert found.username == "created_with_attrs"
+
+
+def test_base_dao_create_with_both_item_and_attributes(app_context: Session) -> None:
+ """Test BaseDAO.create with both item and attributes (override behavior)."""
+ # Create a user item
+ user = User(
+ username="item_username",
+ first_name="Item",
+ last_name="User",
+ email="item@example.com",
+ active=False,
+ )
+
+ # Override some attributes
+ attributes = {
+ "username": "override_username",
+ "active": True,
+ }
+
+ created = UserDAO.create(item=user, attributes=attributes)
+ assert created is not None
+ assert created.username == "override_username" # Should be overridden
+ assert created.active is True # Should be overridden
+ assert created.first_name == "Item" # Should keep original
+ assert created.last_name == "User" # Should keep original
+
+ db.session.commit()
+
+
+def test_base_dao_update_with_item(app_context: Session) -> None:
+ """Test BaseDAO.update with an item parameter."""
+ # Create a user first
+ user = User(
+ username="update_test",
+ first_name="Original",
+ last_name="User",
+ email="original@example.com",
+ active=True,
+ )
+ db.session.add(user)
+ db.session.commit()
+
+ # Update the user
+ user.first_name = "Updated"
+ updated = UserDAO.update(item=user)
+ assert updated is not None
+ assert updated.first_name == "Updated"
+
+ db.session.commit()
+
+ # Verify the update persisted
+ found = UserDAO.find_by_id(user.id, skip_base_filter=True)
+ assert found is not None
+ assert found.first_name == "Updated"
+
+
+def test_base_dao_update_with_attributes(app_context: Session) -> None:
+ """Test BaseDAO.update with attributes parameter."""
+ # Create a user first
+ user = User(
+ username="update_attrs_test",
+ first_name="Original",
+ last_name="User",
+ email="original@example.com",
+ active=True,
+ )
+ db.session.add(user)
+ db.session.commit()
+
+ # Update using attributes
+ attributes = {"first_name": "Updated", "last_name": "Attr User"}
+ updated = UserDAO.update(item=user, attributes=attributes)
+ assert updated is not None
+ assert updated.first_name == "Updated"
+ assert updated.last_name == "Attr User"
+
+ db.session.commit()
+
+
+def test_base_dao_update_detached_item(app_context: Session) -> None:
+ """Test BaseDAO.update with a detached item."""
+ # Create a user first
+ user = User(
+ username="detached_test",
+ first_name="Original",
+ last_name="User",
+ email="detached@example.com",
+ active=True,
+ )
+ db.session.add(user)
+ db.session.commit()
+
+ user_id = user.id
+
+ # Expunge to detach from session
+ db.session.expunge(user)
+
+ # Update the detached user
+ user.first_name = "Updated Detached"
+ updated = UserDAO.update(item=user)
+ assert updated is not None
+ assert updated.first_name == "Updated Detached"
+
+ db.session.commit()
+
+ # Verify the update persisted
+ found = UserDAO.find_by_id(user_id, skip_base_filter=True)
+ assert found is not None
+ assert found.first_name == "Updated Detached"
+
+
+def test_base_dao_delete_single_item(app_context: Session) -> None:
+ """Test BaseDAO.delete with a single item."""
+ # Create a dashboard instead of user to avoid circular dependencies
+ dashboard = Dashboard(
+ dashboard_title="Delete Test",
+ slug="delete-test",
+ published=True,
+ )
+ db.session.add(dashboard)
+ db.session.commit()
+
+ dashboard_id = dashboard.id
+
+ DashboardDAO.delete([dashboard])
+ db.session.commit()
+
+ # Verify it's gone
+ found = DashboardDAO.find_by_id(dashboard_id, skip_base_filter=True)
+ assert found is None
+
+
+def test_base_dao_delete_multiple_items(app_context: Session) -> None:
+ """Test BaseDAO.delete with multiple items."""
+ # Create multiple dashboards instead of users to avoid circular dependencies
+ dashboards = []
+ for i in range(3):
+ dashboard = Dashboard(
+ dashboard_title=f"Delete Multi {i}",
+ slug=f"delete-multi-{i}",
+ published=True,
+ )
+ dashboards.append(dashboard)
+ db.session.add(dashboard)
+ db.session.commit()
+
+ dashboard_ids = [dashboard.id for dashboard in dashboards]
+
+ DashboardDAO.delete(dashboards)
+ db.session.commit()
+
+ # Verify they're all gone
+ for dashboard_id in dashboard_ids:
+ found = DashboardDAO.find_by_id(dashboard_id, skip_base_filter=True)
+ assert found is None
+
+
+def test_base_dao_delete_empty_list(app_context: Session) -> None:
+ """Test BaseDAO.delete with empty list."""
+ # Should not raise any errors
+ UserDAO.delete([])
+ db.session.commit()
+ # Just ensuring no exception is raised
+
+
+def test_base_dao_find_all(app_context: Session) -> None:
+ """Test BaseDAO.find_all method."""
+ # Create some users
+ users = []
+ for i in range(3):
+ user = User(
+ username=f"find_all_{i}",
+ first_name=f"Find{i}",
+ last_name="All",
+ email=f"findall{i}@example.com",
+ active=True,
+ )
+ users.append(user)
+ db.session.add(user)
+ db.session.commit()
+
+ # Find all users
+ all_users = UserDAO.find_all()
+ assert len(all_users) >= 3 # At least our 3 users
+
+ # Check our users are in the results
+ usernames = [u.username for u in all_users]
+ for user in users:
+ assert user.username in usernames
+
+
+def test_base_dao_find_one_or_none(app_context: Session) -> None:
+ """Test BaseDAO.find_one_or_none method."""
+ # Create users with specific criteria
+ user1 = User(
+ username="unique_username_123",
+ first_name="Unique",
+ last_name="User",
+ email="unique@example.com",
+ active=True,
+ )
+ user2 = User(
+ username="another_user",
+ first_name="Another",
+ last_name="User",
+ email="another@example.com",
+ active=True,
+ )
+ user3 = User(
+ username="third_user",
+ first_name="Another", # Same first name as user2
+ last_name="User",
+ email="third@example.com",
+ active=True,
+ )
+ db.session.add_all([user1, user2, user3])
+ db.session.commit()
+
+ # Find one with unique criteria
+ found = UserDAO.find_one_or_none(username="unique_username_123")
+ assert found is not None
+ assert found.username == "unique_username_123"
+
+ # Find none with non-existent criteria
+ not_found = UserDAO.find_one_or_none(username="non_existent_user")
+ assert not_found is None
+
+ db.session.commit()
+
+
+def test_base_dao_list_returns_results(user_with_data: Session) -> None:
+ """Test that BaseDAO.list returns results and total."""
+ results, total = UserDAO.list()
+ assert isinstance(results, list)
+ assert isinstance(total, int)
+ assert total >= 1 # At least the fixture user
+
+
+def test_base_dao_list_with_column_operators(user_with_data: Session) -> None:
+ """Test BaseDAO.list with column operators."""
+ column_operators = [
+ ColumnOperator(col="username", opr=ColumnOperatorEnum.eq, value="testuser")
+ ]
+ results, total = UserDAO.list(column_operators=column_operators)
+ assert total == 1
+ assert results[0].username == "testuser"
+
+
+def test_base_dao_list_with_non_matching_column_operator(
+ user_with_data: Session,
+) -> None:
+ """Test BaseDAO.list with non-matching column operator."""
+ column_operators = [
+ ColumnOperator(
+ col="username", opr=ColumnOperatorEnum.eq, value="nonexistentuser"
+ )
+ ]
+ results, total = UserDAO.list(column_operators=column_operators)
+ assert total == 0
+ assert results == []
+
+
+def test_base_dao_count_returns_value(user_with_data: Session) -> None:
+ """Test that BaseDAO.count returns correct count."""
+ count = UserDAO.count()
+ assert isinstance(count, int)
+ assert count >= 1 # At least the fixture user
+
+
+def test_base_dao_count_with_column_operators(user_with_data: Session) -> None:
+ """Test BaseDAO.count with column operators."""
+ # Count with matching operator
+ column_operators = [
+ ColumnOperator(col="username", opr=ColumnOperatorEnum.eq, value="testuser")
+ ]
+ count = UserDAO.count(column_operators=column_operators)
+ assert count == 1
+
+ # Count with non-matching operator
+ column_operators = [
+ ColumnOperator(
+ col="username", opr=ColumnOperatorEnum.eq, value="nonexistentuser"
+ )
+ ]
+ count = UserDAO.count(column_operators=column_operators)
+ assert count == 0
+
+
+def test_base_dao_list_ordering(user_with_data: Session) -> None:
+ """Test BaseDAO.list with ordering."""
+ # Create additional users with predictable names
+ users = []
+ for i in range(3):
+ user = User(
+ id=200 + i, # Ensure predictable IDs
Review Comment:
<div>
<div id="suggestion">
<div id="issue"><b>Primary key conflict risk</b></div>
<div id="fix">
Hard-coding user IDs (200, 201, 202) in test data creation can cause primary key conflicts in environments where these IDs already exist. This will break the test and potentially corrupt test databases. Use auto-generated IDs or check for existing records instead.
</div>
<details>
<summary>
<b>Code suggestion</b>
</summary>
<blockquote>Check the AI-generated fix before applying</blockquote>
<div id="code">
```suggestion
# Let database auto-generate IDs to avoid conflicts
```
</div>
</details>
</div>
<small><i>Code Review Run <a href=https://github.com/apache/superset/pull/35018#issuecomment-3293390307>#8dadc1</a></i></small>
</div>
---
Should Bito avoid suggestions like this for future reviews? (<a href=https://alpha.bito.ai/home/ai-agents/review-rules>Manage Rules</a>)
- [ ] Yes, avoid them
##########
pyproject.toml:
##########
@@ -84,6 +84,7 @@ dependencies = [
"pgsanity",
"Pillow>=11.0.0, <12",
"polyline>=2.0.0, <3.0",
+ "pydantic>=2.8.0",
Review Comment:
<div>
<div id="suggestion">
<div id="issue"><b>Missing version upper bound</b></div>
<div id="fix">
The pydantic dependency is added without an upper version bound, which could allow installation of pydantic v3.x when released. Since pydantic v2 introduced breaking changes from v1, future v3.x could introduce additional breaking changes. This would break the `from pydantic import BaseModel, Field` usage in `superset/daos/base.py` and potentially other pydantic-based code. Add upper bound `<3.0.0` to prevent breaking changes from future major versions.
</div>
<details>
<summary>
<b>Code suggestion</b>
</summary>
<blockquote>Check the AI-generated fix before applying</blockquote>
<div id="code">
```suggestion
"pydantic>=2.8.0,<3.0.0"
```
</div>
</details>
</div>
<small><i>Code Review Run <a href=https://github.com/apache/superset/pull/35018#issuecomment-3293390307>#8dadc1</a></i></small>
</div>
---
Should Bito avoid suggestions like this for future reviews? (<a href=https://alpha.bito.ai/home/ai-agents/review-rules>Manage Rules</a>)
- [ ] Yes, avoid them
##########
superset/daos/dataset.py:
##########
@@ -35,8 +35,21 @@
logger = logging.getLogger(__name__)
+# Custom filterable fields for datasets
+DATASET_CUSTOM_FIELDS = {
+ "tags": ["eq", "in_", "like"],
+ "owner": ["eq", "in_"],
+}
Review Comment:
<div>
<div id="suggestion">
<div id="issue"><b>Invalid custom field references</b></div>
<div id="fix">
The custom fields "tags" and "owner" being added to DATASET_CUSTOM_FIELDS do not exist on the SqlaTable model. The SqlaTable model has "owners" (plural) as a relationship field, but no "owner" (singular) or "tags" field. This will cause filtering failures when these custom fields are used. Update the custom fields to use "owners" instead of "owner" and remove "tags" since it doesn't exist.
</div>
<details>
<summary>
<b>Code suggestion</b>
</summary>
<blockquote>Check the AI-generated fix before applying</blockquote>
<div id="code">
```suggestion
# Custom filterable fields for datasets
DATASET_CUSTOM_FIELDS = {
"owners": ["eq", "in_"],
}
```
</div>
</details>
</div>
<small><i>Code Review Run <a href=https://github.com/apache/superset/pull/35018#issuecomment-3293390307>#8dadc1</a></i></small>
</div>
---
Should Bito avoid suggestions like this for future reviews? (<a href=https://alpha.bito.ai/home/ai-agents/review-rules>Manage Rules</a>)
- [ ] Yes, avoid them
##########
superset/daos/base.py:
##########
@@ -16,22 +16,136 @@
# under the License.
from __future__ import annotations
-from typing import Any, Generic, get_args, TypeVar
+import logging
+import uuid as uuid_lib
+from enum import Enum
+from typing import (
+ Any,
+ Dict,
+ Generic,
+ get_args,
+ List,
+ Optional,
+ Sequence,
+ Tuple,
+ TypeVar,
+)
+import sqlalchemy as sa
from flask_appbuilder.models.filters import BaseFilter
from flask_appbuilder.models.sqla import Model
from flask_appbuilder.models.sqla.interface import SQLAInterface
from flask_sqlalchemy import BaseQuery
+from pydantic import BaseModel, Field
+from sqlalchemy import asc, cast, desc, or_, Text
from sqlalchemy.exc import SQLAlchemyError, StatementError
+from sqlalchemy.ext.hybrid import hybrid_property
+from sqlalchemy.inspection import inspect
+from sqlalchemy.orm import ColumnProperty, joinedload, RelationshipProperty
from superset.daos.exceptions import (
DAOFindFailedError,
)
from superset.extensions import db
+logger = logging.getLogger(__name__)
+
T = TypeVar("T", bound=Model)
+class ColumnOperatorEnum(str, Enum):
+ eq = "eq"
+ ne = "ne"
+ sw = "sw"
+ ew = "ew"
+ in_ = "in"
+ nin = "nin"
+ gt = "gt"
+ gte = "gte"
+ lt = "lt"
+ lte = "lte"
+ like = "like"
+ ilike = "ilike"
+ is_null = "is_null"
+ is_not_null = "is_not_null"
+
+ def apply(self, column: Any, value: Any) -> Any:
+ op_func = operator_map.get(self)
+ if not op_func:
+ raise ValueError("Unsupported operator: %s" % self)
+ return op_func(column, value)
+
+
+# Define operator_map as a module-level dict after the enum is defined
+operator_map: Dict[ColumnOperatorEnum, Any] = {
+ ColumnOperatorEnum.eq: lambda col, val: col == val,
+ ColumnOperatorEnum.ne: lambda col, val: col != val,
+ ColumnOperatorEnum.sw: lambda col, val: col.like(f"{val}%"),
+ ColumnOperatorEnum.ew: lambda col, val: col.like(f"%{val}"),
+ ColumnOperatorEnum.in_: lambda col, val: col.in_(
+ val if isinstance(val, (list, tuple)) else [val]
+ ),
+ ColumnOperatorEnum.nin: lambda col, val: ~col.in_(
+ val if isinstance(val, (list, tuple)) else [val]
+ ),
+ ColumnOperatorEnum.gt: lambda col, val: col > val,
+ ColumnOperatorEnum.gte: lambda col, val: col >= val,
+ ColumnOperatorEnum.lt: lambda col, val: col < val,
+ ColumnOperatorEnum.lte: lambda col, val: col <= val,
+ ColumnOperatorEnum.like: lambda col, val: col.like(f"%{val}%"),
+ ColumnOperatorEnum.ilike: lambda col, val: col.ilike(f"%{val}%"),
+ ColumnOperatorEnum.is_null: lambda col, _: col.is_(None),
+ ColumnOperatorEnum.is_not_null: lambda col, _: col.isnot(None),
+}
+
+# Map SQLAlchemy types to supported operators
+TYPE_OPERATOR_MAP = {
+ "string": [
+ "eq",
+ "ne",
+ "sw",
+ "ew",
+ "in_",
+ "nin",
+ "like",
+ "ilike",
+ "is_null",
+ "is_not_null",
+ ],
+ "boolean": ["eq", "ne", "is_null", "is_not_null"],
+ "number": [
+ "eq",
+ "ne",
+ "gt",
+ "gte",
+ "lt",
+ "lte",
+ "in_",
+ "nin",
+ "is_null",
+ "is_not_null",
+ ],
+ "datetime": [
+ "eq",
+ "ne",
+ "gt",
+ "gte",
+ "lt",
+ "lte",
+ "in_",
+ "nin",
+ "is_null",
+ "is_not_null",
+ ],
+}
Review Comment:
<div>
<div id="suggestion">
<div id="issue"><b>Type mismatch in operator map</b></div>
<div id="fix">
The TYPE_OPERATOR_MAP is defined with string literals instead of ColumnOperatorEnum values, causing a type mismatch. This will break the filtering system when these values are used with ColumnOperatorEnum.apply(). The TYPE_OPERATOR_MAP should contain ColumnOperatorEnum values to maintain type consistency with the rest of the codebase.
</div>
<details>
<summary>
<b>Code suggestion</b>
</summary>
<blockquote>Check the AI-generated fix before applying</blockquote>
<div id="code">
```suggestion
TYPE_OPERATOR_MAP = {
"string": [
ColumnOperatorEnum.eq,
ColumnOperatorEnum.ne,
ColumnOperatorEnum.sw,
ColumnOperatorEnum.ew,
ColumnOperatorEnum.in_,
ColumnOperatorEnum.nin,
ColumnOperatorEnum.like,
ColumnOperatorEnum.ilike,
ColumnOperatorEnum.is_null,
ColumnOperatorEnum.is_not_null,
],
"boolean": [ColumnOperatorEnum.eq, ColumnOperatorEnum.ne, ColumnOperatorEnum.is_null, ColumnOperatorEnum.is_not_null],
"number": [
ColumnOperatorEnum.eq,
ColumnOperatorEnum.ne,
ColumnOperatorEnum.gt,
ColumnOperatorEnum.gte,
ColumnOperatorEnum.lt,
ColumnOperatorEnum.lte,
ColumnOperatorEnum.in_,
ColumnOperatorEnum.nin,
ColumnOperatorEnum.is_null,
ColumnOperatorEnum.is_not_null,
],
"datetime": [
ColumnOperatorEnum.eq,
ColumnOperatorEnum.ne,
ColumnOperatorEnum.gt,
ColumnOperatorEnum.gte,
ColumnOperatorEnum.lt,
ColumnOperatorEnum.lte,
ColumnOperatorEnum.in_,
ColumnOperatorEnum.nin,
ColumnOperatorEnum.is_null,
ColumnOperatorEnum.is_not_null,
],
}
```
</div>
</details>
</div>
<small><i>Code Review Run <a href=https://github.com/apache/superset/pull/35018#issuecomment-3293390307>#8dadc1</a></i></small>
</div>
---
Should Bito avoid suggestions like this for future reviews? (<a href=https://alpha.bito.ai/home/ai-agents/review-rules>Manage Rules</a>)
- [ ] Yes, avoid them
##########
superset/daos/base.py:
##########
@@ -152,11 +355,7 @@
Get the first that fit the `base_filter`
"""
query = db.session.query(cls.model_cls)
- if cls.base_filter:
- data_model = SQLAInterface(cls.model_cls, db.session)
- query = cls.base_filter( # pylint: disable=not-callable
- cls.id_column_name, data_model
- ).apply(query, None)
+ query = cls._apply_base_filter(query)
Review Comment:
<div>
<div id="suggestion">
<div id="issue"><b>Inconsistent method signature</b></div>
<div id="fix">
The `find_one_or_none` method should also accept a `skip_base_filter` parameter for consistency with other DAO methods. Add `skip_base_filter: bool = False` parameter and pass it to `_apply_base_filter` to maintain uniform base filter handling across all query methods.
</div>
<details>
<summary>
<b>Code suggestion</b>
</summary>
<blockquote>Check the AI-generated fix before applying</blockquote>
<div id="code">
```
- def find_one_or_none(cls, **filter_by: Any) -> T | None:
+ def find_one_or_none(cls, skip_base_filter: bool = False, **filter_by: Any) -> T | None:
"""
Get the first that fit the `base_filter`
"""
query = db.session.query(cls.model_cls)
- query = cls._apply_base_filter(query)
+ query = cls._apply_base_filter(query, skip_base_filter)
return query.filter_by(**filter_by).one_or_none()
```
</div>
</details>
</div>
<small><i>Code Review Run <a href=https://github.com/apache/superset/pull/35018#issuecomment-3293390307>#8dadc1</a></i></small>
</div>
---
Should Bito avoid suggestions like this for future reviews? (<a href=https://alpha.bito.ai/home/ai-agents/review-rules>Manage Rules</a>)
- [ ] Yes, avoid them
##########
tests/integration_tests/dao/base_dao_test.py:
##########
@@ -0,0 +1,1619 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""
+Integration tests for BaseDAO functionality.
+
+This module contains comprehensive integration tests for the BaseDAO class and its
+subclasses, covering database operations, CRUD methods, flexible column support,
+column operators, and error handling.
+
+Tests use an in-memory SQLite database for isolation and to replicate the unit test
+environment behavior. User model deletions are avoided due to circular dependency
+constraints with self-referential foreign keys.
+"""
+
+import datetime
+import time
+import uuid
+
+import pytest
+from flask_appbuilder.models.filters import BaseFilter
+from flask_appbuilder.security.sqla.models import User
+from sqlalchemy import Column, DateTime, Integer, String
+from sqlalchemy.ext.declarative import declarative_base
+from sqlalchemy.orm.session import Session
+
+from superset.daos.base import BaseDAO, ColumnOperator, ColumnOperatorEnum
+from superset.daos.chart import ChartDAO
+from superset.daos.dashboard import DashboardDAO
+from superset.daos.database import DatabaseDAO
+from superset.daos.user import UserDAO
+from superset.extensions import db
+from superset.models.core import Database
+from superset.models.dashboard import Dashboard
+from superset.models.slice import Slice
+
+# Create a test model for comprehensive testing
+Base = declarative_base()
+
+
+class ExampleModel(Base): # type: ignore
+ __tablename__ = "example_model"
+ id = Column(Integer, primary_key=True)
+ uuid = Column(String(36), unique=True, nullable=False)
+ slug = Column(String(100), unique=True)
+ name = Column(String(100))
+ code = Column(String(50), unique=True)
+ created_on = Column(DateTime, default=datetime.datetime.utcnow)
+
+
+class ExampleModelDAO(BaseDAO[ExampleModel]):
+ model_cls = ExampleModel
+ id_column_name = "id"
+ base_filter = None
+
+
+class MockModel:
+ def __init__(self, id=1, name="test"):
+ self.id = id
+ self.name = name
+
+
+class TestDAO(BaseDAO[MockModel]):
+ model_cls = MockModel
+
+
+@pytest.fixture(autouse=True)
+def mock_g_user(app_context):
+ """Mock the flask g.user for security context."""
+ # Within app context, we can safely mock g
+ from flask import g
+
+ mock_user = User()
+ mock_user.id = 1
+ mock_user.username = "test_user"
+
+ # Set g.user directly instead of patching
+ g.user = mock_user
+ yield
+
+ # Clean up
+ if hasattr(g, "user"):
+ delattr(g, "user")
+
+
+# =============================================================================
+# Integration Tests - These tests use the actual database
+# =============================================================================
+
+
+def test_column_operator_enum_complete_coverage(user_with_data: Session) -> None:
+ """
+ Test that every single ColumnOperatorEnum operator is covered by tests.
+ This ensures we have comprehensive test coverage for all operators.
+ """
+ # Simply verify that we can create queries with all operators
+ for operator in ColumnOperatorEnum:
+ column_operator = ColumnOperator(
+ col="username", opr=operator, value="test_value"
+ )
+ # Just check it doesn't raise an error
+ assert column_operator.opr == operator
+
+
+def test_find_by_id_with_default_column(app_context: Session) -> None:
+ """Test find_by_id with default 'id' column."""
+ # Create a user to test with
+ user = User(
+ username="test_find_by_id",
+ first_name="Test",
+ last_name="User",
+ email="test@example.com",
+ active=True,
+ )
+ db.session.add(user)
+ db.session.commit()
+
+ # Find by numeric id
+ found = UserDAO.find_by_id(user.id, skip_base_filter=True)
+ assert found is not None
+ assert found.id == user.id
+ assert found.username == "test_find_by_id"
+
+ # Test with non-existent id
+ not_found = UserDAO.find_by_id(999999, skip_base_filter=True)
+ assert not_found is None
+
+
+def test_find_by_id_with_uuid_column(app_context: Session) -> None:
+ """Test find_by_id with custom uuid column."""
+ # Create a dashboard with uuid
+ dashboard = Dashboard(
+ dashboard_title="Test UUID Dashboard",
+ slug="test-uuid-dashboard",
+ published=True,
+ )
+ db.session.add(dashboard)
+ db.session.commit()
+
+ # Find by uuid string using the uuid column
+ found = DashboardDAO.find_by_id(
+ str(dashboard.uuid), id_column="uuid", skip_base_filter=True
+ )
+ assert found is not None
+ assert found.uuid == dashboard.uuid
+ assert found.dashboard_title == "Test UUID Dashboard"
+
+ # Find by numeric id (should still work)
+ found_by_id = DashboardDAO.find_by_id(dashboard.id, skip_base_filter=True)
+ assert found_by_id is not None
+ assert found_by_id.id == dashboard.id
+
+ # Test with non-existent uuid
+ not_found = DashboardDAO.find_by_id(str(uuid.uuid4()), skip_base_filter=True)
+ assert not_found is None
+
+
+def test_find_by_id_with_slug_column(app_context: Session) -> None:
+ """Test find_by_id with slug column fallback."""
+ # Create a dashboard with slug
+ dashboard = Dashboard(
+ dashboard_title="Test Slug Dashboard",
+ slug="test-slug-dashboard",
+ published=True,
+ )
+ db.session.add(dashboard)
+ db.session.commit()
+
+ # Find by slug using the slug column
+ found = DashboardDAO.find_by_id(
+ "test-slug-dashboard", id_column="slug", skip_base_filter=True
+ )
+ assert found is not None
+ assert found.slug == "test-slug-dashboard"
+ assert found.dashboard_title == "Test Slug Dashboard"
+
+ # Test with non-existent slug
+ not_found = DashboardDAO.find_by_id("non-existent-slug", skip_base_filter=True)
+ assert not_found is None
+
+
+def test_find_by_id_with_invalid_column(app_context: Session) -> None:
+ """Test find_by_id returns None when column doesn't exist."""
+ # This should return None gracefully
+ result = UserDAO.find_by_id("not_a_valid_id", skip_base_filter=True)
+ assert result is None
+
+
+def test_find_by_id_skip_base_filter(app_context: Session) -> None:
+ """Test find_by_id with skip_base_filter parameter."""
+ # Create users with different active states
+ active_user = User(
+ username="active_user",
+ first_name="Active",
+ last_name="User",
+ email="active@example.com",
+ active=True,
+ )
+ inactive_user = User(
+ username="inactive_user",
+ first_name="Inactive",
+ last_name="User",
+ email="inactive@example.com",
+ active=False,
+ )
+ db.session.add_all([active_user, inactive_user])
+ db.session.commit()
+
+ # Without skipping base filter (if one exists)
+ found_active = UserDAO.find_by_id(active_user.id, skip_base_filter=False)
+ assert found_active is not None
+
+ # With skipping base filter
+ found_active_skip = UserDAO.find_by_id(active_user.id, skip_base_filter=True)
+ assert found_active_skip is not None
+
+ # Both should find the user since UserDAO might not have a base filter
+ assert found_active.id == active_user.id
+ assert found_active_skip.id == active_user.id
+
+
+def test_find_by_ids_with_default_column(app_context: Session) -> None:
+ """Test find_by_ids with default 'id' column."""
+ # Create multiple users
+ users = []
+ for i in range(3):
+ user = User(
+ username=f"test_find_by_ids_{i}",
+ first_name=f"Test{i}",
+ last_name="User",
+ email=f"test{i}@example.com",
+ active=True,
+ )
+ users.append(user)
+ db.session.add(user)
+ db.session.commit()
+
+ # Find by multiple ids
+ ids = [user.id for user in users]
+ found = UserDAO.find_by_ids(ids, skip_base_filter=True)
+ assert len(found) == 3
+ found_ids = [u.id for u in found]
+ assert set(found_ids) == set(ids)
+
+ # Test with mix of existent and non-existent ids
+ mixed_ids = [users[0].id, 999999, users[1].id]
+ found_mixed = UserDAO.find_by_ids(mixed_ids, skip_base_filter=True)
+ assert len(found_mixed) == 2
+
+ # Test with empty list
+ found_empty = UserDAO.find_by_ids([], skip_base_filter=True)
+ assert found_empty == []
+
+
+def test_find_by_ids_with_uuid_column(app_context: Session) -> None:
+ """Test find_by_ids with uuid column."""
+ # Create multiple dashboards
+ dashboards = []
+ for i in range(3):
+ dashboard = Dashboard(
+ dashboard_title=f"Test UUID Dashboard {i}",
+ slug=f"test-uuid-dashboard-{i}",
+ published=True,
+ )
+ dashboards.append(dashboard)
+ db.session.add(dashboard)
+ db.session.commit()
+
+ # Find by multiple uuids
+ uuids = [str(dashboard.uuid) for dashboard in dashboards]
+ found = DashboardDAO.find_by_ids(uuids, id_column="uuid", skip_base_filter=True)
+ assert len(found) == 3
+ found_uuids = [str(d.uuid) for d in found]
+ assert set(found_uuids) == set(uuids)
+
+ # Test with mix of ids and uuids - search separately by column
+ found_by_id = DashboardDAO.find_by_ids([dashboards[0].id], skip_base_filter=True)
+ found_by_uuid = DashboardDAO.find_by_ids(
+ [str(dashboards[1].uuid)], id_column="uuid", skip_base_filter=True
+ )
+ assert len(found_by_id) == 1
+ assert len(found_by_uuid) == 1
+
+
+def test_find_by_ids_with_slug_column(app_context: Session) -> None:
+ """Test find_by_ids with slug column."""
+ # Create multiple dashboards
+ dashboards = []
+ for i in range(3):
+ dashboard = Dashboard(
+ dashboard_title=f"Test Slug Dashboard {i}",
+ slug=f"test-slug-dashboard-{i}",
+ published=True,
+ )
+ dashboards.append(dashboard)
+ db.session.add(dashboard)
+ db.session.commit()
+
+ # Find by multiple slugs
+ slugs = [dashboard.slug for dashboard in dashboards]
+ found = DashboardDAO.find_by_ids(slugs, id_column="slug", skip_base_filter=True)
+ assert len(found) == 3
+ found_slugs = [d.slug for d in found]
+ assert set(found_slugs) == set(slugs)
+
+
+def test_find_by_ids_with_invalid_column(app_context: Session) -> None:
+ """Test find_by_ids returns empty list when column doesn't exist."""
+ # This should return empty list gracefully
+ result = UserDAO.find_by_ids(["not_a_valid_id"], skip_base_filter=True)
+ assert result == []
+
+
+def test_find_by_ids_skip_base_filter(app_context: Session) -> None:
+ """Test find_by_ids with skip_base_filter parameter."""
+ # Create users
+ users = []
+ for i in range(3):
+ user = User(
+ username=f"test_skip_filter_{i}",
+ first_name=f"Test{i}",
+ last_name="User",
+ email=f"test{i}@example.com",
+ active=True,
+ )
+ users.append(user)
+ db.session.add(user)
+ db.session.commit()
+
+ ids = [user.id for user in users]
+
+ # Without skipping base filter
+ found_no_skip = UserDAO.find_by_ids(ids, skip_base_filter=False)
+ assert len(found_no_skip) == 3
+
+ # With skipping base filter
+ found_skip = UserDAO.find_by_ids(ids, skip_base_filter=True)
+ assert len(found_skip) == 3
+
+
+def test_base_dao_create_with_item(app_context: Session) -> None:
+ """Test BaseDAO.create with an item parameter."""
+ # Create a user item
+ user = User(
+ username="created_with_item",
+ first_name="Created",
+ last_name="Item",
+ email="created@example.com",
+ active=True,
+ )
+
+ # Create using the item
+ created = UserDAO.create(item=user)
+ assert created is not None
+ assert created.username == "created_with_item"
+ assert created.first_name == "Created"
+
+ # Verify it's in the session
+ assert created in db.session
+
+ # Commit and verify it persists
+ db.session.commit()
+
+ # Find it again to ensure it was saved
+ found = UserDAO.find_by_id(created.id, skip_base_filter=True)
+ assert found is not None
+ assert found.username == "created_with_item"
+
+
+def test_base_dao_create_with_attributes(app_context: Session) -> None:
+ """Test BaseDAO.create with attributes parameter."""
+ # Create using attributes dict
+ attributes = {
+ "username": "created_with_attrs",
+ "first_name": "Created",
+ "last_name": "Attrs",
+ "email": "attrs@example.com",
+ "active": True,
+ }
+
+ created = UserDAO.create(attributes=attributes)
+ assert created is not None
+ assert created.username == "created_with_attrs"
+ assert created.email == "attrs@example.com"
+
+ # Commit and verify
+ db.session.commit()
+ found = UserDAO.find_by_id(created.id, skip_base_filter=True)
+ assert found is not None
+ assert found.username == "created_with_attrs"
+
+
+def test_base_dao_create_with_both_item_and_attributes(app_context: Session) -> None:
+ """Test BaseDAO.create with both item and attributes (override behavior)."""
+ # Create a user item
+ user = User(
+ username="item_username",
+ first_name="Item",
+ last_name="User",
+ email="item@example.com",
+ active=False,
+ )
+
+ # Override some attributes
+ attributes = {
+ "username": "override_username",
+ "active": True,
+ }
+
+ created = UserDAO.create(item=user, attributes=attributes)
+ assert created is not None
+ assert created.username == "override_username" # Should be overridden
+ assert created.active is True # Should be overridden
+ assert created.first_name == "Item" # Should keep original
+ assert created.last_name == "User" # Should keep original
+
+ db.session.commit()
+
+
+def test_base_dao_update_with_item(app_context: Session) -> None:
+ """Test BaseDAO.update with an item parameter."""
+ # Create a user first
+ user = User(
+ username="update_test",
+ first_name="Original",
+ last_name="User",
+ email="original@example.com",
+ active=True,
+ )
+ db.session.add(user)
+ db.session.commit()
+
+ # Update the user
+ user.first_name = "Updated"
+ updated = UserDAO.update(item=user)
+ assert updated is not None
+ assert updated.first_name == "Updated"
+
+ db.session.commit()
+
+ # Verify the update persisted
+ found = UserDAO.find_by_id(user.id, skip_base_filter=True)
+ assert found is not None
+ assert found.first_name == "Updated"
+
+
+def test_base_dao_update_with_attributes(app_context: Session) -> None:
+ """Test BaseDAO.update with attributes parameter."""
+ # Create a user first
+ user = User(
+ username="update_attrs_test",
+ first_name="Original",
+ last_name="User",
+ email="original@example.com",
+ active=True,
+ )
+ db.session.add(user)
+ db.session.commit()
+
+ # Update using attributes
+ attributes = {"first_name": "Updated", "last_name": "Attr User"}
+ updated = UserDAO.update(item=user, attributes=attributes)
+ assert updated is not None
+ assert updated.first_name == "Updated"
+ assert updated.last_name == "Attr User"
+
+ db.session.commit()
+
+
+def test_base_dao_update_detached_item(app_context: Session) -> None:
+ """Test BaseDAO.update with a detached item."""
+ # Create a user first
+ user = User(
+ username="detached_test",
+ first_name="Original",
+ last_name="User",
+ email="detached@example.com",
+ active=True,
+ )
+ db.session.add(user)
+ db.session.commit()
+
+ user_id = user.id
+
+ # Expunge to detach from session
+ db.session.expunge(user)
+
+ # Update the detached user
+ user.first_name = "Updated Detached"
+ updated = UserDAO.update(item=user)
+ assert updated is not None
+ assert updated.first_name == "Updated Detached"
+
+ db.session.commit()
+
+ # Verify the update persisted
+ found = UserDAO.find_by_id(user_id, skip_base_filter=True)
+ assert found is not None
+ assert found.first_name == "Updated Detached"
+
+
+def test_base_dao_delete_single_item(app_context: Session) -> None:
+ """Test BaseDAO.delete with a single item."""
+ # Create a dashboard instead of user to avoid circular dependencies
+ dashboard = Dashboard(
+ dashboard_title="Delete Test",
+ slug="delete-test",
+ published=True,
+ )
+ db.session.add(dashboard)
+ db.session.commit()
+
+ dashboard_id = dashboard.id
+
+ DashboardDAO.delete([dashboard])
+ db.session.commit()
+
+ # Verify it's gone
+ found = DashboardDAO.find_by_id(dashboard_id, skip_base_filter=True)
+ assert found is None
+
+
+def test_base_dao_delete_multiple_items(app_context: Session) -> None:
+ """Test BaseDAO.delete with multiple items."""
+ # Create multiple dashboards instead of users to avoid circular dependencies
+ dashboards = []
+ for i in range(3):
+ dashboard = Dashboard(
+ dashboard_title=f"Delete Multi {i}",
+ slug=f"delete-multi-{i}",
+ published=True,
+ )
+ dashboards.append(dashboard)
+ db.session.add(dashboard)
+ db.session.commit()
+
+ dashboard_ids = [dashboard.id for dashboard in dashboards]
+
+ DashboardDAO.delete(dashboards)
+ db.session.commit()
+
+ # Verify they're all gone
+ for dashboard_id in dashboard_ids:
+ found = DashboardDAO.find_by_id(dashboard_id, skip_base_filter=True)
+ assert found is None
+
+
+def test_base_dao_delete_empty_list(app_context: Session) -> None:
+ """Test BaseDAO.delete with empty list."""
+ # Should not raise any errors
+ UserDAO.delete([])
+ db.session.commit()
+ # Just ensuring no exception is raised
+
+
+def test_base_dao_find_all(app_context: Session) -> None:
+ """Test BaseDAO.find_all method."""
+ # Create some users
+ users = []
+ for i in range(3):
+ user = User(
+ username=f"find_all_{i}",
+ first_name=f"Find{i}",
+ last_name="All",
+ email=f"findall{i}@example.com",
+ active=True,
+ )
+ users.append(user)
+ db.session.add(user)
+ db.session.commit()
+
+ # Find all users
+ all_users = UserDAO.find_all()
+ assert len(all_users) >= 3 # At least our 3 users
+
+ # Check our users are in the results
+ usernames = [u.username for u in all_users]
+ for user in users:
+ assert user.username in usernames
+
+
+def test_base_dao_find_one_or_none(app_context: Session) -> None:
+ """Test BaseDAO.find_one_or_none method."""
+ # Create users with specific criteria
+ user1 = User(
+ username="unique_username_123",
+ first_name="Unique",
+ last_name="User",
+ email="unique@example.com",
+ active=True,
+ )
+ user2 = User(
+ username="another_user",
+ first_name="Another",
+ last_name="User",
+ email="another@example.com",
+ active=True,
+ )
+ user3 = User(
+ username="third_user",
+ first_name="Another", # Same first name as user2
+ last_name="User",
+ email="third@example.com",
+ active=True,
+ )
+ db.session.add_all([user1, user2, user3])
+ db.session.commit()
+
+ # Find one with unique criteria
+ found = UserDAO.find_one_or_none(username="unique_username_123")
+ assert found is not None
+ assert found.username == "unique_username_123"
+
+ # Find none with non-existent criteria
+ not_found = UserDAO.find_one_or_none(username="non_existent_user")
+ assert not_found is None
+
+ db.session.commit()
+
+
+def test_base_dao_list_returns_results(user_with_data: Session) -> None:
+ """Test that BaseDAO.list returns results and total."""
+ results, total = UserDAO.list()
+ assert isinstance(results, list)
+ assert isinstance(total, int)
+ assert total >= 1 # At least the fixture user
+
+
+def test_base_dao_list_with_column_operators(user_with_data: Session) -> None:
+ """Test BaseDAO.list with column operators."""
+ column_operators = [
+ ColumnOperator(col="username", opr=ColumnOperatorEnum.eq, value="testuser")
+ ]
+ results, total = UserDAO.list(column_operators=column_operators)
+ assert total == 1
+ assert results[0].username == "testuser"
+
+
+def test_base_dao_list_with_non_matching_column_operator(
+ user_with_data: Session,
+) -> None:
+ """Test BaseDAO.list with non-matching column operator."""
+ column_operators = [
+ ColumnOperator(
+ col="username", opr=ColumnOperatorEnum.eq, value="nonexistentuser"
+ )
+ ]
+ results, total = UserDAO.list(column_operators=column_operators)
+ assert total == 0
+ assert results == []
+
+
+def test_base_dao_count_returns_value(user_with_data: Session) -> None:
+ """Test that BaseDAO.count returns correct count."""
+ count = UserDAO.count()
+ assert isinstance(count, int)
+ assert count >= 1 # At least the fixture user
+
+
+def test_base_dao_count_with_column_operators(user_with_data: Session) -> None:
+ """Test BaseDAO.count with column operators."""
+ # Count with matching operator
+ column_operators = [
+ ColumnOperator(col="username", opr=ColumnOperatorEnum.eq, value="testuser")
+ ]
+ count = UserDAO.count(column_operators=column_operators)
+ assert count == 1
+
+ # Count with non-matching operator
+ column_operators = [
+ ColumnOperator(
+ col="username", opr=ColumnOperatorEnum.eq, value="nonexistentuser"
+ )
+ ]
+ count = UserDAO.count(column_operators=column_operators)
+ assert count == 0
+
+
+def test_base_dao_list_ordering(user_with_data: Session) -> None:
+ """Test BaseDAO.list with ordering."""
+ # Create additional users with predictable names
+ users = []
+ for i in range(3):
+ user = User(
+ id=200 + i, # Ensure predictable IDs
+ username=f"order_test_{i}",
+ first_name=f"Order{i}",
+ last_name="Test",
+ email=f"order{i}@example.com",
+ active=True,
+ )
+ users.append(user)
+ user_with_data.add(user)
+ user_with_data.commit()
+
+ # Test ascending order
+ results_asc, _ = UserDAO.list(
+ order_column="username", order_direction="asc", page_size=100
+ )
+ usernames_asc = [r.username for r in results_asc]
+ # Check that our test users are in order
+ assert usernames_asc.index("order_test_0") < usernames_asc.index("order_test_1")
+ assert usernames_asc.index("order_test_1") < usernames_asc.index("order_test_2")
+
+ # Test descending order
+ results_desc, _ = UserDAO.list(
+ order_column="username", order_direction="desc", page_size=100
+ )
+ usernames_desc = [r.username for r in results_desc]
+ # Check that our test users are in reverse order
+ assert usernames_desc.index("order_test_2") < usernames_desc.index("order_test_1")
+ assert usernames_desc.index("order_test_1") < usernames_desc.index("order_test_0")
+
+ for user in users:
+ user_with_data.delete(user)
+ user_with_data.commit()
+
+
+def test_base_dao_list_paging(user_with_data: Session) -> None:
+ """Test BaseDAO.list with paging."""
+ # Create additional users for paging
+ users = []
+ for i in range(10):
+ user = User(
+ id=300 + i,
+ username=f"page_test_{i}",
+ first_name=f"Page{i}",
+ last_name="Test",
+ email=f"page{i}@example.com",
+ active=True,
+ )
+ users.append(user)
+ user_with_data.add(user)
+ user_with_data.commit()
+
+ # Test first page
+ page1_results, page1_total = UserDAO.list(page=0, page_size=5)
+ assert len(page1_results) <= 5
+ assert page1_total >= 10 # At least our 10 users
+
+ # Test second page
+ page2_results, page2_total = UserDAO.list(page=1, page_size=5)
Review Comment:
<div>
<div id="suggestion">
<div id="issue"><b>Unreliable pagination assertion</b></div>
<div id="fix">
The assertion `assert set(page1_ids).isdisjoint(set(page2_ids))` on line 763 is unreliable because SQL pagination without explicit ordering can return non-deterministic results. This will cause test flakiness when the database returns overlapping results between pages. Add explicit ordering by adding `order_column="id"` to both `UserDAO.list()` calls on lines 751 and 756 to ensure consistent pagination.
</div>
<details>
<summary>
<b>Code suggestion</b>
</summary>
<blockquote>Check the AI-generated fix before applying</blockquote>
<div id="code">
```suggestion
# Test first page
page1_results, page1_total = UserDAO.list(page=0, page_size=5, order_column="id")
assert len(page1_results) <= 5
assert page1_total >= 10 # At least our 10 users
# Test second page
page2_results, page2_total = UserDAO.list(page=1, page_size=5, order_column="id")
```
</div>
</details>
</div>
<small><i>Code Review Run <a href=https://github.com/apache/superset/pull/35018#issuecomment-3293390307>#8dadc1</a></i></small>
</div>
---
Should Bito avoid suggestions like this for future reviews? (<a href=https://alpha.bito.ai/home/ai-agents/review-rules>Manage Rules</a>)
- [ ] Yes, avoid them
##########
tests/integration_tests/dao/base_dao_test.py:
##########
@@ -0,0 +1,1619 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""
+Integration tests for BaseDAO functionality.
+
+This module contains comprehensive integration tests for the BaseDAO class and its
+subclasses, covering database operations, CRUD methods, flexible column support,
+column operators, and error handling.
+
+Tests use an in-memory SQLite database for isolation and to replicate the unit test
+environment behavior. User model deletions are avoided due to circular dependency
+constraints with self-referential foreign keys.
+"""
+
+import datetime
+import time
+import uuid
+
+import pytest
+from flask_appbuilder.models.filters import BaseFilter
+from flask_appbuilder.security.sqla.models import User
+from sqlalchemy import Column, DateTime, Integer, String
+from sqlalchemy.ext.declarative import declarative_base
+from sqlalchemy.orm.session import Session
+
+from superset.daos.base import BaseDAO, ColumnOperator, ColumnOperatorEnum
+from superset.daos.chart import ChartDAO
+from superset.daos.dashboard import DashboardDAO
+from superset.daos.database import DatabaseDAO
+from superset.daos.user import UserDAO
+from superset.extensions import db
+from superset.models.core import Database
+from superset.models.dashboard import Dashboard
+from superset.models.slice import Slice
+
+# Create a test model for comprehensive testing
+Base = declarative_base()
+
+
+class ExampleModel(Base): # type: ignore
+ __tablename__ = "example_model"
+ id = Column(Integer, primary_key=True)
+ uuid = Column(String(36), unique=True, nullable=False)
+ slug = Column(String(100), unique=True)
+ name = Column(String(100))
+ code = Column(String(50), unique=True)
+ created_on = Column(DateTime, default=datetime.datetime.utcnow)
+
+
+class ExampleModelDAO(BaseDAO[ExampleModel]):
+ model_cls = ExampleModel
+ id_column_name = "id"
+ base_filter = None
+
+
+class MockModel:
+ def __init__(self, id=1, name="test"):
+ self.id = id
+ self.name = name
+
+
+class TestDAO(BaseDAO[MockModel]):
+ model_cls = MockModel
+
+
+@pytest.fixture(autouse=True)
+def mock_g_user(app_context):
+ """Mock the flask g.user for security context."""
+ # Within app context, we can safely mock g
+ from flask import g
+
+ mock_user = User()
+ mock_user.id = 1
+ mock_user.username = "test_user"
+
+ # Set g.user directly instead of patching
+ g.user = mock_user
+ yield
+
+ # Clean up
+ if hasattr(g, "user"):
+ delattr(g, "user")
+
+
+# =============================================================================
+# Integration Tests - These tests use the actual database
+# =============================================================================
+
+
+def test_column_operator_enum_complete_coverage(user_with_data: Session) -> None:
+ """
+ Test that every single ColumnOperatorEnum operator is covered by tests.
+ This ensures we have comprehensive test coverage for all operators.
+ """
+ # Simply verify that we can create queries with all operators
+ for operator in ColumnOperatorEnum:
+ column_operator = ColumnOperator(
+ col="username", opr=operator, value="test_value"
+ )
+ # Just check it doesn't raise an error
+ assert column_operator.opr == operator
+
+
+def test_find_by_id_with_default_column(app_context: Session) -> None:
+ """Test find_by_id with default 'id' column."""
+ # Create a user to test with
+ user = User(
+ username="test_find_by_id",
+ first_name="Test",
+ last_name="User",
+ email="test@example.com",
+ active=True,
+ )
+ db.session.add(user)
+ db.session.commit()
+
+ # Find by numeric id
+ found = UserDAO.find_by_id(user.id, skip_base_filter=True)
+ assert found is not None
+ assert found.id == user.id
+ assert found.username == "test_find_by_id"
+
+ # Test with non-existent id
+ not_found = UserDAO.find_by_id(999999, skip_base_filter=True)
+ assert not_found is None
+
+
+def test_find_by_id_with_uuid_column(app_context: Session) -> None:
+ """Test find_by_id with custom uuid column."""
+ # Create a dashboard with uuid
+ dashboard = Dashboard(
+ dashboard_title="Test UUID Dashboard",
+ slug="test-uuid-dashboard",
+ published=True,
+ )
+ db.session.add(dashboard)
+ db.session.commit()
+
+ # Find by uuid string using the uuid column
+ found = DashboardDAO.find_by_id(
+ str(dashboard.uuid), id_column="uuid", skip_base_filter=True
+ )
+ assert found is not None
+ assert found.uuid == dashboard.uuid
+ assert found.dashboard_title == "Test UUID Dashboard"
+
+ # Find by numeric id (should still work)
+ found_by_id = DashboardDAO.find_by_id(dashboard.id, skip_base_filter=True)
+ assert found_by_id is not None
+ assert found_by_id.id == dashboard.id
+
+ # Test with non-existent uuid
+ not_found = DashboardDAO.find_by_id(str(uuid.uuid4()), skip_base_filter=True)
+ assert not_found is None
+
+
+def test_find_by_id_with_slug_column(app_context: Session) -> None:
+ """Test find_by_id with slug column fallback."""
+ # Create a dashboard with slug
+ dashboard = Dashboard(
+ dashboard_title="Test Slug Dashboard",
+ slug="test-slug-dashboard",
+ published=True,
+ )
+ db.session.add(dashboard)
+ db.session.commit()
+
+ # Find by slug using the slug column
+ found = DashboardDAO.find_by_id(
+ "test-slug-dashboard", id_column="slug", skip_base_filter=True
+ )
+ assert found is not None
+ assert found.slug == "test-slug-dashboard"
+ assert found.dashboard_title == "Test Slug Dashboard"
+
+ # Test with non-existent slug
+ not_found = DashboardDAO.find_by_id("non-existent-slug", skip_base_filter=True)
+ assert not_found is None
+
+
+def test_find_by_id_with_invalid_column(app_context: Session) -> None:
+ """Test find_by_id returns None when column doesn't exist."""
+ # This should return None gracefully
+ result = UserDAO.find_by_id("not_a_valid_id", skip_base_filter=True)
+ assert result is None
+
+
+def test_find_by_id_skip_base_filter(app_context: Session) -> None:
+ """Test find_by_id with skip_base_filter parameter."""
+ # Create users with different active states
+ active_user = User(
+ username="active_user",
+ first_name="Active",
+ last_name="User",
+ email="active@example.com",
+ active=True,
+ )
+ inactive_user = User(
+ username="inactive_user",
+ first_name="Inactive",
+ last_name="User",
+ email="inactive@example.com",
+ active=False,
+ )
+ db.session.add_all([active_user, inactive_user])
+ db.session.commit()
+
+ # Without skipping base filter (if one exists)
+ found_active = UserDAO.find_by_id(active_user.id, skip_base_filter=False)
+ assert found_active is not None
+
+ # With skipping base filter
+ found_active_skip = UserDAO.find_by_id(active_user.id, skip_base_filter=True)
+ assert found_active_skip is not None
+
+ # Both should find the user since UserDAO might not have a base filter
+ assert found_active.id == active_user.id
+ assert found_active_skip.id == active_user.id
+
+
+def test_find_by_ids_with_default_column(app_context: Session) -> None:
+ """Test find_by_ids with default 'id' column."""
+ # Create multiple users
+ users = []
+ for i in range(3):
+ user = User(
+ username=f"test_find_by_ids_{i}",
+ first_name=f"Test{i}",
+ last_name="User",
+ email=f"test{i}@example.com",
+ active=True,
+ )
+ users.append(user)
+ db.session.add(user)
+ db.session.commit()
+
+ # Find by multiple ids
+ ids = [user.id for user in users]
+ found = UserDAO.find_by_ids(ids, skip_base_filter=True)
+ assert len(found) == 3
+ found_ids = [u.id for u in found]
+ assert set(found_ids) == set(ids)
+
+ # Test with mix of existent and non-existent ids
+ mixed_ids = [users[0].id, 999999, users[1].id]
+ found_mixed = UserDAO.find_by_ids(mixed_ids, skip_base_filter=True)
+ assert len(found_mixed) == 2
+
+ # Test with empty list
+ found_empty = UserDAO.find_by_ids([], skip_base_filter=True)
+ assert found_empty == []
+
+
+def test_find_by_ids_with_uuid_column(app_context: Session) -> None:
+ """Test find_by_ids with uuid column."""
+ # Create multiple dashboards
+ dashboards = []
+ for i in range(3):
+ dashboard = Dashboard(
+ dashboard_title=f"Test UUID Dashboard {i}",
+ slug=f"test-uuid-dashboard-{i}",
+ published=True,
+ )
+ dashboards.append(dashboard)
+ db.session.add(dashboard)
+ db.session.commit()
+
+ # Find by multiple uuids
+ uuids = [str(dashboard.uuid) for dashboard in dashboards]
+ found = DashboardDAO.find_by_ids(uuids, id_column="uuid", skip_base_filter=True)
+ assert len(found) == 3
+ found_uuids = [str(d.uuid) for d in found]
+ assert set(found_uuids) == set(uuids)
+
+ # Test with mix of ids and uuids - search separately by column
+ found_by_id = DashboardDAO.find_by_ids([dashboards[0].id], skip_base_filter=True)
+ found_by_uuid = DashboardDAO.find_by_ids(
+ [str(dashboards[1].uuid)], id_column="uuid", skip_base_filter=True
+ )
+ assert len(found_by_id) == 1
+ assert len(found_by_uuid) == 1
+
+
+def test_find_by_ids_with_slug_column(app_context: Session) -> None:
+ """Test find_by_ids with slug column."""
+ # Create multiple dashboards
+ dashboards = []
+ for i in range(3):
+ dashboard = Dashboard(
+ dashboard_title=f"Test Slug Dashboard {i}",
+ slug=f"test-slug-dashboard-{i}",
+ published=True,
+ )
+ dashboards.append(dashboard)
+ db.session.add(dashboard)
+ db.session.commit()
+
+ # Find by multiple slugs
+ slugs = [dashboard.slug for dashboard in dashboards]
+ found = DashboardDAO.find_by_ids(slugs, id_column="slug", skip_base_filter=True)
+ assert len(found) == 3
+ found_slugs = [d.slug for d in found]
+ assert set(found_slugs) == set(slugs)
+
+
+def test_find_by_ids_with_invalid_column(app_context: Session) -> None:
+ """Test find_by_ids returns empty list when column doesn't exist."""
+ # This should return empty list gracefully
+ result = UserDAO.find_by_ids(["not_a_valid_id"], skip_base_filter=True)
+ assert result == []
+
+
+def test_find_by_ids_skip_base_filter(app_context: Session) -> None:
+ """Test find_by_ids with skip_base_filter parameter."""
+ # Create users
+ users = []
+ for i in range(3):
+ user = User(
+ username=f"test_skip_filter_{i}",
+ first_name=f"Test{i}",
+ last_name="User",
+ email=f"test{i}@example.com",
+ active=True,
+ )
+ users.append(user)
+ db.session.add(user)
+ db.session.commit()
+
+ ids = [user.id for user in users]
+
+ # Without skipping base filter
+ found_no_skip = UserDAO.find_by_ids(ids, skip_base_filter=False)
+ assert len(found_no_skip) == 3
+
+ # With skipping base filter
+ found_skip = UserDAO.find_by_ids(ids, skip_base_filter=True)
+ assert len(found_skip) == 3
+
+
+def test_base_dao_create_with_item(app_context: Session) -> None:
+ """Test BaseDAO.create with an item parameter."""
+ # Create a user item
+ user = User(
+ username="created_with_item",
+ first_name="Created",
+ last_name="Item",
+ email="created@example.com",
+ active=True,
+ )
+
+ # Create using the item
+ created = UserDAO.create(item=user)
+ assert created is not None
+ assert created.username == "created_with_item"
+ assert created.first_name == "Created"
+
+ # Verify it's in the session
+ assert created in db.session
+
+ # Commit and verify it persists
+ db.session.commit()
+
+ # Find it again to ensure it was saved
+ found = UserDAO.find_by_id(created.id, skip_base_filter=True)
+ assert found is not None
+ assert found.username == "created_with_item"
+
+
+def test_base_dao_create_with_attributes(app_context: Session) -> None:
+ """Test BaseDAO.create with attributes parameter."""
+ # Create using attributes dict
+ attributes = {
+ "username": "created_with_attrs",
+ "first_name": "Created",
+ "last_name": "Attrs",
+ "email": "attrs@example.com",
+ "active": True,
+ }
+
+ created = UserDAO.create(attributes=attributes)
+ assert created is not None
+ assert created.username == "created_with_attrs"
+ assert created.email == "attrs@example.com"
+
+ # Commit and verify
+ db.session.commit()
+ found = UserDAO.find_by_id(created.id, skip_base_filter=True)
+ assert found is not None
+ assert found.username == "created_with_attrs"
+
+
+def test_base_dao_create_with_both_item_and_attributes(app_context: Session) -> None:
+ """Test BaseDAO.create with both item and attributes (override behavior)."""
+ # Create a user item
+ user = User(
+ username="item_username",
+ first_name="Item",
+ last_name="User",
+ email="item@example.com",
+ active=False,
+ )
+
+ # Override some attributes
+ attributes = {
+ "username": "override_username",
+ "active": True,
+ }
+
+ created = UserDAO.create(item=user, attributes=attributes)
+ assert created is not None
+ assert created.username == "override_username" # Should be overridden
+ assert created.active is True # Should be overridden
+ assert created.first_name == "Item" # Should keep original
+ assert created.last_name == "User" # Should keep original
+
+ db.session.commit()
+
+
+def test_base_dao_update_with_item(app_context: Session) -> None:
+ """Test BaseDAO.update with an item parameter."""
+ # Create a user first
+ user = User(
+ username="update_test",
+ first_name="Original",
+ last_name="User",
+ email="original@example.com",
+ active=True,
+ )
+ db.session.add(user)
+ db.session.commit()
+
+ # Update the user
+ user.first_name = "Updated"
+ updated = UserDAO.update(item=user)
+ assert updated is not None
+ assert updated.first_name == "Updated"
+
+ db.session.commit()
+
+ # Verify the update persisted
+ found = UserDAO.find_by_id(user.id, skip_base_filter=True)
+ assert found is not None
+ assert found.first_name == "Updated"
+
+
+def test_base_dao_update_with_attributes(app_context: Session) -> None:
+ """Test BaseDAO.update with attributes parameter."""
+ # Create a user first
+ user = User(
+ username="update_attrs_test",
+ first_name="Original",
+ last_name="User",
+ email="original@example.com",
+ active=True,
+ )
+ db.session.add(user)
+ db.session.commit()
+
+ # Update using attributes
+ attributes = {"first_name": "Updated", "last_name": "Attr User"}
+ updated = UserDAO.update(item=user, attributes=attributes)
+ assert updated is not None
+ assert updated.first_name == "Updated"
+ assert updated.last_name == "Attr User"
+
+ db.session.commit()
+
+
+def test_base_dao_update_detached_item(app_context: Session) -> None:
+ """Test BaseDAO.update with a detached item."""
+ # Create a user first
+ user = User(
+ username="detached_test",
+ first_name="Original",
+ last_name="User",
+ email="detached@example.com",
+ active=True,
+ )
+ db.session.add(user)
+ db.session.commit()
+
+ user_id = user.id
+
+ # Expunge to detach from session
+ db.session.expunge(user)
+
+ # Update the detached user
+ user.first_name = "Updated Detached"
+ updated = UserDAO.update(item=user)
+ assert updated is not None
+ assert updated.first_name == "Updated Detached"
+
+ db.session.commit()
+
+ # Verify the update persisted
+ found = UserDAO.find_by_id(user_id, skip_base_filter=True)
+ assert found is not None
+ assert found.first_name == "Updated Detached"
+
+
+def test_base_dao_delete_single_item(app_context: Session) -> None:
+ """Test BaseDAO.delete with a single item."""
+ # Create a dashboard instead of user to avoid circular dependencies
+ dashboard = Dashboard(
+ dashboard_title="Delete Test",
+ slug="delete-test",
+ published=True,
+ )
+ db.session.add(dashboard)
+ db.session.commit()
+
+ dashboard_id = dashboard.id
+
+ DashboardDAO.delete([dashboard])
+ db.session.commit()
+
+ # Verify it's gone
+ found = DashboardDAO.find_by_id(dashboard_id, skip_base_filter=True)
+ assert found is None
+
+
+def test_base_dao_delete_multiple_items(app_context: Session) -> None:
+ """Test BaseDAO.delete with multiple items."""
+ # Create multiple dashboards instead of users to avoid circular dependencies
+ dashboards = []
+ for i in range(3):
+ dashboard = Dashboard(
+ dashboard_title=f"Delete Multi {i}",
+ slug=f"delete-multi-{i}",
+ published=True,
+ )
+ dashboards.append(dashboard)
+ db.session.add(dashboard)
+ db.session.commit()
+
+ dashboard_ids = [dashboard.id for dashboard in dashboards]
+
+ DashboardDAO.delete(dashboards)
+ db.session.commit()
+
+ # Verify they're all gone
+ for dashboard_id in dashboard_ids:
+ found = DashboardDAO.find_by_id(dashboard_id, skip_base_filter=True)
+ assert found is None
+
+
+def test_base_dao_delete_empty_list(app_context: Session) -> None:
+ """Test BaseDAO.delete with empty list."""
+ # Should not raise any errors
+ UserDAO.delete([])
+ db.session.commit()
+ # Just ensuring no exception is raised
+
+
+def test_base_dao_find_all(app_context: Session) -> None:
+ """Test BaseDAO.find_all method."""
+ # Create some users
+ users = []
+ for i in range(3):
+ user = User(
+ username=f"find_all_{i}",
+ first_name=f"Find{i}",
+ last_name="All",
+ email=f"findall{i}@example.com",
+ active=True,
+ )
+ users.append(user)
+ db.session.add(user)
+ db.session.commit()
+
+ # Find all users
+ all_users = UserDAO.find_all()
+ assert len(all_users) >= 3 # At least our 3 users
+
+ # Check our users are in the results
+ usernames = [u.username for u in all_users]
+ for user in users:
+ assert user.username in usernames
+
+
+def test_base_dao_find_one_or_none(app_context: Session) -> None:
+ """Test BaseDAO.find_one_or_none method."""
+ # Create users with specific criteria
+ user1 = User(
+ username="unique_username_123",
+ first_name="Unique",
+ last_name="User",
+ email="unique@example.com",
+ active=True,
+ )
+ user2 = User(
+ username="another_user",
+ first_name="Another",
+ last_name="User",
+ email="another@example.com",
+ active=True,
+ )
+ user3 = User(
+ username="third_user",
+ first_name="Another", # Same first name as user2
+ last_name="User",
+ email="third@example.com",
+ active=True,
+ )
+ db.session.add_all([user1, user2, user3])
+ db.session.commit()
+
+ # Find one with unique criteria
+ found = UserDAO.find_one_or_none(username="unique_username_123")
+ assert found is not None
+ assert found.username == "unique_username_123"
+
+ # Find none with non-existent criteria
+ not_found = UserDAO.find_one_or_none(username="non_existent_user")
+ assert not_found is None
+
+ db.session.commit()
+
+
+def test_base_dao_list_returns_results(user_with_data: Session) -> None:
+ """Test that BaseDAO.list returns results and total."""
+ results, total = UserDAO.list()
+ assert isinstance(results, list)
+ assert isinstance(total, int)
+ assert total >= 1 # At least the fixture user
+
+
+def test_base_dao_list_with_column_operators(user_with_data: Session) -> None:
+ """Test BaseDAO.list with column operators."""
+ column_operators = [
+ ColumnOperator(col="username", opr=ColumnOperatorEnum.eq, value="testuser")
+ ]
+ results, total = UserDAO.list(column_operators=column_operators)
+ assert total == 1
+ assert results[0].username == "testuser"
+
+
+def test_base_dao_list_with_non_matching_column_operator(
+ user_with_data: Session,
+) -> None:
+ """Test BaseDAO.list with non-matching column operator."""
+ column_operators = [
+ ColumnOperator(
+ col="username", opr=ColumnOperatorEnum.eq, value="nonexistentuser"
+ )
+ ]
+ results, total = UserDAO.list(column_operators=column_operators)
+ assert total == 0
+ assert results == []
+
+
+def test_base_dao_count_returns_value(user_with_data: Session) -> None:
+ """Test that BaseDAO.count returns correct count."""
+ count = UserDAO.count()
+ assert isinstance(count, int)
+ assert count >= 1 # At least the fixture user
+
+
+def test_base_dao_count_with_column_operators(user_with_data: Session) -> None:
+ """Test BaseDAO.count with column operators."""
+ # Count with matching operator
+ column_operators = [
+ ColumnOperator(col="username", opr=ColumnOperatorEnum.eq, value="testuser")
+ ]
+ count = UserDAO.count(column_operators=column_operators)
+ assert count == 1
+
+ # Count with non-matching operator
+ column_operators = [
+ ColumnOperator(
+ col="username", opr=ColumnOperatorEnum.eq, value="nonexistentuser"
+ )
+ ]
+ count = UserDAO.count(column_operators=column_operators)
+ assert count == 0
+
+
+def test_base_dao_list_ordering(user_with_data: Session) -> None:
+ """Test BaseDAO.list with ordering."""
+ # Create additional users with predictable names
+ users = []
+ for i in range(3):
+ user = User(
+ id=200 + i, # Ensure predictable IDs
+ username=f"order_test_{i}",
+ first_name=f"Order{i}",
+ last_name="Test",
+ email=f"order{i}@example.com",
+ active=True,
+ )
+ users.append(user)
+ user_with_data.add(user)
+ user_with_data.commit()
+
+ # Test ascending order
+ results_asc, _ = UserDAO.list(
+ order_column="username", order_direction="asc", page_size=100
+ )
+ usernames_asc = [r.username for r in results_asc]
+ # Check that our test users are in order
+ assert usernames_asc.index("order_test_0") < usernames_asc.index("order_test_1")
+ assert usernames_asc.index("order_test_1") < usernames_asc.index("order_test_2")
+
+ # Test descending order
+ results_desc, _ = UserDAO.list(
+ order_column="username", order_direction="desc", page_size=100
+ )
+ usernames_desc = [r.username for r in results_desc]
+ # Check that our test users are in reverse order
+ assert usernames_desc.index("order_test_2") < usernames_desc.index("order_test_1")
+ assert usernames_desc.index("order_test_1") < usernames_desc.index("order_test_0")
+
+ for user in users:
+ user_with_data.delete(user)
+ user_with_data.commit()
+
+
+def test_base_dao_list_paging(user_with_data: Session) -> None:
+ """Test BaseDAO.list with paging."""
+ # Create additional users for paging
+ users = []
+ for i in range(10):
+ user = User(
+ id=300 + i,
+ username=f"page_test_{i}",
+ first_name=f"Page{i}",
+ last_name="Test",
+ email=f"page{i}@example.com",
+ active=True,
+ )
+ users.append(user)
+ user_with_data.add(user)
+ user_with_data.commit()
+
+ # Test first page
+ page1_results, page1_total = UserDAO.list(page=0, page_size=5)
+ assert len(page1_results) <= 5
+ assert page1_total >= 10 # At least our 10 users
+
+ # Test second page
+ page2_results, page2_total = UserDAO.list(page=1, page_size=5)
+ assert len(page2_results) <= 5
+ assert page2_total >= 10
+
+ # Results should be different
+ page1_ids = [r.id for r in page1_results]
+ page2_ids = [r.id for r in page2_results]
+ assert set(page1_ids).isdisjoint(set(page2_ids)) # No overlap
+
+ for user in users:
+ user_with_data.delete(user)
+ user_with_data.commit()
+
+
+def test_base_dao_list_search(user_with_data: Session) -> None:
+ """Test BaseDAO.list with search."""
+ # Create users with searchable names
+ users = []
+ for i in range(3):
+ user = User(
+ id=400 + i,
+ username=f"searchable_{i}",
+ first_name=f"Searchable{i}",
+ last_name="User",
+ email=f"search{i}@example.com",
+ active=True,
+ )
+ users.append(user)
+ user_with_data.add(user)
+
+ # Create some non-matching users
+ for i in range(2):
+ user = User(
+ id=410 + i,
+ username=f"other_{i}",
+ first_name=f"Other{i}",
+ last_name="Person",
+ email=f"other{i}@example.com",
+ active=True,
+ )
+ users.append(user)
+ user_with_data.add(user)
+
+ user_with_data.commit()
+
+ # Search for "searchable"
+ results, total = UserDAO.list(
+ search="searchable", search_columns=["username", "first_name"]
+ )
+ assert total >= 3 # At least our 3 searchable users
+
+ # Verify all results contain "searchable"
+ for result in results:
+ assert (
+ "searchable" in result.username.lower()
+ or "searchable" in result.first_name.lower()
+ )
+
+ for user in users:
+ user_with_data.delete(user)
+ user_with_data.commit()
+
+
+def test_base_dao_list_custom_filter(user_with_data: Session) -> None:
+ """Test BaseDAO.list with custom filters."""
+ # Create users with specific attributes
+ active_user = User(
+ id=500,
+ username="active_custom",
+ first_name="Active",
+ last_name="Custom",
+ email="active@custom.com",
+ active=True,
+ )
+ inactive_user = User(
+ id=501,
+ username="inactive_custom",
+ first_name="Inactive",
+ last_name="Custom",
+ email="inactive@custom.com",
+ active=False,
+ )
+ user_with_data.add_all([active_user, inactive_user])
+ user_with_data.commit()
+
+ # Create a custom filter for active users only
+ class ActiveUsersFilter(BaseFilter):
+ def __init__(self):
+ self.column_name = "active"
+ self.datamodel = None
+ self.model = User # Set model directly
+
+ def apply(self, query, value):
+ return query.filter(User.active.is_(True))
+
+ custom_filters = {"active_only": ActiveUsersFilter()}
+
+ results, total = UserDAO.list(custom_filters=custom_filters)
+
+ # All results should be active users
+ for result in results:
+ assert result.active is True
+
+
+def test_base_dao_list_base_filter(user_with_data: Session) -> None:
+ """Test BaseDAO.list with base_filter."""
+
+ # Create a DAO with a base filter
+ class FilteredUserDAO(BaseDAO[User]):
+ model_cls = User
+
+ class ActiveFilter(BaseFilter):
+ def apply(self, query, value):
+ return query.filter(User.active.is_(True))
+
+ base_filter = ActiveFilter
+
+ # Create active and inactive users
+ active_user = User(
+ id=600,
+ username="active_base",
+ first_name="Active",
+ last_name="Base",
+ email="active@base.com",
+ active=True,
+ )
+ inactive_user = User(
+ id=601,
+ username="inactive_base",
+ first_name="Inactive",
+ last_name="Base",
+ email="inactive@base.com",
+ active=False,
+ )
+ user_with_data.add_all([active_user, inactive_user])
+ user_with_data.commit()
+
+ # List should only return active users when base filter is applied
+ results, total = FilteredUserDAO.list()
+
+ # All results should be active
+ for result in results:
+ assert result.active is True
+
+ user_with_data.delete(active_user)
+ user_with_data.delete(inactive_user)
+ user_with_data.commit()
+
+
+def test_base_dao_list_edge_cases(user_with_data: Session) -> None:
+ """Test BaseDAO.list with edge cases."""
+ # Test with invalid order column (should not raise)
+ results, total = UserDAO.list(order_column="invalid_column")
+ assert isinstance(results, list)
+ assert isinstance(total, int)
+
+ # Test with invalid order direction (should default to asc)
+ results, total = UserDAO.list(order_direction="invalid")
+ assert isinstance(results, list)
+ assert isinstance(total, int)
+
+ # Test with negative page (should default to 0)
+ results, total = UserDAO.list(page=-1, page_size=10)
+ assert isinstance(results, list)
+ assert isinstance(total, int)
+
+ # Test with zero page size (should use default)
+ results, total = UserDAO.list(page_size=0)
+ assert isinstance(results, list)
+ assert isinstance(total, int)
+
+ # Test with very large page number (should return empty)
+ results, total = UserDAO.list(page=99999, page_size=10)
+ assert results == []
+ assert isinstance(total, int)
+
+ # Test with both search and column operators
+ column_operators = [
+ ColumnOperator(col="username", opr=ColumnOperatorEnum.sw, value="test")
+ ]
+ results, total = UserDAO.list(
+ search="user", search_columns=["username"], column_operators=column_operators
+ )
+ assert isinstance(results, list)
+ assert isinstance(total, int)
+
+
+def test_base_dao_list_with_default_columns(user_with_data: Session) -> None:
+ """Test BaseDAO.list with default columns when select_columns is None."""
+ # Create test user
+ user = User(
+ id=800,
+ username="default_columns_test",
+ first_name="Default",
+ last_name="Columns",
+ email="default@columns.com",
+ active=True,
+ )
+ user_with_data.add(user)
+ user_with_data.commit()
+
+ # Test without select_columns (should use default)
+ results, total = UserDAO.list(
+ column_operators=[
+ ColumnOperator(
+ col="username", opr=ColumnOperatorEnum.eq, value="default_columns_test"
+ )
+ ]
+ )
+
+ assert total == 1
+ assert results[0].username == "default_columns_test"
+
+ user_with_data.delete(user)
+ user_with_data.commit()
+
+
+def test_base_dao_list_with_invalid_operator(app_context: Session) -> None:
+ """Test BaseDAO.list with invalid operator value."""
+ # This test ensures that invalid operators are handled gracefully
+ try:
+ # Try to create an invalid operator (this might raise ValueError)
+ invalid_op = ColumnOperator(col="username", opr="invalid_op", value="test")
+ results, total = UserDAO.list(column_operators=[invalid_op])
+ except (ValueError, KeyError):
+ # Expected behavior - invalid operator is rejected
+ pass
+
+
+def test_base_dao_get_filterable_columns(app_context: Session) -> None:
+ """Test get_filterable_columns_and_operators method."""
+ # Get filterable columns for UserDAO
+ filterable = UserDAO.get_filterable_columns_and_operators()
+
+ # Should return a dict
+ assert isinstance(filterable, dict)
+
+ # Check for expected columns (User model has these)
+ expected_columns = ["id", "username", "email", "active"]
+ for col in expected_columns:
+ assert col in filterable
+ assert isinstance(filterable[col], list)
+ assert len(filterable[col]) > 0 # Should have at least some operators
+
+
+def test_base_dao_count_with_filters(app_context: Session) -> None:
+ """Test BaseDAO.count with various filters."""
+ # Create test users
+ users = []
+ for i in range(5):
+ user = User(
+ username=f"count_test_{i}",
+ first_name=f"Count{i}",
+ last_name="Test",
+ email=f"count{i}@example.com",
+ active=i % 2 == 0, # Alternate active/inactive
+ )
+ users.append(user)
+ db.session.add(user)
+ db.session.commit()
+
+ # Count all test users
+ column_operators = [
+ ColumnOperator(col="username", opr=ColumnOperatorEnum.sw, value="count_test_")
+ ]
+ count = UserDAO.count(column_operators=column_operators)
+ assert count == 5
+
+
+def test_find_by_ids_preserves_order(app_context: Session) -> None:
+ """Test that find_by_ids preserves the order of input IDs."""
+ # Create users with specific IDs
+ users = []
+ for i in [3, 1, 2]: # Create in different order
+ user = User(
+ username=f"order_test_{i}",
+ first_name=f"Order{i}",
+ last_name="Test",
+ email=f"order{i}@example.com",
+ active=True,
+ )
+ users.append(user)
+ db.session.add(user)
+ db.session.commit()
+
+ # Get their IDs in a specific order
+ user_ids = [users[1].id, users[2].id, users[0].id] # 1, 2, 3
+
+ # Find by IDs
+ found = UserDAO.find_by_ids(user_ids, skip_base_filter=True)
+
+ # The order might not be preserved by default SQL behavior
+ # but we should get all users back
+ assert len(found) == 3
+ found_ids = [u.id for u in found]
+ assert set(found_ids) == set(user_ids)
+
+
+def test_find_by_ids_with_mixed_types(app_context: Session) -> None:
+ """Test find_by_ids with mixed ID types (int, str, uuid)."""
+ # Create a database with mixed ID types
+ database = Database(
+ database_name="test_mixed_ids_db",
+ sqlalchemy_uri="sqlite:///:memory:",
+ )
+ db.session.add(database)
+ db.session.commit()
+
+ # Find by numeric ID
+ found = DatabaseDAO.find_by_ids([database.id], skip_base_filter=True)
+ assert len(found) == 1
+ assert found[0].id == database.id
+
+
+def test_find_by_column_helper_method(app_context: Session) -> None:
+ """Test the _find_by_column helper method."""
+ # Create a user
+ user = User(
+ username="find_by_column_test",
+ first_name="FindBy",
+ last_name="Column",
+ email="findby@column.com",
+ active=True,
+ )
+ db.session.add(user)
+ db.session.commit()
+
+ # Use the helper method directly
+ found = UserDAO._find_by_column("username", "find_by_column_test")
+ assert found is not None
+ assert found.username == "find_by_column_test"
+
+ # Test with non-existent value
+ not_found = UserDAO._find_by_column("username", "non_existent")
+ assert not_found is None
+
+
+def test_find_methods_with_special_characters(app_context: Session) -> None:
+ """Test find methods with special characters in values."""
+ # Create a dashboard with special characters in slug
+ dashboard = Dashboard(
+ dashboard_title="Test Special Chars",
+ slug="test-special_chars.v1",
+ published=True,
+ )
+ db.session.add(dashboard)
+ db.session.commit()
+
+ # Find by slug with special characters
+ found = DashboardDAO.find_by_id(
+ "test-special_chars.v1", id_column="slug", skip_base_filter=True
+ )
+ assert found is not None
+ assert found.slug == "test-special_chars.v1"
+
+ # Find by IDs with special characters
+ found_list = DashboardDAO.find_by_ids(
+ ["test-special_chars.v1"], id_column="slug", skip_base_filter=True
+ )
+ assert len(found_list) == 1
+ assert found_list[0].slug == "test-special_chars.v1"
+
+
+def test_find_methods_case_sensitivity(app_context: Session) -> None:
+ """Test find methods with case sensitivity."""
+ # Create users with similar usernames differing in case
+ user1 = User(
+ username="CaseSensitive",
+ first_name="Case",
+ last_name="Sensitive",
+ email="case1@example.com",
+ active=True,
+ )
+ user2 = User(
+ username="casesensitive",
+ first_name="Case",
+ last_name="Insensitive",
+ email="case2@example.com",
+ active=True,
+ )
+ db.session.add_all([user1, user2])
+
+ try:
+ db.session.commit()
+
+ # Find with exact case
+ found = UserDAO.find_one_or_none(username="CaseSensitive")
+ assert found is not None
+ assert found.username == "CaseSensitive"
+
+ # Find with different case
+ found_lower = UserDAO.find_one_or_none(username="casesensitive")
+ assert found_lower is not None
+ assert found_lower.username == "casesensitive"
+
+ finally:
+ pass
Review Comment:
<div>
<div id="suggestion">
<div id="issue"><b>Missing test cleanup</b></div>
<div id="fix">
The test case sensitivity method has a try-finally block with an empty pass statement that doesn't properly clean up test data. This could lead to test pollution where failed tests leave data in the database, affecting subsequent tests. Add proper database cleanup using `db.session.rollback()` and `db.session.remove()` to ensure test isolation.
</div>
<details>
<summary>
<b>Code suggestion</b>
</summary>
<blockquote>Check the AI-generated fix before applying</blockquote>
<div id="code">
```suggestion
finally:
db.session.rollback()
db.session.remove()
```
</div>
</details>
</div>
<small><i>Code Review Run <a href=https://github.com/apache/superset/pull/35018#issuecomment-3293390307>#8dadc1</a></i></small>
</div>
---
Should Bito avoid suggestions like this for future reviews? (<a href=https://alpha.bito.ai/home/ai-agents/review-rules>Manage Rules</a>)
- [ ] Yes, avoid them
##########
superset/daos/base.py:
##########
@@ -139,11 +346,7 @@
Get all that fit the `base_filter`
"""
query = db.session.query(cls.model_cls)
- if cls.base_filter:
- data_model = SQLAInterface(cls.model_cls, db.session)
- query = cls.base_filter( # pylint: disable=not-callable
- cls.id_column_name, data_model
- ).apply(query, None)
+ query = cls._apply_base_filter(query)
Review Comment:
<div>
<div id="suggestion">
<div id="issue"><b>Inconsistent base filter handling</b></div>
<div id="fix">
The refactoring to use `_apply_base_filter` is incomplete. The `find_all` method should accept a `skip_base_filter` parameter to maintain consistency with other query methods like `find_by_id_or_uuid`, `find_by_column`, and `find_by_ids` that allow skipping base filters. Add a `skip_base_filter: bool = False` parameter to `find_all` and pass it to `_apply_base_filter`.
</div>
<details>
<summary>
<b>Code suggestion</b>
</summary>
<blockquote>Check the AI-generated fix before applying</blockquote>
<div id="code">
```
- def find_all(cls) -> list[T]:
+ def find_all(cls, skip_base_filter: bool = False) -> list[T]:
"""
Get all that fit the `base_filter`
"""
query = db.session.query(cls.model_cls)
- query = cls._apply_base_filter(query)
+ query = cls._apply_base_filter(query, skip_base_filter)
return query.all()
```
</div>
</details>
</div>
<small><i>Code Review Run <a href=https://github.com/apache/superset/pull/35018#issuecomment-3293390307>#8dadc1</a></i></small>
</div>
---
Should Bito avoid suggestions like this for future reviews? (<a href=https://alpha.bito.ai/home/ai-agents/review-rules>Manage Rules</a>)
- [ ] Yes, avoid them
##########
tests/integration_tests/dao/base_dao_test.py:
##########
@@ -0,0 +1,1619 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""
+Integration tests for BaseDAO functionality.
+
+This module contains comprehensive integration tests for the BaseDAO class and its
+subclasses, covering database operations, CRUD methods, flexible column support,
+column operators, and error handling.
+
+Tests use an in-memory SQLite database for isolation and to replicate the unit test
+environment behavior. User model deletions are avoided due to circular dependency
+constraints with self-referential foreign keys.
+"""
+
+import datetime
+import time
+import uuid
+
+import pytest
+from flask_appbuilder.models.filters import BaseFilter
+from flask_appbuilder.security.sqla.models import User
+from sqlalchemy import Column, DateTime, Integer, String
+from sqlalchemy.ext.declarative import declarative_base
+from sqlalchemy.orm.session import Session
+
+from superset.daos.base import BaseDAO, ColumnOperator, ColumnOperatorEnum
+from superset.daos.chart import ChartDAO
+from superset.daos.dashboard import DashboardDAO
+from superset.daos.database import DatabaseDAO
+from superset.daos.user import UserDAO
+from superset.extensions import db
+from superset.models.core import Database
+from superset.models.dashboard import Dashboard
+from superset.models.slice import Slice
+
+# Create a test model for comprehensive testing
+Base = declarative_base()
+
+
+class ExampleModel(Base): # type: ignore
+ __tablename__ = "example_model"
+ id = Column(Integer, primary_key=True)
+ uuid = Column(String(36), unique=True, nullable=False)
+ slug = Column(String(100), unique=True)
+ name = Column(String(100))
+ code = Column(String(50), unique=True)
+ created_on = Column(DateTime, default=datetime.datetime.utcnow)
+
+
+class ExampleModelDAO(BaseDAO[ExampleModel]):
+ model_cls = ExampleModel
+ id_column_name = "id"
+ base_filter = None
+
+
+class MockModel:
+ def __init__(self, id=1, name="test"):
+ self.id = id
+ self.name = name
+
+
+class TestDAO(BaseDAO[MockModel]):
+ model_cls = MockModel
+
+
+@pytest.fixture(autouse=True)
+def mock_g_user(app_context):
+ """Mock the flask g.user for security context."""
+ # Within app context, we can safely mock g
+ from flask import g
+
+ mock_user = User()
+ mock_user.id = 1
+ mock_user.username = "test_user"
+
+ # Set g.user directly instead of patching
+ g.user = mock_user
+ yield
+
+ # Clean up
+ if hasattr(g, "user"):
+ delattr(g, "user")
+
+
+# =============================================================================
+# Integration Tests - These tests use the actual database
+# =============================================================================
+
+
+def test_column_operator_enum_complete_coverage(user_with_data: Session) -> None:
+ """
+ Test that every single ColumnOperatorEnum operator is covered by tests.
+ This ensures we have comprehensive test coverage for all operators.
+ """
+ # Simply verify that we can create queries with all operators
+ for operator in ColumnOperatorEnum:
+ column_operator = ColumnOperator(
+ col="username", opr=operator, value="test_value"
+ )
+ # Just check it doesn't raise an error
+ assert column_operator.opr == operator
+
+
+def test_find_by_id_with_default_column(app_context: Session) -> None:
+ """Test find_by_id with default 'id' column."""
+ # Create a user to test with
+ user = User(
+ username="test_find_by_id",
+ first_name="Test",
+ last_name="User",
+ email="test@example.com",
+ active=True,
+ )
+ db.session.add(user)
+ db.session.commit()
+
+ # Find by numeric id
+ found = UserDAO.find_by_id(user.id, skip_base_filter=True)
+ assert found is not None
+ assert found.id == user.id
+ assert found.username == "test_find_by_id"
+
+ # Test with non-existent id
+ not_found = UserDAO.find_by_id(999999, skip_base_filter=True)
+ assert not_found is None
+
+
+def test_find_by_id_with_uuid_column(app_context: Session) -> None:
+ """Test find_by_id with custom uuid column."""
+ # Create a dashboard with uuid
+ dashboard = Dashboard(
+ dashboard_title="Test UUID Dashboard",
+ slug="test-uuid-dashboard",
+ published=True,
+ )
+ db.session.add(dashboard)
+ db.session.commit()
+
+ # Find by uuid string using the uuid column
+ found = DashboardDAO.find_by_id(
+ str(dashboard.uuid), id_column="uuid", skip_base_filter=True
+ )
+ assert found is not None
+ assert found.uuid == dashboard.uuid
+ assert found.dashboard_title == "Test UUID Dashboard"
+
+ # Find by numeric id (should still work)
+ found_by_id = DashboardDAO.find_by_id(dashboard.id, skip_base_filter=True)
+ assert found_by_id is not None
+ assert found_by_id.id == dashboard.id
+
+ # Test with non-existent uuid
+ not_found = DashboardDAO.find_by_id(str(uuid.uuid4()), skip_base_filter=True)
+ assert not_found is None
+
+
+def test_find_by_id_with_slug_column(app_context: Session) -> None:
+ """Test find_by_id with slug column fallback."""
+ # Create a dashboard with slug
+ dashboard = Dashboard(
+ dashboard_title="Test Slug Dashboard",
+ slug="test-slug-dashboard",
+ published=True,
+ )
+ db.session.add(dashboard)
+ db.session.commit()
+
+ # Find by slug using the slug column
+ found = DashboardDAO.find_by_id(
+ "test-slug-dashboard", id_column="slug", skip_base_filter=True
+ )
+ assert found is not None
+ assert found.slug == "test-slug-dashboard"
+ assert found.dashboard_title == "Test Slug Dashboard"
+
+ # Test with non-existent slug
+ not_found = DashboardDAO.find_by_id("non-existent-slug", skip_base_filter=True)
+ assert not_found is None
+
+
+def test_find_by_id_with_invalid_column(app_context: Session) -> None:
+ """Test find_by_id returns None when column doesn't exist."""
+ # This should return None gracefully
+ result = UserDAO.find_by_id("not_a_valid_id", skip_base_filter=True)
+ assert result is None
+
+
+def test_find_by_id_skip_base_filter(app_context: Session) -> None:
+ """Test find_by_id with skip_base_filter parameter."""
+ # Create users with different active states
+ active_user = User(
+ username="active_user",
+ first_name="Active",
+ last_name="User",
+ email="active@example.com",
+ active=True,
+ )
+ inactive_user = User(
+ username="inactive_user",
+ first_name="Inactive",
+ last_name="User",
+ email="inactive@example.com",
+ active=False,
+ )
+ db.session.add_all([active_user, inactive_user])
+ db.session.commit()
+
+ # Without skipping base filter (if one exists)
+ found_active = UserDAO.find_by_id(active_user.id, skip_base_filter=False)
+ assert found_active is not None
+
+ # With skipping base filter
+ found_active_skip = UserDAO.find_by_id(active_user.id, skip_base_filter=True)
+ assert found_active_skip is not None
+
+ # Both should find the user since UserDAO might not have a base filter
+ assert found_active.id == active_user.id
+ assert found_active_skip.id == active_user.id
+
+
+def test_find_by_ids_with_default_column(app_context: Session) -> None:
+ """Test find_by_ids with default 'id' column."""
+ # Create multiple users
+ users = []
+ for i in range(3):
+ user = User(
+ username=f"test_find_by_ids_{i}",
+ first_name=f"Test{i}",
+ last_name="User",
+ email=f"test{i}@example.com",
+ active=True,
+ )
+ users.append(user)
+ db.session.add(user)
+ db.session.commit()
+
+ # Find by multiple ids
+ ids = [user.id for user in users]
+ found = UserDAO.find_by_ids(ids, skip_base_filter=True)
+ assert len(found) == 3
+ found_ids = [u.id for u in found]
+ assert set(found_ids) == set(ids)
+
+ # Test with mix of existent and non-existent ids
+ mixed_ids = [users[0].id, 999999, users[1].id]
+ found_mixed = UserDAO.find_by_ids(mixed_ids, skip_base_filter=True)
+ assert len(found_mixed) == 2
+
+ # Test with empty list
+ found_empty = UserDAO.find_by_ids([], skip_base_filter=True)
+ assert found_empty == []
+
+
+def test_find_by_ids_with_uuid_column(app_context: Session) -> None:
+ """Test find_by_ids with uuid column."""
+ # Create multiple dashboards
+ dashboards = []
+ for i in range(3):
+ dashboard = Dashboard(
+ dashboard_title=f"Test UUID Dashboard {i}",
+ slug=f"test-uuid-dashboard-{i}",
+ published=True,
+ )
+ dashboards.append(dashboard)
+ db.session.add(dashboard)
+ db.session.commit()
+
+ # Find by multiple uuids
+ uuids = [str(dashboard.uuid) for dashboard in dashboards]
+ found = DashboardDAO.find_by_ids(uuids, id_column="uuid", skip_base_filter=True)
+ assert len(found) == 3
+ found_uuids = [str(d.uuid) for d in found]
+ assert set(found_uuids) == set(uuids)
+
+ # Test with mix of ids and uuids - search separately by column
+ found_by_id = DashboardDAO.find_by_ids([dashboards[0].id], skip_base_filter=True)
+ found_by_uuid = DashboardDAO.find_by_ids(
+ [str(dashboards[1].uuid)], id_column="uuid", skip_base_filter=True
+ )
+ assert len(found_by_id) == 1
+ assert len(found_by_uuid) == 1
+
+
+def test_find_by_ids_with_slug_column(app_context: Session) -> None:
+ """Test find_by_ids with slug column."""
+ # Create multiple dashboards
+ dashboards = []
+ for i in range(3):
+ dashboard = Dashboard(
+ dashboard_title=f"Test Slug Dashboard {i}",
+ slug=f"test-slug-dashboard-{i}",
+ published=True,
+ )
+ dashboards.append(dashboard)
+ db.session.add(dashboard)
+ db.session.commit()
+
+ # Find by multiple slugs
+ slugs = [dashboard.slug for dashboard in dashboards]
+ found = DashboardDAO.find_by_ids(slugs, id_column="slug", skip_base_filter=True)
+ assert len(found) == 3
+ found_slugs = [d.slug for d in found]
+ assert set(found_slugs) == set(slugs)
+
+
+def test_find_by_ids_with_invalid_column(app_context: Session) -> None:
+ """Test find_by_ids returns empty list when column doesn't exist."""
+ # This should return empty list gracefully
+ result = UserDAO.find_by_ids(["not_a_valid_id"], skip_base_filter=True)
+ assert result == []
+
+
+def test_find_by_ids_skip_base_filter(app_context: Session) -> None:
+ """Test find_by_ids with skip_base_filter parameter."""
+ # Create users
+ users = []
+ for i in range(3):
+ user = User(
+ username=f"test_skip_filter_{i}",
+ first_name=f"Test{i}",
+ last_name="User",
+ email=f"test{i}@example.com",
+ active=True,
+ )
+ users.append(user)
+ db.session.add(user)
+ db.session.commit()
+
+ ids = [user.id for user in users]
+
+ # Without skipping base filter
+ found_no_skip = UserDAO.find_by_ids(ids, skip_base_filter=False)
+ assert len(found_no_skip) == 3
+
+ # With skipping base filter
+ found_skip = UserDAO.find_by_ids(ids, skip_base_filter=True)
+ assert len(found_skip) == 3
+
+
+def test_base_dao_create_with_item(app_context: Session) -> None:
+ """Test BaseDAO.create with an item parameter."""
+ # Create a user item
+ user = User(
+ username="created_with_item",
+ first_name="Created",
+ last_name="Item",
+ email="created@example.com",
+ active=True,
+ )
+
+ # Create using the item
+ created = UserDAO.create(item=user)
+ assert created is not None
+ assert created.username == "created_with_item"
+ assert created.first_name == "Created"
+
+ # Verify it's in the session
+ assert created in db.session
+
+ # Commit and verify it persists
+ db.session.commit()
+
+ # Find it again to ensure it was saved
+ found = UserDAO.find_by_id(created.id, skip_base_filter=True)
+ assert found is not None
+ assert found.username == "created_with_item"
+
+
+def test_base_dao_create_with_attributes(app_context: Session) -> None:
+ """Test BaseDAO.create with attributes parameter."""
+ # Create using attributes dict
+ attributes = {
+ "username": "created_with_attrs",
+ "first_name": "Created",
+ "last_name": "Attrs",
+ "email": "attrs@example.com",
+ "active": True,
+ }
+
+ created = UserDAO.create(attributes=attributes)
+ assert created is not None
+ assert created.username == "created_with_attrs"
+ assert created.email == "attrs@example.com"
+
+ # Commit and verify
+ db.session.commit()
+ found = UserDAO.find_by_id(created.id, skip_base_filter=True)
+ assert found is not None
+ assert found.username == "created_with_attrs"
+
+
+def test_base_dao_create_with_both_item_and_attributes(app_context: Session) -> None:
+ """Test BaseDAO.create with both item and attributes (override behavior)."""
+ # Create a user item
+ user = User(
+ username="item_username",
+ first_name="Item",
+ last_name="User",
+ email="item@example.com",
+ active=False,
+ )
+
+ # Override some attributes
+ attributes = {
+ "username": "override_username",
+ "active": True,
+ }
+
+ created = UserDAO.create(item=user, attributes=attributes)
+ assert created is not None
+ assert created.username == "override_username" # Should be overridden
+ assert created.active is True # Should be overridden
+ assert created.first_name == "Item" # Should keep original
+ assert created.last_name == "User" # Should keep original
+
+ db.session.commit()
+
+
+def test_base_dao_update_with_item(app_context: Session) -> None:
+ """Test BaseDAO.update with an item parameter."""
+ # Create a user first
+ user = User(
+ username="update_test",
+ first_name="Original",
+ last_name="User",
+ email="original@example.com",
+ active=True,
+ )
+ db.session.add(user)
+ db.session.commit()
+
+ # Update the user
+ user.first_name = "Updated"
+ updated = UserDAO.update(item=user)
+ assert updated is not None
+ assert updated.first_name == "Updated"
+
+ db.session.commit()
+
+ # Verify the update persisted
+ found = UserDAO.find_by_id(user.id, skip_base_filter=True)
+ assert found is not None
+ assert found.first_name == "Updated"
+
+
+def test_base_dao_update_with_attributes(app_context: Session) -> None:
+ """Test BaseDAO.update with attributes parameter."""
+ # Create a user first
+ user = User(
+ username="update_attrs_test",
+ first_name="Original",
+ last_name="User",
+ email="original@example.com",
+ active=True,
+ )
+ db.session.add(user)
+ db.session.commit()
+
+ # Update using attributes
+ attributes = {"first_name": "Updated", "last_name": "Attr User"}
+ updated = UserDAO.update(item=user, attributes=attributes)
+ assert updated is not None
+ assert updated.first_name == "Updated"
+ assert updated.last_name == "Attr User"
+
+ db.session.commit()
+
+
+def test_base_dao_update_detached_item(app_context: Session) -> None:
+ """Test BaseDAO.update with a detached item."""
+ # Create a user first
+ user = User(
+ username="detached_test",
+ first_name="Original",
+ last_name="User",
+ email="detached@example.com",
+ active=True,
+ )
+ db.session.add(user)
+ db.session.commit()
+
+ user_id = user.id
+
+ # Expunge to detach from session
+ db.session.expunge(user)
+
+ # Update the detached user
+ user.first_name = "Updated Detached"
+ updated = UserDAO.update(item=user)
+ assert updated is not None
+ assert updated.first_name == "Updated Detached"
+
+ db.session.commit()
+
+ # Verify the update persisted
+ found = UserDAO.find_by_id(user_id, skip_base_filter=True)
+ assert found is not None
+ assert found.first_name == "Updated Detached"
+
+
+def test_base_dao_delete_single_item(app_context: Session) -> None:
+ """Test BaseDAO.delete with a single item."""
+ # Create a dashboard instead of user to avoid circular dependencies
+ dashboard = Dashboard(
+ dashboard_title="Delete Test",
+ slug="delete-test",
+ published=True,
+ )
+ db.session.add(dashboard)
+ db.session.commit()
+
+ dashboard_id = dashboard.id
+
+ DashboardDAO.delete([dashboard])
+ db.session.commit()
+
+ # Verify it's gone
+ found = DashboardDAO.find_by_id(dashboard_id, skip_base_filter=True)
+ assert found is None
+
+
+def test_base_dao_delete_multiple_items(app_context: Session) -> None:
+ """Test BaseDAO.delete with multiple items."""
+ # Create multiple dashboards instead of users to avoid circular dependencies
+ dashboards = []
+ for i in range(3):
+ dashboard = Dashboard(
+ dashboard_title=f"Delete Multi {i}",
+ slug=f"delete-multi-{i}",
+ published=True,
+ )
+ dashboards.append(dashboard)
+ db.session.add(dashboard)
+ db.session.commit()
+
+ dashboard_ids = [dashboard.id for dashboard in dashboards]
+
+ DashboardDAO.delete(dashboards)
+ db.session.commit()
+
+ # Verify they're all gone
+ for dashboard_id in dashboard_ids:
+ found = DashboardDAO.find_by_id(dashboard_id, skip_base_filter=True)
+ assert found is None
+
+
+def test_base_dao_delete_empty_list(app_context: Session) -> None:
+ """Test BaseDAO.delete with empty list."""
+ # Should not raise any errors
+ UserDAO.delete([])
+ db.session.commit()
+ # Just ensuring no exception is raised
+
+
+def test_base_dao_find_all(app_context: Session) -> None:
+ """Test BaseDAO.find_all method."""
+ # Create some users
+ users = []
+ for i in range(3):
+ user = User(
+ username=f"find_all_{i}",
+ first_name=f"Find{i}",
+ last_name="All",
+ email=f"findall{i}@example.com",
+ active=True,
+ )
+ users.append(user)
+ db.session.add(user)
+ db.session.commit()
+
+ # Find all users
+ all_users = UserDAO.find_all()
+ assert len(all_users) >= 3 # At least our 3 users
+
+ # Check our users are in the results
+ usernames = [u.username for u in all_users]
+ for user in users:
+ assert user.username in usernames
+
+
+def test_base_dao_find_one_or_none(app_context: Session) -> None:
+ """Test BaseDAO.find_one_or_none method."""
+ # Create users with specific criteria
+ user1 = User(
+ username="unique_username_123",
+ first_name="Unique",
+ last_name="User",
+ email="unique@example.com",
+ active=True,
+ )
+ user2 = User(
+ username="another_user",
+ first_name="Another",
+ last_name="User",
+ email="another@example.com",
+ active=True,
+ )
+ user3 = User(
+ username="third_user",
+ first_name="Another", # Same first name as user2
+ last_name="User",
+ email="third@example.com",
+ active=True,
+ )
+ db.session.add_all([user1, user2, user3])
+ db.session.commit()
+
+ # Find one with unique criteria
+ found = UserDAO.find_one_or_none(username="unique_username_123")
+ assert found is not None
+ assert found.username == "unique_username_123"
+
+ # Find none with non-existent criteria
+ not_found = UserDAO.find_one_or_none(username="non_existent_user")
+ assert not_found is None
+
+ db.session.commit()
+
+
+def test_base_dao_list_returns_results(user_with_data: Session) -> None:
+ """Test that BaseDAO.list returns results and total."""
+ results, total = UserDAO.list()
+ assert isinstance(results, list)
+ assert isinstance(total, int)
+ assert total >= 1 # At least the fixture user
+
+
+def test_base_dao_list_with_column_operators(user_with_data: Session) -> None:
+ """Test BaseDAO.list with column operators."""
+ column_operators = [
+ ColumnOperator(col="username", opr=ColumnOperatorEnum.eq, value="testuser")
+ ]
+ results, total = UserDAO.list(column_operators=column_operators)
+ assert total == 1
+ assert results[0].username == "testuser"
+
+
+def test_base_dao_list_with_non_matching_column_operator(
+ user_with_data: Session,
+) -> None:
+ """Test BaseDAO.list with non-matching column operator."""
+ column_operators = [
+ ColumnOperator(
+ col="username", opr=ColumnOperatorEnum.eq, value="nonexistentuser"
+ )
+ ]
+ results, total = UserDAO.list(column_operators=column_operators)
+ assert total == 0
+ assert results == []
+
+
+def test_base_dao_count_returns_value(user_with_data: Session) -> None:
+ """Test that BaseDAO.count returns correct count."""
+ count = UserDAO.count()
+ assert isinstance(count, int)
+ assert count >= 1 # At least the fixture user
+
+
+def test_base_dao_count_with_column_operators(user_with_data: Session) -> None:
+ """Test BaseDAO.count with column operators."""
+ # Count with matching operator
+ column_operators = [
+ ColumnOperator(col="username", opr=ColumnOperatorEnum.eq, value="testuser")
+ ]
+ count = UserDAO.count(column_operators=column_operators)
+ assert count == 1
+
+ # Count with non-matching operator
+ column_operators = [
+ ColumnOperator(
+ col="username", opr=ColumnOperatorEnum.eq, value="nonexistentuser"
+ )
+ ]
+ count = UserDAO.count(column_operators=column_operators)
+ assert count == 0
+
+
+def test_base_dao_list_ordering(user_with_data: Session) -> None:
+ """Test BaseDAO.list with ordering."""
+ # Create additional users with predictable names
+ users = []
+ for i in range(3):
+ user = User(
+ id=200 + i, # Ensure predictable IDs
+ username=f"order_test_{i}",
+ first_name=f"Order{i}",
+ last_name="Test",
+ email=f"order{i}@example.com",
+ active=True,
+ )
+ users.append(user)
+ user_with_data.add(user)
+ user_with_data.commit()
+
+ # Test ascending order
+ results_asc, _ = UserDAO.list(
+ order_column="username", order_direction="asc", page_size=100
+ )
+ usernames_asc = [r.username for r in results_asc]
+ # Check that our test users are in order
+ assert usernames_asc.index("order_test_0") < usernames_asc.index("order_test_1")
+ assert usernames_asc.index("order_test_1") < usernames_asc.index("order_test_2")
+
+ # Test descending order
+ results_desc, _ = UserDAO.list(
+ order_column="username", order_direction="desc", page_size=100
+ )
+ usernames_desc = [r.username for r in results_desc]
+ # Check that our test users are in reverse order
+ assert usernames_desc.index("order_test_2") < usernames_desc.index("order_test_1")
+ assert usernames_desc.index("order_test_1") < usernames_desc.index("order_test_0")
+
+ for user in users:
+ user_with_data.delete(user)
+ user_with_data.commit()
+
+
+def test_base_dao_list_paging(user_with_data: Session) -> None:
+ """Test BaseDAO.list with paging."""
+ # Create additional users for paging
+ users = []
+ for i in range(10):
+ user = User(
+ id=300 + i,
+ username=f"page_test_{i}",
+ first_name=f"Page{i}",
+ last_name="Test",
+ email=f"page{i}@example.com",
+ active=True,
+ )
+ users.append(user)
+ user_with_data.add(user)
+ user_with_data.commit()
+
+ # Test first page
+ page1_results, page1_total = UserDAO.list(page=0, page_size=5)
+ assert len(page1_results) <= 5
+ assert page1_total >= 10 # At least our 10 users
+
+ # Test second page
+ page2_results, page2_total = UserDAO.list(page=1, page_size=5)
+ assert len(page2_results) <= 5
+ assert page2_total >= 10
+
+ # Results should be different
+ page1_ids = [r.id for r in page1_results]
+ page2_ids = [r.id for r in page2_results]
+ assert set(page1_ids).isdisjoint(set(page2_ids)) # No overlap
+
+ for user in users:
+ user_with_data.delete(user)
+ user_with_data.commit()
+
+
+def test_base_dao_list_search(user_with_data: Session) -> None:
+ """Test BaseDAO.list with search."""
+ # Create users with searchable names
+ users = []
+ for i in range(3):
+ user = User(
+ id=400 + i,
+ username=f"searchable_{i}",
+ first_name=f"Searchable{i}",
+ last_name="User",
+ email=f"search{i}@example.com",
+ active=True,
+ )
+ users.append(user)
+ user_with_data.add(user)
+
+ # Create some non-matching users
+ for i in range(2):
+ user = User(
+ id=410 + i,
+ username=f"other_{i}",
+ first_name=f"Other{i}",
+ last_name="Person",
+ email=f"other{i}@example.com",
+ active=True,
+ )
+ users.append(user)
+ user_with_data.add(user)
+
+ user_with_data.commit()
+
+ # Search for "searchable"
+ results, total = UserDAO.list(
+ search="searchable", search_columns=["username", "first_name"]
+ )
+ assert total >= 3 # At least our 3 searchable users
+
+ # Verify all results contain "searchable"
+ for result in results:
+ assert (
+ "searchable" in result.username.lower()
+ or "searchable" in result.first_name.lower()
+ )
+
+ for user in users:
+ user_with_data.delete(user)
+ user_with_data.commit()
+
+
+def test_base_dao_list_custom_filter(user_with_data: Session) -> None:
+ """Test BaseDAO.list with custom filters."""
+ # Create users with specific attributes
+ active_user = User(
+ id=500,
+ username="active_custom",
+ first_name="Active",
+ last_name="Custom",
+ email="active@custom.com",
+ active=True,
+ )
+ inactive_user = User(
+ id=501,
+ username="inactive_custom",
+ first_name="Inactive",
+ last_name="Custom",
+ email="inactive@custom.com",
+ active=False,
+ )
+ user_with_data.add_all([active_user, inactive_user])
+ user_with_data.commit()
+
+ # Create a custom filter for active users only
+ class ActiveUsersFilter(BaseFilter):
+ def __init__(self):
+ self.column_name = "active"
+ self.datamodel = None
+ self.model = User # Set model directly
+
+ def apply(self, query, value):
+ return query.filter(User.active.is_(True))
+
+ custom_filters = {"active_only": ActiveUsersFilter()}
+
+ results, total = UserDAO.list(custom_filters=custom_filters)
+
+ # All results should be active users
+ for result in results:
+ assert result.active is True
+
+
+def test_base_dao_list_base_filter(user_with_data: Session) -> None:
+ """Test BaseDAO.list with base_filter."""
+
+ # Create a DAO with a base filter
+ class FilteredUserDAO(BaseDAO[User]):
+ model_cls = User
+
+ class ActiveFilter(BaseFilter):
+ def apply(self, query, value):
+ return query.filter(User.active.is_(True))
+
+ base_filter = ActiveFilter
+
+ # Create active and inactive users
+ active_user = User(
+ id=600,
+ username="active_base",
+ first_name="Active",
+ last_name="Base",
+ email="active@base.com",
+ active=True,
+ )
+ inactive_user = User(
+ id=601,
+ username="inactive_base",
+ first_name="Inactive",
+ last_name="Base",
+ email="inactive@base.com",
+ active=False,
+ )
+ user_with_data.add_all([active_user, inactive_user])
+ user_with_data.commit()
+
+ # List should only return active users when base filter is applied
+ results, total = FilteredUserDAO.list()
+
+ # All results should be active
+ for result in results:
+ assert result.active is True
+
+ user_with_data.delete(active_user)
+ user_with_data.delete(inactive_user)
+ user_with_data.commit()
+
+
+def test_base_dao_list_edge_cases(user_with_data: Session) -> None:
+ """Test BaseDAO.list with edge cases."""
+ # Test with invalid order column (should not raise)
+ results, total = UserDAO.list(order_column="invalid_column")
+ assert isinstance(results, list)
+ assert isinstance(total, int)
+
+ # Test with invalid order direction (should default to asc)
+ results, total = UserDAO.list(order_direction="invalid")
+ assert isinstance(results, list)
+ assert isinstance(total, int)
+
+ # Test with negative page (should default to 0)
+ results, total = UserDAO.list(page=-1, page_size=10)
+ assert isinstance(results, list)
+ assert isinstance(total, int)
+
+ # Test with zero page size (should use default)
+ results, total = UserDAO.list(page_size=0)
+ assert isinstance(results, list)
+ assert isinstance(total, int)
+
+ # Test with very large page number (should return empty)
+ results, total = UserDAO.list(page=99999, page_size=10)
+ assert results == []
+ assert isinstance(total, int)
+
+ # Test with both search and column operators
+ column_operators = [
+ ColumnOperator(col="username", opr=ColumnOperatorEnum.sw, value="test")
+ ]
+ results, total = UserDAO.list(
+ search="user", search_columns=["username"], column_operators=column_operators
+ )
+ assert isinstance(results, list)
+ assert isinstance(total, int)
+
+
+def test_base_dao_list_with_default_columns(user_with_data: Session) -> None:
+ """Test BaseDAO.list with default columns when select_columns is None."""
+ # Create test user
+ user = User(
+ id=800,
+ username="default_columns_test",
+ first_name="Default",
+ last_name="Columns",
+ email="default@columns.com",
+ active=True,
+ )
+ user_with_data.add(user)
+ user_with_data.commit()
+
+ # Test without select_columns (should use default)
+ results, total = UserDAO.list(
+ column_operators=[
+ ColumnOperator(
+ col="username", opr=ColumnOperatorEnum.eq, value="default_columns_test"
+ )
+ ]
+ )
+
+ assert total == 1
+ assert results[0].username == "default_columns_test"
+
+ user_with_data.delete(user)
+ user_with_data.commit()
+
+
+def test_base_dao_list_with_invalid_operator(app_context: Session) -> None:
+ """Test BaseDAO.list with invalid operator value."""
+ # This test ensures that invalid operators are handled gracefully
+ try:
+ # Try to create an invalid operator (this might raise ValueError)
+ invalid_op = ColumnOperator(col="username", opr="invalid_op", value="test")
+ results, total = UserDAO.list(column_operators=[invalid_op])
+ except (ValueError, KeyError):
+ # Expected behavior - invalid operator is rejected
+ pass
+
+
+def test_base_dao_get_filterable_columns(app_context: Session) -> None:
+ """Test get_filterable_columns_and_operators method."""
+ # Get filterable columns for UserDAO
+ filterable = UserDAO.get_filterable_columns_and_operators()
+
+ # Should return a dict
+ assert isinstance(filterable, dict)
+
+ # Check for expected columns (User model has these)
+ expected_columns = ["id", "username", "email", "active"]
+ for col in expected_columns:
+ assert col in filterable
+ assert isinstance(filterable[col], list)
+ assert len(filterable[col]) > 0 # Should have at least some operators
+
+
+def test_base_dao_count_with_filters(app_context: Session) -> None:
+ """Test BaseDAO.count with various filters."""
+ # Create test users
+ users = []
+ for i in range(5):
+ user = User(
+ username=f"count_test_{i}",
+ first_name=f"Count{i}",
+ last_name="Test",
+ email=f"count{i}@example.com",
+ active=i % 2 == 0, # Alternate active/inactive
+ )
+ users.append(user)
+ db.session.add(user)
+ db.session.commit()
+
+ # Count all test users
+ column_operators = [
+ ColumnOperator(col="username", opr=ColumnOperatorEnum.sw, value="count_test_")
+ ]
+ count = UserDAO.count(column_operators=column_operators)
+ assert count == 5
+
+
+def test_find_by_ids_preserves_order(app_context: Session) -> None:
+ """Test that find_by_ids preserves the order of input IDs."""
+ # Create users with specific IDs
+ users = []
+ for i in [3, 1, 2]: # Create in different order
+ user = User(
+ username=f"order_test_{i}",
+ first_name=f"Order{i}",
+ last_name="Test",
+ email=f"order{i}@example.com",
+ active=True,
+ )
+ users.append(user)
+ db.session.add(user)
+ db.session.commit()
+
+ # Get their IDs in a specific order
+ user_ids = [users[1].id, users[2].id, users[0].id] # 1, 2, 3
+
+ # Find by IDs
+ found = UserDAO.find_by_ids(user_ids, skip_base_filter=True)
+
+ # The order might not be preserved by default SQL behavior
+ # but we should get all users back
+ assert len(found) == 3
+ found_ids = [u.id for u in found]
+ assert set(found_ids) == set(user_ids)
+
+
+def test_find_by_ids_with_mixed_types(app_context: Session) -> None:
+ """Test find_by_ids with mixed ID types (int, str, uuid)."""
+ # Create a database with mixed ID types
+ database = Database(
+ database_name="test_mixed_ids_db",
+ sqlalchemy_uri="sqlite:///:memory:",
+ )
+ db.session.add(database)
+ db.session.commit()
+
+ # Find by numeric ID
+ found = DatabaseDAO.find_by_ids([database.id], skip_base_filter=True)
+ assert len(found) == 1
+ assert found[0].id == database.id
+
+
+def test_find_by_column_helper_method(app_context: Session) -> None:
+ """Test the _find_by_column helper method."""
+ # Create a user
+ user = User(
+ username="find_by_column_test",
+ first_name="FindBy",
+ last_name="Column",
+ email="findby@column.com",
+ active=True,
+ )
+ db.session.add(user)
+ db.session.commit()
+
+ # Use the helper method directly
+ found = UserDAO._find_by_column("username", "find_by_column_test")
+ assert found is not None
+ assert found.username == "find_by_column_test"
+
+ # Test with non-existent value
+ not_found = UserDAO._find_by_column("username", "non_existent")
+ assert not_found is None
+
+
+def test_find_methods_with_special_characters(app_context: Session) -> None:
+ """Test find methods with special characters in values."""
+ # Create a dashboard with special characters in slug
+ dashboard = Dashboard(
+ dashboard_title="Test Special Chars",
+ slug="test-special_chars.v1",
+ published=True,
+ )
+ db.session.add(dashboard)
+ db.session.commit()
+
+ # Find by slug with special characters
+ found = DashboardDAO.find_by_id(
+ "test-special_chars.v1", id_column="slug", skip_base_filter=True
+ )
+ assert found is not None
+ assert found.slug == "test-special_chars.v1"
+
+ # Find by IDs with special characters
+ found_list = DashboardDAO.find_by_ids(
+ ["test-special_chars.v1"], id_column="slug", skip_base_filter=True
+ )
+ assert len(found_list) == 1
+ assert found_list[0].slug == "test-special_chars.v1"
+
+
+def test_find_methods_case_sensitivity(app_context: Session) -> None:
+ """Test find methods with case sensitivity."""
+ # Create users with similar usernames differing in case
+ user1 = User(
+ username="CaseSensitive",
+ first_name="Case",
+ last_name="Sensitive",
+ email="case1@example.com",
+ active=True,
+ )
+ user2 = User(
+ username="casesensitive",
+ first_name="Case",
+ last_name="Insensitive",
+ email="case2@example.com",
+ active=True,
+ )
+ db.session.add_all([user1, user2])
+
+ try:
+ db.session.commit()
+
+ # Find with exact case
+ found = UserDAO.find_one_or_none(username="CaseSensitive")
+ assert found is not None
+ assert found.username == "CaseSensitive"
+
+ # Find with different case
+ found_lower = UserDAO.find_one_or_none(username="casesensitive")
+ assert found_lower is not None
+ assert found_lower.username == "casesensitive"
+
+ finally:
+ pass
+
+
+def test_find_by_ids_empty_and_none_handling(app_context: Session) -> None:
+ """Test find_by_ids with empty list and None values."""
+ # Test with empty list
+ found_empty = UserDAO.find_by_ids([], skip_base_filter=True)
+ assert found_empty == []
+
+ # Test with None in list - cast to list with proper type
+ found_with_none = UserDAO.find_by_ids([None], skip_base_filter=True) # type: ignore[list-item]
+ assert found_with_none == []
+
+ # Test with mix of valid and None
+ user = User(
+ username="test_none_handling",
+ first_name="Test",
+ last_name="None",
+ email="none@example.com",
+ active=True,
+ )
+ db.session.add(user)
+ db.session.commit()
+
+ found_mixed = UserDAO.find_by_ids([user.id, None], skip_base_filter=True) # type: ignore[list-item]
+ assert len(found_mixed) == 1
+ assert found_mixed[0].id == user.id
+
+
+def test_find_methods_performance_with_large_lists(app_context: Session) -> None:
+ """Test find_by_ids performance with large lists."""
+ # Create a batch of users
+ users = []
+ for i in range(50):
+ user = User(
+ username=f"perf_test_{i}",
+ first_name=f"Perf{i}",
+ last_name="Test",
+ email=f"perf{i}@example.com",
+ active=True,
+ )
+ users.append(user)
+ db.session.add(user)
+ db.session.commit()
+
+ # Test with large list of IDs
+ user_ids = [user.id for user in users]
+
+ start_time = time.time()
+ found = UserDAO.find_by_ids(user_ids, skip_base_filter=True)
+ elapsed = time.time() - start_time
+
+ assert len(found) == 50
+ # Should complete reasonably quickly (within 1 second for 50 items)
+ assert elapsed < 1.0
+
+
+def test_base_dao_model_cls_property(app_context: Session) -> None:
+ """Test that model_cls is properly set on DAO classes."""
+ # UserDAO should have User as model_cls
+ assert UserDAO.model_cls == User
+
+ # DashboardDAO should have Dashboard as model_cls
+ assert DashboardDAO.model_cls == Dashboard
+
+ # ChartDAO should have Slice as model_cls
+ assert ChartDAO.model_cls == Slice
+
+
+def test_base_dao_list_with_relationships_pagination(app_context: Session) -> None:
+ """
+ Test that pagination works correctly when loading relationships.
+
+ This test addresses the concern that joinedload() with many-to-many
+ relationships can cause incorrect pagination due to SQL JOINs multiplying rows.
+ """
+ # Create dashboards with owners (many-to-many relationship)
+ users = []
+ for i in range(3):
+ user = User(
+ username=f"rel_test_user_{i}",
+ first_name=f"RelUser{i}",
+ last_name="Test",
+ email=f"reluser{i}@example.com",
+ active=True,
+ )
+ users.append(user)
+ db.session.add(user)
+
+ dashboards = []
+ for i in range(10):
+ dashboard = Dashboard(
+ dashboard_title=f"Relationship Test Dashboard {i}",
+ slug=f"rel-test-dash-{i}",
+ )
+ # Add multiple owners to create many-to-many relationship
+ dashboard.owners = users[:2] # Each dashboard has 2 owners
+ dashboards.append(dashboard)
+ db.session.add(dashboard)
+
+ db.session.commit()
+
+ # Test pagination without relationships - baseline
+ results_no_rel, count_no_rel = DashboardDAO.list(
+ page=0,
+ page_size=5,
+ order_column="dashboard_title",
+ order_direction="asc",
+ )
+
+ assert count_no_rel >= 10 # At least our 10 dashboards
+ assert len(results_no_rel) == 5 # Should get exactly 5 due to page_size
+
+ # Test pagination WITH relationships loaded
+ # This is the critical test - it should NOT inflate the count
+ results_with_rel, count_with_rel = DashboardDAO.list(
+ page=0,
+ page_size=5,
+ columns=[
+ "id",
+ "dashboard_title",
+ "owners",
+ ], # Include many-to-many relationship
+ order_column="dashboard_title",
+ order_direction="asc",
+ )
+
+ # CRITICAL ASSERTIONS:
+ # 1. Count should be the same regardless of joins
+ assert count_with_rel == count_no_rel, (
+ f"Count inflated by joins! Without relationships: {count_no_rel}, "
+ f"With relationships: {count_with_rel}"
+ )
+
+ # 2. Should still get exactly 5 dashboards, not affected by joins
+ assert len(results_with_rel) == 5, (
+ f"Pagination broken by joins! Expected 5 dashboards, got "
+ f"{len(results_with_rel)}"
+ )
+
+ # 3. Verify relationships are actually loaded
+ for result in results_with_rel:
+ # Check that owners relationship is loaded (would raise if not)
+ assert hasattr(result, "owners")
+ # In our test setup, each dashboard should have 2 owners
+ assert len(result.owners) == 2
+
+ # Test second page to ensure offset works correctly
+ results_page2, _ = DashboardDAO.list(
+ page=1,
+ page_size=5,
+ columns=["id", "dashboard_title", "owners"],
+ order_column="dashboard_title",
+ order_direction="asc",
+ )
+
+ assert len(results_page2) == 5 # Should get next 5 dashboards
+ # Ensure no overlap between pages
+ page1_ids = {d.id for d in results_with_rel}
+ page2_ids = {d.id for d in results_page2}
+ assert page1_ids.isdisjoint(page2_ids), "Pages should not overlap"
+
+
+def test_base_dao_list_with_one_to_many_relationship(app_context: Session) -> None:
+ """
+ Test pagination with one-to-many relationships.
+
+ Charts have a many-to-one relationship with databases.
+ When we load the database relationship, it shouldn't affect pagination.
+ """
+ # Create a database
+ database = Database(
+ database_name="TestDB for Relationships",
+ sqlalchemy_uri="sqlite:///:memory:",
+ )
+ db.session.add(database)
+ db.session.commit()
+
+ # Create charts linked to this database
+ charts = []
+ for i in range(15):
+ chart = Slice(
+ slice_name=f"Chart with Relationship {i}",
+ datasource_type="table",
+ datasource_id=1,
+ viz_type="line",
+ params="{}",
+ )
+ charts.append(chart)
+ db.session.add(chart)
+
+ db.session.commit()
+
+ # Test with relationship loading (skip_base_filter since ChartDAO may have filters)
+ # We'll use a simpler approach - directly test the BaseDAO.list method
+ from superset.daos.base import BaseDAO
+
+ class TestChartDAO(BaseDAO[Slice]):
+ model_cls = Slice
+ base_filter = None # No base filter for testing
+
+ results, total_count = TestChartDAO.list(
+ page=0,
+ page_size=10,
+ columns=["id", "slice_name", "datasource_type"],
+ order_column="slice_name",
+ order_direction="asc",
+ )
+
+ # Should get exactly 10 charts
+ assert len(results) == 10
+ # Total count should reflect all charts, not be affected by any joins
+ assert total_count >= 15
+
+
+def test_base_dao_list_count_accuracy_with_filters_and_relationships(
+ app_context: Session,
+) -> None:
+ """
+ Test that count remains accurate when combining filters with relationship loading.
+
+ This ensures the fix (counting before joins) works correctly with complex queries.
+ """
+ # Create users with specific patterns
+ active_users = []
+ inactive_users = []
+
+ for i in range(8):
+ user = User(
+ username=f"count_test_active_{i}",
+ first_name="Active",
+ last_name=f"User{i}",
+ email=f"active{i}@test.com",
+ active=True,
+ )
+ active_users.append(user)
+ db.session.add(user)
+
+ for i in range(5):
+ user = User(
+ username=f"count_test_inactive_{i}",
+ first_name="Inactive",
+ last_name=f"User{i}",
+ email=f"inactive{i}@test.com",
+ active=False,
+ )
+ inactive_users.append(user)
+ db.session.add(user)
+
+ db.session.commit()
+
+ # Create dashboards owned by these users
+ for i in range(6):
+ dashboard = Dashboard(
+ dashboard_title=f"Count Test Dashboard {i}",
+ slug=f"count-test-{i}",
+ )
+ dashboard.owners = active_users[:3] # 3 owners per dashboard
+ db.session.add(dashboard)
+
+ db.session.commit()
+
+ # Test with filters and relationship loading
+ filters = [
+ ColumnOperator(
+ col="dashboard_title",
+ opr=ColumnOperatorEnum.sw,
+ value="Count Test",
+ )
+ ]
+
+ results, count = DashboardDAO.list(
+ column_operators=filters,
+ columns=["id", "dashboard_title", "owners"], # Load many-to-many
+ page=0,
+ page_size=3,
+ )
+
+ # Should find exactly 6 dashboards that match the filter
+ assert count == 6, f"Expected 6 dashboards, but count was {count}"
+
+ # Should return only 3 due to page_size
+ assert len(results) == 3, (
+ f"Expected 3 results due to pagination, got {len(results)}"
+ )
+
+ # Each should have 3 owners as we set up
+ for dashboard in results:
+ assert len(dashboard.owners) == 3, (
+ f"Dashboard {dashboard.dashboard_title} should have 3 owners, "
+ f"has {len(dashboard.owners)}"
+ )
+
+
+def test_base_dao_id_column_name_property(app_context: Session) -> None:
+ """Test that id_column_name property works correctly."""
+ # Create a user to test with
+ user = User(
+ username="id_column_test",
+ first_name="ID",
+ last_name="Column",
+ email="id@column.com",
+ active=True,
+ )
+ db.session.add(user)
+ db.session.commit()
+
+ # UserDAO should use 'id' by default
+ assert UserDAO.id_column_name == "id"
+
+ # Find by ID should work
+ found = UserDAO.find_by_id(user.id, skip_base_filter=True)
+ assert found is not None
+ assert found.id == user.id
+
+
+def test_base_dao_base_filter_integration(app_context: Session) -> None:
+ """Test that base_filter is properly applied when set."""
+ # Create test users
+ users = []
+ for i in range(3):
+ user = User(
+ username=f"filter_test_{i}",
+ first_name=f"Filter{i}",
+ last_name="Test",
+ email=f"filter{i}@example.com",
+ active=i % 2 == 0, # Alternate active/inactive
+ )
+ users.append(user)
+ db.session.add(user)
+ db.session.commit()
+
+ # UserDAO might not have a base filter, so we just verify it doesn't break
+ all_users = UserDAO.find_all()
+ assert len(all_users) >= 3
+
+ # With skip_base_filter - find_all doesn't support this parameter
+ # Just test the regular find_all
+ all_users_normal = UserDAO.find_all()
+ assert len(all_users_normal) >= 3
+
+
+def test_base_dao_edge_cases(app_context: Session) -> None:
+ """Test BaseDAO edge cases and error conditions."""
+ # Test create without item or attributes
+ created = UserDAO.create()
+ assert created is not None
+ # User model has required fields, so we expect them to be None
+ assert created.username is None
+
+ # Don't commit - would fail due to constraints
+ db.session.rollback()
+
+ # Test update without item (creates new)
+ updated = UserDAO.update(
+ attributes={"username": "no_item_update", "email": "test@example.com"}
+ )
+ assert updated is not None
+ assert updated.username == "no_item_update"
+
+ # Don't commit - would fail due to constraints
+ db.session.rollback()
+
+ # Test list with search
+ results, total = UserDAO.list(search="test")
+ # Should handle gracefully
+ assert isinstance(results, list)
+ assert isinstance(total, int)
+
+ # Test list with column operators
+ results, total = UserDAO.list(
+ column_operators=[
+ ColumnOperator(col="username", opr=ColumnOperatorEnum.like, value="test")
+ ]
+ )
+ # Should handle gracefully
+ assert isinstance(results, list)
+ assert isinstance(total, int)
+
+
+def test_convert_value_for_column_uuid(app_context: Session) -> None:
+ """Test the _convert_value_for_column method with UUID columns."""
+ # Create a dashboard to get a real UUID column
+ dashboard = Dashboard(
+ dashboard_title="Test UUID Conversion",
+ slug="test-uuid-conversion",
+ published=True,
+ )
+ db.session.add(dashboard)
+ db.session.commit()
+
+ # Get the UUID column
+ uuid_column = Dashboard.uuid
+
+ # Test with valid UUID string
+ uuid_str = str(dashboard.uuid)
+ converted = DashboardDAO._convert_value_for_column(uuid_column, uuid_str)
+ # Should convert string to UUID object
+ assert converted == dashboard.uuid
+
+ # Test with UUID object (should return as-is)
+ converted = DashboardDAO._convert_value_for_column(uuid_column, dashboard.uuid)
+ assert converted == dashboard.uuid
+
+ # Test with invalid UUID string
+ invalid = DashboardDAO._convert_value_for_column(uuid_column, "not-a-uuid")
+ # Should return None or original value for invalid UUID
+ assert invalid == "not-a-uuid" or invalid is None
Review Comment:
<div>
<div id="suggestion">
<div id="issue"><b>Invalid UUID handling</b></div>
<div id="fix">
The test assertion expects either the original string "not-a-uuid" or None for invalid UUID conversion, but the _convert_value_for_column method consistently returns None for invalid UUID strings. The assertion should be updated to strictly expect None to match the actual implementation behavior.
</div>
<details>
<summary>
<b>Code suggestion</b>
</summary>
<blockquote>Check the AI-generated fix before applying</blockquote>
<div id="code">
```suggestion
invalid = DashboardDAO._convert_value_for_column(uuid_column, "not-a-uuid")
# Should return None for invalid UUID
assert invalid is None
```
</div>
</details>
</div>
<small><i>Code Review Run <a href=https://github.com/apache/superset/pull/35018#issuecomment-3293390307>#8dadc1</a></i></small>
</div>
---
Should Bito avoid suggestions like this for future reviews? (<a href=https://alpha.bito.ai/home/ai-agents/review-rules>Manage Rules</a>)
- [ ] Yes, avoid them
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat: Add BaseDAO improvements and test reorganization [superset]
Posted by "mistercrunch (via GitHub)" <gi...@apache.org>.
mistercrunch commented on code in PR #35018:
URL: https://github.com/apache/superset/pull/35018#discussion_r2331703702
##########
pyproject.toml:
##########
@@ -84,6 +84,7 @@ dependencies = [
"pgsanity",
"Pillow>=11.0.0, <12",
"polyline>=2.0.0, <3.0",
+ "pydantic>=2.8.0",
Review Comment:
this is sufficient here, meaning adding the dep here will affect both requirements/base.txt and requirements/development.txt automatically when re-running `./scripts/uv-pip-compile.sh`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] Feat/dao updates [superset]
Posted by "mistercrunch (via GitHub)" <gi...@apache.org>.
mistercrunch commented on code in PR #35018:
URL: https://github.com/apache/superset/pull/35018#discussion_r2326358879
##########
tests/unit_tests/dao/base_dao_test.py:
##########
@@ -37,6 +79,1924 @@ class TestDAOWithNoneModel(BaseDAO[MockModel]):
model_cls = None
+@pytest.fixture(autouse=True)
+def mock_g_user():
+ """Mock the flask g.user for security context."""
+ with patch("superset.security.manager.g") as mock_g:
+ mock_user = User()
+ mock_user.id = 1
+ mock_user.username = "test_user"
+ mock_g.user = mock_user
+ yield mock_g
+
+
+# =============================================================================
+# Column Operator Enum Tests
+# =============================================================================
+
+
+def test_column_operator_enum_apply_method() -> None: # noqa: C901
+ """
+ Test that the apply method works correctly for each operator.
+ This verifies the actual SQL generation for each operator.
+ """
+ Base_test = declarative_base() # noqa: N806
+
+ class TestModel(Base_test): # type: ignore
+ __tablename__ = "test_model"
+ id = Column(Integer, primary_key=True)
+ name = Column(String(50))
+ age = Column(Integer)
+ active = Column(Boolean)
+
+ # Test each operator's apply method
+ test_cases = [
+ # (operator, column, value, expected_sql_fragment)
+ (ColumnOperatorEnum.eq, TestModel.name, "test", "name = 'test'"),
+ (ColumnOperatorEnum.ne, TestModel.name, "test", "name != 'test'"),
+ (ColumnOperatorEnum.sw, TestModel.name, "test", "name LIKE 'test%'"),
+ (ColumnOperatorEnum.ew, TestModel.name, "test", "name LIKE '%test'"),
+ (ColumnOperatorEnum.in_, TestModel.id, [1, 2, 3], "id IN (1, 2, 3)"),
+ (ColumnOperatorEnum.nin, TestModel.id, [1, 2, 3], "id NOT IN (1, 2, 3)"),
+ (ColumnOperatorEnum.gt, TestModel.age, 25, "age > 25"),
+ (ColumnOperatorEnum.gte, TestModel.age, 25, "age >= 25"),
+ (ColumnOperatorEnum.lt, TestModel.age, 25, "age < 25"),
+ (ColumnOperatorEnum.lte, TestModel.age, 25, "age <= 25"),
+ (ColumnOperatorEnum.like, TestModel.name, "test", "name LIKE '%test%'"),
+ (ColumnOperatorEnum.ilike, TestModel.name, "test", "name ILIKE '%test%'"),
+ (ColumnOperatorEnum.is_null, TestModel.name, None, "name IS NULL"),
+ (ColumnOperatorEnum.is_not_null, TestModel.name, None, "name IS NOT NULL"),
+ ]
+
+ for operator, column, value, expected_sql_fragment in test_cases:
+ # Apply the operator
+ result = operator.apply(column, value)
+
+ # Convert to string to check SQL generation
+ sql_str = str(result.compile(compile_kwargs={"literal_binds": True}))
+
+ # Verify the SQL contains the expected fragment
+ # Note: SQLAlchemy might format this differently, so we check for key parts
+ if "=" in expected_sql_fragment:
+ assert "=" in sql_str
+ elif "!=" in expected_sql_fragment:
+ assert "!=" in sql_str
+ elif "LIKE" in expected_sql_fragment:
+ assert "LIKE" in sql_str
+ elif "ILIKE" in expected_sql_fragment:
+ assert "ILIKE" in sql_str
+ elif "IN" in expected_sql_fragment:
+ assert "IN" in sql_str
+ elif "NOT IN" in expected_sql_fragment:
+ assert "NOT IN" in sql_str
+ elif ">" in expected_sql_fragment:
+ assert ">" in sql_str
+ elif ">=" in expected_sql_fragment:
+ assert ">=" in sql_str
+ elif "<" in expected_sql_fragment:
+ assert "<" in sql_str
+ elif "<=" in expected_sql_fragment:
+ assert "<=" in sql_str
+ elif "IS NULL" in expected_sql_fragment:
+ assert "IS NULL" in sql_str
+ elif "IS NOT NULL" in expected_sql_fragment:
+ assert "IS NOT NULL" in sql_str
+
+ # Test that all operators are covered
+ all_operators = set(ColumnOperatorEnum)
+ tested_operators = {
+ ColumnOperatorEnum.eq,
+ ColumnOperatorEnum.ne,
+ ColumnOperatorEnum.sw,
+ ColumnOperatorEnum.ew,
+ ColumnOperatorEnum.in_,
+ ColumnOperatorEnum.nin,
+ ColumnOperatorEnum.gt,
+ ColumnOperatorEnum.gte,
+ ColumnOperatorEnum.lt,
+ ColumnOperatorEnum.lte,
+ ColumnOperatorEnum.like,
+ ColumnOperatorEnum.ilike,
+ ColumnOperatorEnum.is_null,
+ ColumnOperatorEnum.is_not_null,
+ }
+
+ # Ensure we've tested all operators
+ assert tested_operators == all_operators, (
+ f"Missing operators: {all_operators - tested_operators}"
+ )
+
+
+def test_column_operator_enum_complete_coverage(user_with_data: Session) -> None:
+ """
+ Test that every single ColumnOperatorEnum operator is covered by tests.
+ This ensures we have comprehensive test coverage for all operators.
+ """
+ # Create test users for comprehensive operator testing
+ test_users = [
+ User(
+ id=900,
+ username="testuser1",
+ first_name="Test",
+ last_name="User1",
+ email="test1@example.com",
+ active=True,
+ ),
+ User(
+ id=901,
+ username="testuser2",
+ first_name="Test",
+ last_name="User2",
+ email="test2@example.com",
+ active=False,
+ ),
+ User(
+ id=902,
+ username="adminuser",
+ first_name="Admin",
+ last_name="User",
+ email="admin@example.com",
+ active=True,
+ ),
+ ]
+
+ for user in test_users:
+ user_with_data.add(user)
+ user_with_data.commit()
+
+ # Test eq (equals)
+ results, _ = UserDAO.list(
+ column_operators=[
+ ColumnOperator(col="username", opr=ColumnOperatorEnum.eq, value="testuser1")
+ ]
+ )
+ assert any(u.username == "testuser1" for u in results)
+
+ # Test ne (not equals)
+ results, _ = UserDAO.list(
+ column_operators=[
+ ColumnOperator(col="username", opr=ColumnOperatorEnum.ne, value="testuser1")
+ ]
+ )
+ assert all(u.username != "testuser1" for u in results)
+
+ # Test sw (starts with)
+ results, _ = UserDAO.list(
+ column_operators=[
+ ColumnOperator(col="username", opr=ColumnOperatorEnum.sw, value="test")
+ ]
+ )
+ assert any(u.username.startswith("test") for u in results)
+
+ # Test ew (ends with)
+ results, _ = UserDAO.list(
+ column_operators=[
+ ColumnOperator(col="username", opr=ColumnOperatorEnum.ew, value="user1")
+ ]
+ )
+ assert any(u.username.endswith("user1") for u in results)
+
+ # Test in_ (in list)
+ user_ids = [900, 901]
+ results, _ = UserDAO.list(
+ column_operators=[
+ ColumnOperator(col="id", opr=ColumnOperatorEnum.in_, value=user_ids)
+ ]
+ )
+ result_ids = [u.id for u in results]
+ assert all(uid in result_ids for uid in user_ids)
+
+ # Test nin (not in list)
+ results, _ = UserDAO.list(
+ column_operators=[
+ ColumnOperator(col="id", opr=ColumnOperatorEnum.nin, value=user_ids)
+ ]
+ )
+ result_ids = [u.id for u in results]
+ assert all(uid not in result_ids for uid in user_ids)
+
+ # Test gt (greater than)
+ results, _ = UserDAO.list(
+ column_operators=[
+ ColumnOperator(col="id", opr=ColumnOperatorEnum.gt, value=900)
+ ]
+ )
+ assert all(u.id > 900 for u in results)
+
+ # Test gte (greater than or equal)
+ results, _ = UserDAO.list(
+ column_operators=[
+ ColumnOperator(col="id", opr=ColumnOperatorEnum.gte, value=901)
+ ]
+ )
+ assert all(u.id >= 901 for u in results)
+
+ # Test lt (less than)
+ results, _ = UserDAO.list(
+ column_operators=[
+ ColumnOperator(col="id", opr=ColumnOperatorEnum.lt, value=902)
+ ]
+ )
+ assert all(u.id < 902 for u in results)
+
+ # Test lte (less than or equal)
+ results, _ = UserDAO.list(
+ column_operators=[
+ ColumnOperator(col="id", opr=ColumnOperatorEnum.lte, value=901)
+ ]
+ )
+ assert all(u.id <= 901 for u in results)
+
+ # Test like (like pattern)
+ results, _ = UserDAO.list(
+ column_operators=[
+ ColumnOperator(col="username", opr=ColumnOperatorEnum.like, value="test")
+ ]
+ )
+ assert any("test" in u.username for u in results)
+
+ # Test ilike (case insensitive like)
+ results, _ = UserDAO.list(
+ column_operators=[
+ ColumnOperator(col="username", opr=ColumnOperatorEnum.ilike, value="TEST")
+ ]
+ )
+ assert any("test" in u.username.lower() for u in results)
+
+ # Test is_null (is null)
+ # Create a user with a null field that allows nulls
+ user_with_data.add(
+ User(
+ id=903,
+ username="nulluser",
+ first_name="Test",
+ last_name="User",
+ email="nulluser@example.com",
+ active=True,
+ )
+ )
+ user_with_data.commit()
+
+ # Update to set a field that can be null (we'll use a field that allows nulls)
+ # For this test, we'll check if we can find users where a field is null
+ # Since we can't easily set first_name to null, we'll test the is_null operator
+ # by checking if it works with fields that might be null
+ results, _ = UserDAO.list(
+ column_operators=[
+ ColumnOperator(col="first_name", opr=ColumnOperatorEnum.is_null, value=None)
+ ]
+ )
+ # This should return empty since all users have first_name set
+ assert len(results) == 0
+
+ # Test is_not_null (is not null)
+ results, _ = UserDAO.list(
+ column_operators=[
+ ColumnOperator(
+ col="first_name", opr=ColumnOperatorEnum.is_not_null, value=None
+ )
+ ]
+ )
+ # This should return all users since all have first_name set
+ assert len(results) > 0
+
+ # Test boolean operators
+ results, _ = UserDAO.list(
+ column_operators=[
+ ColumnOperator(col="active", opr=ColumnOperatorEnum.eq, value=True)
+ ]
+ )
+ assert all(u.active for u in results)
+
+ results, _ = UserDAO.list(
+ column_operators=[
+ ColumnOperator(col="active", opr=ColumnOperatorEnum.eq, value=False)
+ ]
+ )
+ assert all(not u.active for u in results)
+
+ # Test that all operators are covered
+ all_operators = set(ColumnOperatorEnum)
+ tested_operators = {
+ ColumnOperatorEnum.eq,
+ ColumnOperatorEnum.ne,
+ ColumnOperatorEnum.sw,
+ ColumnOperatorEnum.ew,
+ ColumnOperatorEnum.in_,
+ ColumnOperatorEnum.nin,
+ ColumnOperatorEnum.gt,
+ ColumnOperatorEnum.gte,
+ ColumnOperatorEnum.lt,
+ ColumnOperatorEnum.lte,
+ ColumnOperatorEnum.like,
+ ColumnOperatorEnum.ilike,
+ ColumnOperatorEnum.is_null,
+ ColumnOperatorEnum.is_not_null,
+ }
+
+ # Ensure we've tested all operators
+ assert tested_operators == all_operators, (
+ f"Missing operators: {all_operators - tested_operators}"
+ )
+
+
+# =============================================================================
+# Find by ID Tests - Testing flexible column support
+# =============================================================================
+
+
+def test_find_by_id_with_default_column(app_context: None) -> None:
+ """Test find_by_id using the default id column."""
+ # Create test data
+ user = User(
+ id=1000,
+ username="test_find_user",
+ first_name="Test",
+ last_name="User",
+ email="testfind@example.com",
+ active=True,
+ )
+ db.session.add(user)
+ db.session.commit()
+
+ # Find by ID using default column
+ found_user = UserDAO.find_by_id(1000)
+ assert found_user is not None
+ assert found_user.id == 1000
+ assert found_user.username == "test_find_user"
+
+ # Test not found
+ not_found = UserDAO.find_by_id(9999)
+ assert not_found is None
+
+ # Cleanup
+ db.session.delete(user)
+ db.session.commit()
+
+
+def test_find_by_id_with_uuid_column(app_context: None) -> None:
+ """Test find_by_id using UUID column."""
+ # Create dashboard with UUID
+ dashboard = Dashboard(
+ dashboard_title="Test UUID Dashboard",
+ slug="test-uuid-dashboard",
+ published=True,
+ )
+ db.session.add(dashboard)
+ db.session.commit()
+
+ # Verify dashboard exists (skip base filter for test)
+ found_by_id = DashboardDAO.find_by_id(dashboard.id, skip_base_filter=True)
+ assert found_by_id is not None
+
+ # Get the UUID as string for lookup
+ dashboard_uuid_str = str(dashboard.uuid)
+
+ # Find by UUID (skip base filter)
+ found_dashboard = DashboardDAO.find_by_id(
+ dashboard_uuid_str, id_column="uuid", skip_base_filter=True
+ )
+ assert found_dashboard is not None
+ assert str(found_dashboard.uuid) == dashboard_uuid_str
+ assert found_dashboard.dashboard_title == "Test UUID Dashboard"
+
+ # Test not found with invalid UUID
+ not_found = DashboardDAO.find_by_id("invalid-uuid", id_column="uuid")
+ assert not_found is None
+
+ # Cleanup
+ db.session.delete(dashboard)
+ db.session.commit()
+
+
+def test_find_by_id_with_slug_column(app_context: None) -> None:
+ """Test find_by_id using slug column."""
+ # Create test dashboard with slug
+ dashboard = Dashboard(
+ dashboard_title="Test Slug Dashboard",
+ slug="unique-test-slug",
+ published=True,
+ )
+ db.session.add(dashboard)
+ db.session.commit()
+
+ # Find by slug (skip base filter)
+ found_dashboard = DashboardDAO.find_by_id(
+ "unique-test-slug", id_column="slug", skip_base_filter=True
+ )
+ assert found_dashboard is not None
+ assert found_dashboard.slug == "unique-test-slug"
+ assert found_dashboard.dashboard_title == "Test Slug Dashboard"
+
+ # Test not found
+ not_found = DashboardDAO.find_by_id(
+ "non-existent-slug", id_column="slug", skip_base_filter=True
+ )
+ assert not_found is None
+
+ # Cleanup
+ db.session.delete(dashboard)
+ db.session.commit()
+
+
+def test_find_by_id_with_invalid_column(app_context: None) -> None:
+ """Test find_by_id with invalid column name."""
+ # Should return None when column doesn't exist
+ result = UserDAO.find_by_id(1, id_column="non_existent_column")
+ assert result is None
+
+
+def test_find_by_id_skip_base_filter(app_context: None) -> None:
+ """Test find_by_id with skip_base_filter option."""
+ # Create two users - one active, one inactive
+ active_user = User(
+ id=1001,
+ username="active_user",
+ first_name="Active",
+ last_name="User",
+ email="active@example.com",
+ active=True,
+ )
+ inactive_user = User(
+ id=1002,
+ username="inactive_user",
+ first_name="Inactive",
+ last_name="User",
+ email="inactive@example.com",
+ active=False,
+ )
+ db.session.add_all([active_user, inactive_user])
+ db.session.commit()
+
+ # Test basic finding without base filter
+ found_active = UserDAO.find_by_id(1001)
+ assert found_active is not None
+ assert found_active.id == 1001
+
+ found_inactive = UserDAO.find_by_id(1002)
+ assert found_inactive is not None
+ assert found_inactive.id == 1002
+
+ # Test skip_base_filter parameter works (should have same result for UserDAO)
+ found_with_skip = UserDAO.find_by_id(1001, skip_base_filter=True)
+ assert found_with_skip is not None
+ assert found_with_skip.id == 1001
+
+ # Cleanup
+ db.session.delete(active_user)
+ db.session.delete(inactive_user)
+ db.session.commit()
+
+
+# =============================================================================
+# Find by IDs Tests - Testing bulk operations with flexible columns
+# =============================================================================
+
+
+def test_find_by_ids_with_default_column(app_context: None) -> None:
+ """Test find_by_ids using the default id column."""
+ # Create multiple test users
+ users = []
+ for i in range(1100, 1105):
+ user = User(
+ id=i,
+ username=f"bulk_user_{i}",
+ first_name="Bulk",
+ last_name=f"User{i}",
+ email=f"bulk{i}@example.com",
+ active=True,
+ )
+ users.append(user)
+ db.session.add(user)
+ db.session.commit()
+
+ # Find multiple by IDs
+ ids_to_find = [1100, 1102, 1104]
+ found_users = UserDAO.find_by_ids(ids_to_find)
+ assert len(found_users) == 3
+ found_ids = [u.id for u in found_users]
+ assert set(found_ids) == set(ids_to_find)
+
+ # Test with some non-existent IDs
+ mixed_ids = [1101, 1103, 9999, 8888]
+ found_users = UserDAO.find_by_ids(mixed_ids)
+ assert len(found_users) == 2
+ found_ids = [u.id for u in found_users]
+ assert set(found_ids) == {1101, 1103}
+
+ # Test empty list
+ found_users = UserDAO.find_by_ids([])
+ assert found_users == []
+
+ # Cleanup
+ for user in users:
+ db.session.delete(user)
+ db.session.commit()
+
+
+def test_find_by_ids_with_uuid_column(app_context: None) -> None:
+ """Test find_by_ids using UUID column."""
+ # Create multiple charts with UUIDs
+ charts = []
+ chart_uuids = []
+ for i in range(3):
+ chart_uuid = str(uuid.uuid4())
+ chart_uuids.append(chart_uuid)
+ chart = Slice(
+ slice_name=f"Test Chart {i}",
+ viz_type="table",
+ datasource_id=1,
+ datasource_type="table",
+ params="{}",
+ uuid=chart_uuid,
+ )
+ charts.append(chart)
+ db.session.add(chart)
+ db.session.commit()
+
+ # Find multiple by UUIDs (skip base filter)
+ found_charts = ChartDAO.find_by_ids(
+ chart_uuids[:2], id_column="uuid", skip_base_filter=True
+ )
+ assert len(found_charts) == 2
+ found_uuids = [str(c.uuid) for c in found_charts]
+ assert set(found_uuids) == set(chart_uuids[:2])
+
+ # Test with mixed valid and invalid UUIDs (skip base filter)
+ mixed_uuids = [chart_uuids[0], "invalid-uuid", chart_uuids[2]]
+ found_charts = ChartDAO.find_by_ids(
+ mixed_uuids, id_column="uuid", skip_base_filter=True
+ )
+ assert len(found_charts) == 2
+ found_uuids = [str(c.uuid) for c in found_charts]
+ assert set(found_uuids) == {chart_uuids[0], chart_uuids[2]}
+
+ # Cleanup
+ for chart in charts:
+ db.session.delete(chart)
+ db.session.commit()
+
+
+def test_find_by_ids_with_slug_column(app_context: None) -> None:
+ """Test find_by_ids using slug column."""
+ # Create multiple dashboards with slugs
+ dashboards = []
+ slugs = []
+ for i in range(3):
+ slug = f"test-dashboard-{i}"
+ slugs.append(slug)
+ dashboard = Dashboard(
+ dashboard_title=f"Test Dashboard {i}",
+ slug=slug,
+ published=True,
+ )
+ dashboards.append(dashboard)
+ db.session.add(dashboard)
+ db.session.commit()
+
+ # Find multiple by slugs (skip base filter)
+ found_dashboards = DashboardDAO.find_by_ids(
+ slugs[:2], id_column="slug", skip_base_filter=True
+ )
+ assert len(found_dashboards) == 2
+ found_slugs = [d.slug for d in found_dashboards]
+ assert set(found_slugs) == set(slugs[:2])
+
+ # Test with non-existent slugs
+ mixed_slugs = [slugs[0], "non-existent-slug", slugs[2]]
+ found_dashboards = DashboardDAO.find_by_ids(
+ mixed_slugs, id_column="slug", skip_base_filter=True
+ )
+ assert len(found_dashboards) == 2
+ found_slugs = [d.slug for d in found_dashboards]
+ assert set(found_slugs) == {slugs[0], slugs[2]}
+
+ # Cleanup
+ for dashboard in dashboards:
+ db.session.delete(dashboard)
+ db.session.commit()
+
+
+def test_find_by_ids_with_invalid_column(app_context: None) -> None:
+ """Test find_by_ids with invalid column name."""
+ # Should return empty list when column doesn't exist
+ result = UserDAO.find_by_ids([1, 2, 3], id_column="non_existent_column")
+ assert result == []
+
+
+def test_find_by_ids_skip_base_filter(app_context: None) -> None:
+ """Test find_by_ids with skip_base_filter option."""
+ # Create mix of active and inactive users
+ users = []
+ user_ids = []
+ for i, active in enumerate([True, False, True, False]):
+ user_id = 1200 + i
+ user_ids.append(user_id)
+ user = User(
+ id=user_id,
+ username=f"filter_user_{user_id}",
+ first_name="Filter",
+ last_name=f"User{user_id}",
+ email=f"filter{user_id}@example.com",
+ active=active,
+ )
+ users.append(user)
+ db.session.add(user)
+ db.session.commit()
+
+ # Test finding all users
+ found_users = UserDAO.find_by_ids(user_ids)
+ assert len(found_users) == 4
+ found_ids = [u.id for u in found_users]
+ assert set(found_ids) == set(user_ids)
+
+ # Test skip_base_filter parameter works (should have same result for UserDAO)
+ found_users_skip = UserDAO.find_by_ids(user_ids, skip_base_filter=True)
+ assert len(found_users_skip) == 4
+ found_ids_skip = [u.id for u in found_users_skip]
+ assert set(found_ids_skip) == set(user_ids)
+
+ # Cleanup
+ for user in users:
+ db.session.delete(user)
+ db.session.commit()
+
+
+# =============================================================================
+# CRUD Operations Tests - Create, Read, Update, Delete
+# =============================================================================
+
+
+def test_base_dao_create_with_item(app_context: None) -> None:
+ """Test BaseDAO create method with an item."""
+ # Create a new user
+ user = User(
+ id=5000,
+ username="test_create_user",
+ first_name="Test",
+ last_name="Create",
+ email="testcreate@example.com",
+ active=True,
+ )
+ created = UserDAO.create(item=user)
+
+ # Verify it's added to session
+ assert created is user
+ assert created in db.session
+
+ # Commit and verify it persists
+ db.session.commit()
+ assert created.id is not None
+
+ # Verify we can find it
+ found = UserDAO.find_by_id(created.id)
+ assert found is not None
+ assert found.username == "test_create_user"
+ assert found.first_name == "Test"
+
+ # Cleanup
+ db.session.delete(created)
+ db.session.commit()
+
+
+def test_base_dao_create_with_attributes(app_context: None) -> None:
+ """Test BaseDAO create method with attributes dict."""
+ # Create with attributes
+ attributes = {
+ "id": 5001,
+ "username": "attr_user",
+ "first_name": "Attr",
+ "last_name": "User",
+ "email": "attr@example.com",
+ "active": True,
+ }
+ created = UserDAO.create(attributes=attributes)
+
+ # Verify attributes were set
+ assert created.username == "attr_user"
+ assert created.first_name == "Attr"
+ assert created in db.session
+
+ # Commit and verify
+ db.session.commit()
+ assert created.id is not None
+
+ # Cleanup
+ db.session.delete(created)
+ db.session.commit()
+
+
+def test_base_dao_create_with_both_item_and_attributes(app_context: None) -> None:
+ """Test BaseDAO create with both item and attributes."""
+ # Create user with some values
+ user = User(
+ id=5002,
+ username="initial_user",
+ first_name="Initial",
+ last_name="User",
+ email="initial@example.com",
+ active=True,
+ )
+
+ # Override with attributes
+ attributes = {
+ "username": "updated_user",
+ "first_name": "Updated",
+ }
+ created = UserDAO.create(item=user, attributes=attributes)
+
+ # Attributes should override item values
+ assert created.username == "updated_user"
+ assert created.first_name == "Updated"
+ assert created.last_name == "User" # Should keep original
+
+ db.session.commit()
+
+ # Cleanup
+ db.session.delete(created)
+ db.session.commit()
+
+
+def test_base_dao_update_with_item(app_context: None) -> None:
+ """Test BaseDAO update method with an item."""
+ # Create a user first
+ user = User(
+ id=5003,
+ username="original_user",
+ first_name="Original",
+ last_name="User",
+ email="original@example.com",
+ active=True,
+ )
+ db.session.add(user)
+ db.session.commit()
+
+ # Update it
+ user.first_name = "Updated"
+ updated = UserDAO.update(item=user)
+
+ assert updated is user
+ assert updated.first_name == "Updated"
+
+ db.session.commit()
+
+ # Verify update persisted
+ found = UserDAO.find_by_id(user.id)
+ assert found is not None
+ assert found.first_name == "Updated"
+
+ # Cleanup
+ db.session.delete(user)
+ db.session.commit()
+
+
+def test_base_dao_update_with_attributes(app_context: None) -> None:
+ """Test BaseDAO update method with attributes."""
+ # Create a user first
+ user = User(
+ id=5004,
+ username="attr_update_user",
+ first_name="Original",
+ last_name="User",
+ email="attrupdate@example.com",
+ active=True,
+ )
+ db.session.add(user)
+ db.session.commit()
+
+ # Update with attributes
+ attributes = {
+ "first_name": "Attr Updated",
+ "last_name": "Attr User",
+ }
+ updated = UserDAO.update(item=user, attributes=attributes)
+
+ assert updated.first_name == "Attr Updated"
+ assert updated.last_name == "Attr User"
+
+ db.session.commit()
+
+ # Cleanup
+ db.session.delete(user)
+ db.session.commit()
+
+
+def test_base_dao_update_detached_item(app_context: None) -> None:
+ """Test BaseDAO update with a detached item."""
+ # Create and commit a user
+ user = User(
+ id=5005,
+ username="detached_user",
+ first_name="Detached",
+ last_name="User",
+ email="detached@example.com",
+ active=True,
+ )
+ db.session.add(user)
+ db.session.commit()
+ user_id = user.id
+
+ # Detach it from session
+ db.session.expunge(user)
+
+ # Update the detached user
+ user.first_name = "Updated Detached"
+ updated = UserDAO.update(item=user)
+
+ # Should merge it back into session
+ assert updated in db.session
+ assert updated.first_name == "Updated Detached"
+
+ db.session.commit()
+
+ # Verify update
+ found = UserDAO.find_by_id(user_id)
+ assert found is not None
+ assert found.first_name == "Updated Detached"
+
+ # Cleanup
+ db.session.delete(found)
+ db.session.commit()
+
+
+def test_base_dao_delete_single_item(app_context: None) -> None:
+ """Test BaseDAO delete method with single item."""
+ # Create a user
+ user = User(
+ id=5006,
+ username="delete_user",
+ first_name="Delete",
+ last_name="User",
+ email="delete@example.com",
+ active=True,
+ )
+ db.session.add(user)
+ db.session.commit()
+ user_id = user.id
+
+ # Delete it
+ UserDAO.delete([user])
+ db.session.commit()
+
+ # Verify it's gone
+ found = UserDAO.find_by_id(user_id)
+ assert found is None
+
+
+def test_base_dao_delete_multiple_items(app_context: None) -> None:
+ """Test BaseDAO delete method with multiple items."""
+ # Create multiple users
+ users = []
+ for i in range(3):
+ user = User(
+ id=5007 + i,
+ username=f"delete_user_{i}",
+ first_name="Delete",
+ last_name=f"User{i}",
+ email=f"delete{i}@example.com",
+ active=True,
+ )
+ users.append(user)
+ db.session.add(user)
+ db.session.commit()
+
+ user_ids = [user.id for user in users]
+
+ # Delete all at once
+ UserDAO.delete(users)
+ db.session.commit()
+
+ # Verify all are gone
+ for user_id in user_ids:
+ found = UserDAO.find_by_id(user_id)
+ assert found is None
+
+
+def test_base_dao_delete_empty_list(app_context: None) -> None:
+ """Test BaseDAO delete with empty list."""
+ # Should not raise any errors
+ UserDAO.delete([])
+ db.session.commit()
+
+
+def test_base_dao_find_all(app_context: None) -> None:
+ """Test BaseDAO find_all method."""
+ # Create test users
+ users = []
+ for i in range(3):
+ user = User(
+ id=5010 + i,
+ username=f"find_all_user_{i}",
+ first_name="FindAll",
+ last_name=f"User{i}",
+ email=f"findall{i}@example.com",
+ active=True,
+ )
+ users.append(user)
+ db.session.add(user)
+ db.session.commit()
+
+ # Find all
+ all_users = UserDAO.find_all()
+
+ # Should include our created users
+ usernames = [user.username for user in all_users]
+ for i in range(3):
+ assert f"find_all_user_{i}" in usernames
+
+ # Cleanup
+ for user in users:
+ db.session.delete(user)
+ db.session.commit()
+
+
+def test_base_dao_find_one_or_none(app_context: None) -> None:
+ """Test BaseDAO find_one_or_none method."""
+ # Create test users
+ user1 = User(
+ id=5013,
+ username="unique_user",
+ first_name="Unique",
+ last_name="User",
+ email="unique@example.com",
+ active=True,
+ )
+ user2 = User(
+ id=5014,
+ username="duplicate_user_1",
+ first_name="Duplicate",
+ last_name="First",
+ email="dup1@example.com",
+ active=True,
+ )
+ user3 = User(
+ id=5015,
+ username="duplicate_user_2",
+ first_name="Duplicate",
+ last_name="Second",
+ email="dup2@example.com",
+ active=True,
+ )
+ db.session.add_all([user1, user2, user3])
+ db.session.commit()
+
+ # Find unique user
+ found = UserDAO.find_one_or_none(username="unique_user")
+ assert found is not None
+ assert found.first_name == "Unique"
+
+ # Find non-existent user
+ not_found = UserDAO.find_one_or_none(username="does_not_exist")
+ assert not_found is None
+
+ # Cleanup
+ db.session.delete(user1)
+ db.session.delete(user2)
+ db.session.delete(user3)
+ db.session.commit()
+
+
+# =============================================================================
+# List and Filtering Tests
+# =============================================================================
+
+
+def test_base_dao_list_returns_results(user_with_data: Session) -> None:
+ results, total = UserDAO.list()
+ assert total >= 1
+ assert any(u.username == "testuser" for u in results)
+
+
+def test_base_dao_list_with_column_operators(user_with_data: Session) -> None:
+ results, total = UserDAO.list(
+ column_operators=[
+ ColumnOperator(col="username", opr=ColumnOperatorEnum.eq, value="testuser")
+ ]
+ )
+ assert total >= 1
+ assert all(u.username == "testuser" for u in results)
+
+
+def test_base_dao_list_with_non_matching_column_operator(
+ user_with_data: Session,
+) -> None:
+ results, total = UserDAO.list(
+ column_operators=[
+ ColumnOperator(
+ col="username", opr=ColumnOperatorEnum.eq, value="doesnotexist"
+ )
+ ]
+ )
+ assert total == 0
+ assert results == []
+
+
+def test_base_dao_count_returns_value(user_with_data: Session) -> None:
+ count = UserDAO.count()
+ assert count >= 1
+
+
+def test_base_dao_count_with_column_operators(user_with_data: Session) -> None:
+ count = UserDAO.count(
+ column_operators=[
+ ColumnOperator(col="username", opr=ColumnOperatorEnum.eq, value="testuser")
+ ]
+ )
+ assert count >= 1
+ count = UserDAO.count(
+ column_operators=[
+ ColumnOperator(
+ col="username", opr=ColumnOperatorEnum.eq, value="doesnotexist"
+ )
+ ]
+ )
+ assert count == 0
+
+
+def test_base_dao_list_and_count_skip_base_filter(user_with_data: Session) -> None:
+ results, total = UserDAO.list()
+ results_skip, total_skip = UserDAO.list()
+ assert total == total_skip
+ count = UserDAO.count()
+ count_skip = UserDAO.count(skip_base_filter=True)
+ assert count_skip == count
+
+
+def test_base_dao_list_ordering(user_with_data: Session) -> None:
+ user_with_data.add_all(
+ [
+ User(
+ id=201,
+ username="buser",
+ first_name="B",
+ last_name="User",
+ email="buser@example.com",
+ active=True,
+ ),
+ User(
+ id=202,
+ username="auser",
+ first_name="A",
+ last_name="User",
+ email="auser@example.com",
+ active=True,
+ ),
+ User(
+ id=203,
+ username="zuser",
+ first_name="Z",
+ last_name="User",
+ email="zuser@example.com",
+ active=True,
+ ),
+ ]
+ )
+ user_with_data.commit()
+ results, _ = UserDAO.list(order_column="username", order_direction="asc")
+ usernames = [u.username for u in results]
+ assert usernames == sorted(usernames)
+ results, _ = UserDAO.list(order_column="id", order_direction="desc")
+ ids = [u.id for u in results]
+ assert ids == sorted(ids, reverse=True)
+
+
+def test_base_dao_list_paging(user_with_data: Session) -> None:
+ users = [
+ User(
+ id=300 + i,
+ username=f"user{i}",
+ first_name=f"F{i}",
+ last_name="User",
+ email=f"user{i}@example.com",
+ active=True,
+ )
+ for i in range(10)
+ ]
+ user_with_data.add_all(users)
+ user_with_data.commit()
+ results, total = UserDAO.list(
+ order_column="id", order_direction="asc", page=0, page_size=5
+ )
+ ids = [u.id for u in results]
+ all_results, _ = UserDAO.list(order_column="id", order_direction="asc")
+ all_ids = [u.id for u in all_results]
+ assert ids == all_ids[:5]
+ assert total == len(all_ids)
+ results, total = UserDAO.list(
+ order_column="id", order_direction="asc", page=1, page_size=5
+ )
+ ids = [u.id for u in results]
+ assert ids == all_ids[5:10]
+ results, total = UserDAO.list(
+ order_column="id", order_direction="asc", page=10, page_size=5
+ )
+ assert results == []
+
+
+def test_base_dao_list_search(user_with_data: Session) -> None:
+ user_with_data.add_all(
+ [
+ User(
+ id=400,
+ username="searchuser1",
+ first_name="Alice",
+ last_name="Wonderland",
+ email="alice@example.com",
+ active=True,
+ ),
+ User(
+ id=401,
+ username="searchuser2",
+ first_name="Bob",
+ last_name="Builder",
+ email="bob@example.com",
+ active=True,
+ ),
+ User(
+ id=402,
+ username="searchuser3",
+ first_name="Charlie",
+ last_name="Chocolate",
+ email="charlie@example.com",
+ active=True,
+ ),
+ ]
+ )
+ user_with_data.commit()
+ results, _ = UserDAO.list(search="Alice", search_columns=["first_name"])
+ assert any(u.first_name == "Alice" for u in results)
+ results, _ = UserDAO.list(search="Builder", search_columns=["last_name"])
+ assert any(u.last_name == "Builder" for u in results)
+ results, _ = UserDAO.list(search="ar", search_columns=["first_name", "last_name"])
+ names = [(u.first_name, u.last_name) for u in results]
+ assert ("Charlie", "Chocolate") in names or ("Bob", "Builder") in names
+
+
+def test_base_dao_list_custom_filter(user_with_data: Session) -> None:
+ datamodel = SQLAInterface(User, user_with_data)
+
+ class EmailDomainFilter(BaseFilter):
+ def apply(self, query, value):
+ return query.filter(User.email.like(f"%@{value}"))
+
+ user_with_data.add_all(
+ [
+ User(
+ id=500,
+ username="customuser1",
+ first_name="Dom",
+ last_name="Ain",
+ email="dom@domain.com",
+ active=True,
+ ),
+ User(
+ id=501,
+ username="customuser2",
+ first_name="Jane",
+ last_name="Doe",
+ email="jane@other.com",
+ active=True,
+ ),
+ ]
+ )
+ user_with_data.commit()
+ results, _ = UserDAO.list(
+ column_operators=None,
+ custom_filters={"email_domain": EmailDomainFilter("email_domain", datamodel)},
+ )
+ assert all(u.email.endswith("@domain.com") for u in results)
+
+
+def test_base_dao_list_base_filter(user_with_data: Session) -> None:
+ class OnlyActiveFilter(BaseFilter):
+ def apply(self, query, value):
+ return query.filter(User.active)
+
+ class ActiveUserDAO(UserDAO):
+ base_filter = OnlyActiveFilter
+
+ user_with_data.add_all(
+ [
+ User(
+ id=600,
+ username="inactiveuser",
+ first_name="Inactive",
+ last_name="User",
+ email="inactive@example.com",
+ active=False,
+ ),
+ User(
+ id=601,
+ username="activeuser",
+ first_name="Active",
+ last_name="User",
+ email="active@example.com",
+ active=True,
+ ),
+ ]
+ )
+ user_with_data.commit()
+ results, _ = ActiveUserDAO.list()
+ assert all(u.active for u in results)
+
+
+def test_base_dao_list_edge_cases(user_with_data: Session) -> None:
+ user_with_data.add_all(
+ [
+ User(
+ id=700,
+ username="booluser1",
+ first_name="Bool",
+ last_name="User",
+ email="bool1@example.com",
+ active=True,
+ ),
+ User(
+ id=701,
+ username="booluser2",
+ first_name="Bool",
+ last_name="User",
+ email="bool2@example.com",
+ active=False,
+ ),
+ ]
+ )
+ user_with_data.commit()
+ results, _ = UserDAO.list(
+ column_operators=[
+ ColumnOperator(col="active", opr=ColumnOperatorEnum.eq, value=True)
+ ]
+ )
+ assert all(u.active for u in results)
+ results, _ = UserDAO.list(
+ column_operators=[
+ ColumnOperator(col="active", opr=ColumnOperatorEnum.eq, value=False)
+ ]
+ )
+ assert all(not u.active for u in results)
+ results, _ = UserDAO.list(
+ column_operators=[
+ ColumnOperator(col="first_name", opr=ColumnOperatorEnum.eq, value=None)
+ ]
+ )
+ ids = [700, 701]
+ results, _ = UserDAO.list(
+ column_operators=[
+ ColumnOperator(col="id", opr=ColumnOperatorEnum.in_, value=ids)
+ ]
+ )
+ result_ids = [u.id for u in results]
+ assert set(result_ids) == set(ids)
+ results, _ = UserDAO.list(page=100, page_size=10)
+ assert results == []
+
+
+def test_base_dao_list_with_select_columns(user_with_data: Session) -> None:
+ # Add a user to ensure at least one exists
+ user_with_data.add(
+ User(
+ id=900,
+ username="coluser",
+ first_name="Col",
+ last_name="User",
+ email="coluser@example.com",
+ active=True,
+ )
+ )
+ user_with_data.commit()
+ # Request only username and email columns
+ results, total = UserDAO.list(columns=["username", "email"])
+ assert total >= 1
+ # Should return Row objects with correct values
+ found = False
+ for row in results:
+ # SQLAlchemy Row supports both index and key access
+ if row[0] == "coluser" and row[1] == "coluser@example.com":
+ found = True
+ assert found
+ # Request only id column
+ results, total = UserDAO.list(columns=["id"])
+ found = False
+ for row in results:
+ if row[0] == 900:
+ found = True
+ assert found
+
+
+def test_base_dao_list_with_default_columns(user_with_data: Session) -> None:
+ user_with_data.add(
+ User(
+ id=901,
+ username="defaultuser",
+ first_name="Default",
+ last_name="User",
+ email="defaultuser@example.com",
+ active=True,
+ )
+ )
+ user_with_data.commit()
+ results, total = UserDAO.list()
+ assert total >= 1
+ # Should return model instances
+ found = False
+ for user in results:
+ if hasattr(user, "id") and hasattr(user, "username") and hasattr(user, "email"):
+ if (
+ user.id == 901
+ and user.username == "defaultuser"
+ and user.email == "defaultuser@example.com"
+ ):
+ found = True
+ assert found
+
+
+# =============================================================================
+# Advanced Features Tests - Filtering, Operators, Performance
+# =============================================================================
+
+
+def test_base_dao_apply_column_operators_invalid_column(app_context: None) -> None:
+ """Test apply_column_operators with invalid column."""
+ # Create a query
+ query = db.session.query(User)
+
+ # Apply operator with invalid column
+ with patch("superset.daos.base.logging.error") as mock_log:
+ column_operators = [
+ ColumnOperator(
+ col="non_existent_column", opr=ColumnOperatorEnum.eq, value="test"
+ )
+ ]
+
+ # Should raise ValueError
+ with pytest.raises(ValueError, match="Invalid filter") as exc_info:
+ UserDAO.apply_column_operators(query, column_operators)
+
+ assert "Invalid filter" in str(exc_info.value)
+ mock_log.assert_called_once()
+
+
+def test_base_dao_apply_column_operators_none_value(app_context: None) -> None:
+ """Test apply_column_operators with None col value."""
+ query = db.session.query(User)
+
+ # Operator with None column name
+ column_operators = [
+ ColumnOperator(
+ col="", # Empty string instead of None
+ opr=ColumnOperatorEnum.eq,
+ value="test",
+ )
+ ]
+
+ # Should raise ValueError
+ with pytest.raises(ValueError, match="Invalid filter") as exc_info:
+ UserDAO.apply_column_operators(query, column_operators)
+
+ assert "Invalid filter" in str(exc_info.value)
+
+
+def test_base_dao_list_with_invalid_operator(app_context: None) -> None:
+ """Test list method with invalid column operator type."""
+ # Pass a non-ColumnOperator object
+ invalid_operators = [{"col": "username", "opr": "eq", "value": "test"}]
+
+ # Should handle gracefully (skip invalid operators)
+ results, total = UserDAO.list(column_operators=invalid_operators) # type: ignore
+
+ # Should return results without applying invalid filter
+ assert isinstance(results, list)
+ assert isinstance(total, int)
+
+
+def test_base_dao_get_filterable_columns(app_context: None) -> None:
+ """Test get_filterable_columns_and_operators method."""
+ # This is typically implemented by FAB, but we can test the interface
+ filterable = UserDAO.get_filterable_columns_and_operators()
+
+ # Should return a dict
+ assert isinstance(filterable, dict)
+
+ # Common user columns should be filterable
+ if filterable: # Only test if FAB integration is available
+ common_columns = ["username", "email", "active", "first_name", "last_name"]
+ for col in common_columns:
+ if col in filterable:
+ # Should have operator information
+ assert isinstance(filterable[col], (list, tuple, set))
+
+
+def test_base_dao_count_with_filters(app_context: None) -> None:
+ """Test count method with various filters."""
+ # Create test users
+ users = []
+ for i in range(5):
+ user = User(
+ id=2000 + i,
+ username=f"count_user_{i}",
+ first_name="Count",
+ last_name=f"User{i}",
+ email=f"count{i}@example.com",
+ active=(i % 2 == 0), # Alternate active/inactive
+ )
+ users.append(user)
+ db.session.add(user)
+ db.session.commit()
+
+ # Count all
+ total_count = UserDAO.count()
+ assert total_count >= 5
+
+ # Count with filter
+ active_count = UserDAO.count(
+ column_operators=[
+ ColumnOperator(col="active", opr=ColumnOperatorEnum.eq, value=True)
+ ]
+ )
+ assert active_count >= 3 # We created 3 active users (0, 2, 4)
+
+ # Count with multiple filters
+ specific_count = UserDAO.count(
+ column_operators=[
+ ColumnOperator(col="first_name", opr=ColumnOperatorEnum.eq, value="Count"),
+ ColumnOperator(col="active", opr=ColumnOperatorEnum.eq, value=False),
+ ]
+ )
+ assert specific_count >= 2 # We created 2 inactive Count users (1, 3)
+
+ # Cleanup
+ for user in users:
+ db.session.delete(user)
+ db.session.commit()
+
+
+# =============================================================================
+# Edge Cases and Special Scenarios
+# =============================================================================
+
+
+def test_find_by_ids_preserves_order(app_context: None) -> None:
+ """Test that find_by_ids returns results correctly."""
+ # Create test users
+ users = []
+ for i in [1300, 1301, 1302, 1303]:
+ user = User(
+ id=i,
+ username=f"order_user_{i}",
+ first_name="Order",
+ last_name=f"User{i}",
+ email=f"order{i}@example.com",
+ active=True,
+ )
+ users.append(user)
+ db.session.add(user)
+ db.session.commit()
+
+ # Request in specific order
+ requested_ids = [1302, 1300, 1303, 1301]
+ found_users = UserDAO.find_by_ids(requested_ids)
+
+ # Note: find_by_ids doesn't guarantee order preservation
+ # but should return all requested items
+ found_ids = [u.id for u in found_users]
+ assert set(found_ids) == set(requested_ids)
+
+ # Cleanup
+ for user in users:
+ db.session.delete(user)
+ db.session.commit()
+
+
+def test_find_by_ids_with_mixed_types(app_context: None) -> None:
+ """Test find_by_ids with integer IDs."""
+ # Create test data
+ database = Database(
+ id=100,
+ database_name="Test DB 100",
+ sqlalchemy_uri="sqlite:///:memory:",
+ )
+ db.session.add(database)
+ db.session.commit()
+
+ # Should handle integer IDs (skip base filter)
+ found_dbs = DatabaseDAO.find_by_ids([100, 100], skip_base_filter=True)
+ # Should find the database
+ assert len(found_dbs) >= 1
+ assert any(d.id == 100 for d in found_dbs)
+
+ # Cleanup
+ db.session.delete(database)
+ db.session.commit()
+
+
+def test_find_by_column_helper_method(app_context: None) -> None:
+ """Test the internal _find_by_column helper method."""
+ # Create test user
+ user = User(
+ id=1400,
+ username="helper_test_user",
+ first_name="Helper",
+ last_name="Test",
+ email="helper@example.com",
+ active=True,
+ )
+ db.session.add(user)
+ db.session.commit()
+
+ # Test the helper method directly
+ found_user = UserDAO._find_by_column("username", "helper_test_user")
+ assert found_user is not None
+ assert found_user.id == 1400
+
+ # Test with non-existent value
+ not_found = UserDAO._find_by_column("username", "non_existent")
+ assert not_found is None
+
+ # Test with invalid column
+ not_found = UserDAO._find_by_column("invalid_column", "value")
+ assert not_found is None
+
+ # Cleanup
+ db.session.delete(user)
+ db.session.commit()
+
+
+def test_find_methods_with_special_characters(app_context: None) -> None:
+ """Test find methods with values containing special characters."""
+ # Create dashboard with special characters in slug
+ special_slug = "test-dashboard-with-special-@#$"
+ dashboard = Dashboard(
+ dashboard_title="Special Dashboard",
+ slug=special_slug,
+ published=True,
+ )
+ db.session.add(dashboard)
+ db.session.commit()
+
+ # Find by slug with special characters (skip base filter)
+ found = DashboardDAO.find_by_id(
+ special_slug, id_column="slug", skip_base_filter=True
+ )
+ assert found is not None
+ assert found.slug == special_slug
+
+ # Test find_by_ids with special characters (skip base filter)
+ found_list = DashboardDAO.find_by_ids(
+ [special_slug], id_column="slug", skip_base_filter=True
+ )
+ assert len(found_list) == 1
+ assert found_list[0].slug == special_slug
+
+ # Cleanup
+ db.session.delete(dashboard)
+ db.session.commit()
+
+
+def test_find_methods_case_sensitivity(app_context: None) -> None:
+ """Test find methods with case-sensitive columns."""
+ # Create users with similar usernames differing in case
+ user1 = User(
+ id=1500,
+ username="CaseSensitive",
+ first_name="Case",
+ last_name="Sensitive",
+ email="case1@example.com",
+ active=True,
+ )
+ user2 = User(
+ id=1501,
+ username="casesensitive",
+ first_name="Case",
+ last_name="Sensitive",
+ email="case2@example.com",
+ active=True,
+ )
+ db.session.add_all([user1, user2])
+
+ try:
+ db.session.commit()
+
+ # Find by exact case
+ found = UserDAO.find_by_id("CaseSensitive", id_column="username")
+ assert found is not None
+ assert found.id == 1500
+
+ found = UserDAO.find_by_id("casesensitive", id_column="username")
+ assert found is not None
+ assert found.id == 1501
+
+ # Cleanup
+ db.session.delete(user1)
+ db.session.delete(user2)
+ db.session.commit()
+ except Exception:
+ # Some databases may have case-insensitive unique constraints
+ db.session.rollback()
+ pytest.skip("Database has case-insensitive unique constraint on username")
+
+
+def test_find_by_ids_empty_and_none_handling(app_context: None) -> None:
+ """Test find_by_ids with empty lists."""
+ # Test with empty list
+ result = UserDAO.find_by_ids([])
+ assert result == []
+
+ # Create test user
+ user = User(
+ id=1600,
+ username="none_test_user",
+ first_name="None",
+ last_name="Test",
+ email="none@example.com",
+ active=True,
+ )
+ db.session.add(user)
+ db.session.commit()
+
+ # Find with valid IDs only
+ found = UserDAO.find_by_ids([1600, 1601])
+ assert len(found) == 1
+ assert found[0].id == 1600
+
+ # Cleanup
+ db.session.delete(user)
+ db.session.commit()
+
+
+def test_find_methods_performance_with_large_lists(app_context: None) -> None:
+ """Test find_by_ids performance with large ID lists."""
+ # Create a batch of users
+ users = []
+ user_ids = []
+ for i in range(1700, 1750): # 50 users
+ user_ids.append(i)
+ user = User(
+ id=i,
+ username=f"perf_user_{i}",
+ first_name="Perf",
+ last_name=f"User{i}",
+ email=f"perf{i}@example.com",
+ active=True,
+ )
+ users.append(user)
+ db.session.add(user)
+ db.session.commit()
+
+ # Find all at once
+ start_time = time.time()
+ found_users = UserDAO.find_by_ids(user_ids)
+ end_time = time.time()
+
+ assert len(found_users) == 50
+ assert (end_time - start_time) < 1.0 # Should complete within 1 second
+
+ # Cleanup
+ for user in users:
+ db.session.delete(user)
+ db.session.commit()
+
+
+# =============================================================================
+# DAO Properties and Configuration Tests
+# =============================================================================
+
+
+def test_base_dao_model_cls_property(app_context: None) -> None:
+ """Test that model_cls property is properly set."""
+ assert UserDAO.model_cls == User
+
+ # Test with a DAO that doesn't set model_cls
+ class InvalidDAO(BaseDAO[User]):
+ pass
+
+ # Should have model_cls attribute set by __init_subclass__
+ # Just verify it exists and is the expected type
+ assert hasattr(InvalidDAO, "model_cls")
+ assert InvalidDAO.model_cls == User # It inherits the generic type
+
+
+def test_base_dao_id_column_name_property(app_context: None) -> None:
+ """Test that id_column_name property is properly used."""
+ assert UserDAO.id_column_name == "id"
+
+ # Test custom id column name
+ class CustomIdDAO(BaseDAO[User]):
+ model_cls = User
+ id_column_name = "username" # Use username as ID column
+
+ # Create test user
+ user = User(
+ id=5016,
+ username="unique-username-id",
+ first_name="Custom",
+ last_name="User",
+ email="custom@example.com",
+ active=True,
+ )
+ db.session.add(user)
+ db.session.commit()
+
+ # Find by custom ID column
+ found = CustomIdDAO.find_by_id("unique-username-id")
+ assert found is not None
+ assert found.first_name == "Custom"
+
+ # Cleanup
+ db.session.delete(user)
+ db.session.commit()
+
+
+def test_base_dao_base_filter_integration(app_context: None) -> None:
+ """Test base_filter integration with list operations."""
+ # Create mix of users
+ users = []
+ for i, active in enumerate([True, False, True, False, True]):
+ user = User(
+ id=2100 + i,
+ username=f"base_filter_user_{i}",
+ first_name="BaseFilter",
+ last_name=f"User{i}",
+ email=f"basefilter{i}@example.com",
+ active=active,
+ )
+ users.append(user)
+ db.session.add(user)
+ db.session.commit()
+
+ # Test basic list functionality
+ results, total = UserDAO.list()
+ assert isinstance(results, list)
+ assert isinstance(total, int)
+ assert total >= 5 # Should include our 5 test users
+
+ # Test skip_base_filter functionality
+ results_skip, total_skip = UserDAO.list()
+ assert isinstance(results_skip, list)
+ assert isinstance(total_skip, int)
+
+ # Test count functionality
+ count_all = UserDAO.count()
+ assert count_all >= 5
+
+ count_skip = UserDAO.count(skip_base_filter=True)
+ assert count_skip >= 5
+
+ # Cleanup
+ for user in users:
+ db.session.delete(user)
+ db.session.commit()
+
+
+def test_base_dao_edge_cases(app_context: None) -> None:
+ """Test BaseDAO edge cases and error conditions."""
+ # Test create without item or attributes
+ created = UserDAO.create()
+ assert created is not None
+ # User model has required fields, so we expect them to be None
+ assert created.username is None
+
+ # Don't commit - would fail due to constraints
+ db.session.rollback()
Review Comment:
I think technically you might have integration tests in here (seem to use the database, didn't analyze to see if there are module-level mocks setup). I don't think we've been great at segmenting properly, so there are probably integration tests in unit tests in Superset and vice-versa. So in theory might belong in `tests/unit_tests/` (?)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat: Add BaseDAO improvements and test reorganization [superset]
Posted by "aminghadersohi (via GitHub)" <gi...@apache.org>.
aminghadersohi commented on code in PR #35018:
URL: https://github.com/apache/superset/pull/35018#discussion_r2334555965
##########
superset/daos/base.py:
##########
@@ -251,3 +455,205 @@ def filter_by(cls, **filter_by: Any) -> list[T]:
cls.id_column_name, data_model
).apply(query, None)
return query.filter_by(**filter_by).all()
+
+ @classmethod
+ def apply_column_operators(
+ cls, query: Any, column_operators: Optional[List[ColumnOperator]] = None
+ ) -> Any:
+ """
+ Apply column operators (list of ColumnOperator) to the query using
+ ColumnOperatorEnum logic. Raises ValueError if a filter references a
+ non-existent column.
+ """
+ if not column_operators:
+ return query
+ for c in column_operators:
+ if not isinstance(c, ColumnOperator):
+ continue
+ col = c.col
+ opr = c.opr
+ value = c.value
+ if not col or not hasattr(cls.model_cls, col):
+ model_name = cls.model_cls.__name__ if cls.model_cls else "Unknown"
+ logging.error(
+ f"Invalid filter: column '{col}' does not exist on {model_name}"
+ )
+ raise ValueError(
+ f"Invalid filter: column '{col}' does not exist on {model_name}"
+ )
+ column = getattr(cls.model_cls, col)
+ try:
+ # Always use ColumnOperatorEnum's apply method
+ operator_enum = ColumnOperatorEnum(opr)
+ query = query.filter(operator_enum.apply(column, value))
+ except Exception as e:
+ logging.error(f"Error applying filter on column '{col}': {e}")
+ raise
+ return query
+
+ @classmethod
+ def get_filterable_columns_and_operators(cls) -> Dict[str, List[str]]:
+ """
+ Returns a dict mapping filterable columns (including hybrid/computed fields if
+ present) to their supported operators. Used by MCP tools to dynamically expose
+ filter options. Custom fields supported by the DAO but not present on the model
+ should be documented here.
+ """
+
+ mapper = inspect(cls.model_cls)
+ columns = {c.key: c for c in mapper.columns}
+ # Add hybrid properties
+ hybrids = {
+ name: attr
+ for name, attr in vars(cls.model_cls).items()
+ if isinstance(attr, hybrid_property)
+ }
+ # You may add custom fields here, e.g.:
+ # custom_fields = {"tags": ["eq", "in_", "like"], ...}
+ custom_fields: Dict[str, List[str]] = {}
+
+ filterable = {}
+ for name, col in columns.items():
+ if isinstance(col.type, (sa.String, sa.Text)):
+ filterable[name] = TYPE_OPERATOR_MAP["string"]
+ elif isinstance(col.type, (sa.Boolean,)):
+ filterable[name] = TYPE_OPERATOR_MAP["boolean"]
+ elif isinstance(col.type, (sa.Integer, sa.Float, sa.Numeric)):
+ filterable[name] = TYPE_OPERATOR_MAP["number"]
+ elif isinstance(col.type, (sa.DateTime, sa.Date, sa.Time)):
+ filterable[name] = TYPE_OPERATOR_MAP["datetime"]
+ else:
+ # Fallback to eq/ne/null
+ filterable[name] = ["eq", "ne", "is_null", "is_not_null"]
+ # Add hybrid properties as string fields by default
+ for name in hybrids:
+ filterable[name] = TYPE_OPERATOR_MAP["string"]
+ # Add custom fields
+ filterable.update(custom_fields)
+ return filterable
+
+ @classmethod
+ def _build_query(
+ cls,
+ column_operators: Optional[List[ColumnOperator]] = None,
+ search: Optional[str] = None,
+ search_columns: Optional[List[str]] = None,
+ custom_filters: Optional[Dict[str, BaseFilter]] = None,
+ skip_base_filter: bool = False,
+ data_model: Optional[SQLAInterface] = None,
+ ) -> Any:
+ """
+ Build a SQLAlchemy query with base filter, column operators, search, and
+ custom filters.
+ """
+ if data_model is None:
+ data_model = SQLAInterface(cls.model_cls, db.session)
+ query = data_model.session.query(cls.model_cls)
+ query = cls._apply_base_filter(
+ query, skip_base_filter=skip_base_filter, data_model=data_model
+ )
+ if search and search_columns:
+ search_filters = []
+ for column_name in search_columns:
+ if hasattr(cls.model_cls, column_name):
+ column = getattr(cls.model_cls, column_name)
+ search_filters.append(cast(column, Text).ilike(f"%{search}%"))
+ if search_filters:
+ query = query.filter(or_(*search_filters))
+ if custom_filters:
+ for filter_class in custom_filters.values():
+ query = filter_class.apply(query, None)
+ if column_operators:
+ query = cls.apply_column_operators(query, column_operators)
+ return query
+
+ @classmethod
+ def list( # noqa: C901
+ cls,
+ column_operators: Optional[List[ColumnOperator]] = None,
+ order_column: str = "changed_on",
+ order_direction: str = "desc",
+ page: int = 0,
+ page_size: int = 100,
+ search: Optional[str] = None,
+ search_columns: Optional[List[str]] = None,
+ custom_filters: Optional[Dict[str, BaseFilter]] = None,
+ columns: Optional[List[str]] = None,
+ ) -> Tuple[List[Any], int]:
+ """
+ Generic list method for filtered, sorted, and paginated results.
+ If columns is specified, returns a list of tuples (one per row),
+ otherwise returns model instances.
+ """
+ data_model = SQLAInterface(cls.model_cls, db.session)
+
+ column_attrs = []
+ relationship_loads = []
+ if columns is None:
+ columns = []
+ for name in columns:
+ attr = getattr(cls.model_cls, name, None)
+ if attr is None:
+ continue
+ prop = getattr(attr, "property", None)
+ if isinstance(prop, ColumnProperty):
+ column_attrs.append(attr)
+ elif isinstance(prop, RelationshipProperty):
+ relationship_loads.append(joinedload(attr))
Review Comment:
i had some initial trouble with using relations in filters and sorting, so they aren't used right now. but thats a good flag to revisit. i will write back my findings.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat: Add BaseDAO improvements and test reorganization [superset]
Posted by "dpgaspar (via GitHub)" <gi...@apache.org>.
dpgaspar commented on code in PR #35018:
URL: https://github.com/apache/superset/pull/35018#discussion_r2332753572
##########
superset/daos/base.py:
##########
@@ -251,3 +455,205 @@ def filter_by(cls, **filter_by: Any) -> list[T]:
cls.id_column_name, data_model
).apply(query, None)
return query.filter_by(**filter_by).all()
+
+ @classmethod
+ def apply_column_operators(
+ cls, query: Any, column_operators: Optional[List[ColumnOperator]] = None
+ ) -> Any:
+ """
+ Apply column operators (list of ColumnOperator) to the query using
+ ColumnOperatorEnum logic. Raises ValueError if a filter references a
+ non-existent column.
+ """
+ if not column_operators:
+ return query
+ for c in column_operators:
+ if not isinstance(c, ColumnOperator):
+ continue
+ col = c.col
+ opr = c.opr
+ value = c.value
+ if not col or not hasattr(cls.model_cls, col):
+ model_name = cls.model_cls.__name__ if cls.model_cls else "Unknown"
+ logging.error(
+ f"Invalid filter: column '{col}' does not exist on {model_name}"
Review Comment:
nit: we should not use f strings when logging for performance reasons:
The f-string is evaluated immediately, even if the log level is higher (e.g., INFO or WARNING) and the DEBUG message will never be logged.
This means you pay the cost of:
Building the string
Evaluating any expressions inside {}
Even if the message is dropped due to the log level.
let's use:
``` python
logging.error("Invalid filter: column '%s' does not exist on %s", col, model_name)
```
##########
superset/daos/base.py:
##########
@@ -251,3 +455,205 @@ def filter_by(cls, **filter_by: Any) -> list[T]:
cls.id_column_name, data_model
).apply(query, None)
return query.filter_by(**filter_by).all()
+
+ @classmethod
+ def apply_column_operators(
+ cls, query: Any, column_operators: Optional[List[ColumnOperator]] = None
+ ) -> Any:
+ """
+ Apply column operators (list of ColumnOperator) to the query using
+ ColumnOperatorEnum logic. Raises ValueError if a filter references a
+ non-existent column.
+ """
+ if not column_operators:
+ return query
+ for c in column_operators:
+ if not isinstance(c, ColumnOperator):
+ continue
+ col = c.col
+ opr = c.opr
+ value = c.value
+ if not col or not hasattr(cls.model_cls, col):
+ model_name = cls.model_cls.__name__ if cls.model_cls else "Unknown"
+ logging.error(
+ f"Invalid filter: column '{col}' does not exist on {model_name}"
+ )
+ raise ValueError(
+ f"Invalid filter: column '{col}' does not exist on {model_name}"
+ )
+ column = getattr(cls.model_cls, col)
+ try:
+ # Always use ColumnOperatorEnum's apply method
+ operator_enum = ColumnOperatorEnum(opr)
+ query = query.filter(operator_enum.apply(column, value))
+ except Exception as e:
+ logging.error(f"Error applying filter on column '{col}': {e}")
+ raise
+ return query
+
+ @classmethod
+ def get_filterable_columns_and_operators(cls) -> Dict[str, List[str]]:
+ """
+ Returns a dict mapping filterable columns (including hybrid/computed fields if
+ present) to their supported operators. Used by MCP tools to dynamically expose
+ filter options. Custom fields supported by the DAO but not present on the model
+ should be documented here.
+ """
+
+ mapper = inspect(cls.model_cls)
+ columns = {c.key: c for c in mapper.columns}
+ # Add hybrid properties
+ hybrids = {
+ name: attr
+ for name, attr in vars(cls.model_cls).items()
+ if isinstance(attr, hybrid_property)
+ }
+ # You may add custom fields here, e.g.:
+ # custom_fields = {"tags": ["eq", "in_", "like"], ...}
+ custom_fields: Dict[str, List[str]] = {}
+
+ filterable = {}
+ for name, col in columns.items():
+ if isinstance(col.type, (sa.String, sa.Text)):
+ filterable[name] = TYPE_OPERATOR_MAP["string"]
+ elif isinstance(col.type, (sa.Boolean,)):
+ filterable[name] = TYPE_OPERATOR_MAP["boolean"]
+ elif isinstance(col.type, (sa.Integer, sa.Float, sa.Numeric)):
+ filterable[name] = TYPE_OPERATOR_MAP["number"]
+ elif isinstance(col.type, (sa.DateTime, sa.Date, sa.Time)):
+ filterable[name] = TYPE_OPERATOR_MAP["datetime"]
+ else:
+ # Fallback to eq/ne/null
+ filterable[name] = ["eq", "ne", "is_null", "is_not_null"]
+ # Add hybrid properties as string fields by default
+ for name in hybrids:
+ filterable[name] = TYPE_OPERATOR_MAP["string"]
+ # Add custom fields
+ filterable.update(custom_fields)
+ return filterable
+
+ @classmethod
+ def _build_query(
+ cls,
+ column_operators: Optional[List[ColumnOperator]] = None,
+ search: Optional[str] = None,
+ search_columns: Optional[List[str]] = None,
+ custom_filters: Optional[Dict[str, BaseFilter]] = None,
+ skip_base_filter: bool = False,
+ data_model: Optional[SQLAInterface] = None,
+ ) -> Any:
+ """
+ Build a SQLAlchemy query with base filter, column operators, search, and
+ custom filters.
+ """
+ if data_model is None:
+ data_model = SQLAInterface(cls.model_cls, db.session)
+ query = data_model.session.query(cls.model_cls)
+ query = cls._apply_base_filter(
+ query, skip_base_filter=skip_base_filter, data_model=data_model
+ )
+ if search and search_columns:
+ search_filters = []
+ for column_name in search_columns:
+ if hasattr(cls.model_cls, column_name):
+ column = getattr(cls.model_cls, column_name)
+ search_filters.append(cast(column, Text).ilike(f"%{search}%"))
+ if search_filters:
+ query = query.filter(or_(*search_filters))
+ if custom_filters:
+ for filter_class in custom_filters.values():
+ query = filter_class.apply(query, None)
+ if column_operators:
+ query = cls.apply_column_operators(query, column_operators)
+ return query
+
+ @classmethod
+ def list( # noqa: C901
+ cls,
+ column_operators: Optional[List[ColumnOperator]] = None,
+ order_column: str = "changed_on",
+ order_direction: str = "desc",
+ page: int = 0,
+ page_size: int = 100,
+ search: Optional[str] = None,
+ search_columns: Optional[List[str]] = None,
+ custom_filters: Optional[Dict[str, BaseFilter]] = None,
+ columns: Optional[List[str]] = None,
+ ) -> Tuple[List[Any], int]:
+ """
+ Generic list method for filtered, sorted, and paginated results.
+ If columns is specified, returns a list of tuples (one per row),
+ otherwise returns model instances.
+ """
+ data_model = SQLAInterface(cls.model_cls, db.session)
+
+ column_attrs = []
+ relationship_loads = []
+ if columns is None:
+ columns = []
+ for name in columns:
+ attr = getattr(cls.model_cls, name, None)
+ if attr is None:
+ continue
+ prop = getattr(attr, "property", None)
+ if isinstance(prop, ColumnProperty):
+ column_attrs.append(attr)
+ elif isinstance(prop, RelationshipProperty):
+ relationship_loads.append(joinedload(attr))
Review Comment:
It seems we will support many-to-many relations here, double check if pagination will work as expected on these cases.
do we have tests for relations?
##########
superset/daos/base.py:
##########
@@ -251,3 +455,205 @@ def filter_by(cls, **filter_by: Any) -> list[T]:
cls.id_column_name, data_model
).apply(query, None)
return query.filter_by(**filter_by).all()
+
+ @classmethod
+ def apply_column_operators(
+ cls, query: Any, column_operators: Optional[List[ColumnOperator]] = None
+ ) -> Any:
+ """
+ Apply column operators (list of ColumnOperator) to the query using
+ ColumnOperatorEnum logic. Raises ValueError if a filter references a
+ non-existent column.
+ """
+ if not column_operators:
+ return query
+ for c in column_operators:
+ if not isinstance(c, ColumnOperator):
+ continue
+ col = c.col
+ opr = c.opr
+ value = c.value
+ if not col or not hasattr(cls.model_cls, col):
+ model_name = cls.model_cls.__name__ if cls.model_cls else "Unknown"
+ logging.error(
+ f"Invalid filter: column '{col}' does not exist on {model_name}"
+ )
+ raise ValueError(
+ f"Invalid filter: column '{col}' does not exist on {model_name}"
+ )
+ column = getattr(cls.model_cls, col)
+ try:
+ # Always use ColumnOperatorEnum's apply method
+ operator_enum = ColumnOperatorEnum(opr)
+ query = query.filter(operator_enum.apply(column, value))
+ except Exception as e:
+ logging.error(f"Error applying filter on column '{col}': {e}")
Review Comment:
nit: let's use lazy formatting
##########
tests/integration_tests/dao/base_dao_test.py:
##########
@@ -0,0 +1,1397 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""
+Integration tests for BaseDAO functionality.
+
+This module contains comprehensive integration tests for the BaseDAO class and its
+subclasses, covering database operations, CRUD methods, flexible column support,
+column operators, and error handling.
+
+Tests use an in-memory SQLite database for isolation and to replicate the unit test
+environment behavior. User model deletions are avoided due to circular dependency
+constraints with self-referential foreign keys.
+"""
+
+import datetime
+import time
+import uuid
+
+import pytest
+from flask_appbuilder.models.filters import BaseFilter
+from flask_appbuilder.security.sqla.models import User
+from sqlalchemy import Column, DateTime, Integer, String
+from sqlalchemy.ext.declarative import declarative_base
+from sqlalchemy.orm.session import Session
+
+from superset.daos.base import BaseDAO, ColumnOperator, ColumnOperatorEnum
+from superset.daos.chart import ChartDAO
+from superset.daos.dashboard import DashboardDAO
+from superset.daos.database import DatabaseDAO
+from superset.daos.user import UserDAO
+from superset.extensions import db
+from superset.models.core import Database
+from superset.models.dashboard import Dashboard
+from superset.models.slice import Slice
+
+# Create a test model for comprehensive testing
+Base = declarative_base()
+
+
+class ExampleModel(Base): # type: ignore
+ __tablename__ = "example_model"
+ id = Column(Integer, primary_key=True)
+ uuid = Column(String(36), unique=True, nullable=False)
+ slug = Column(String(100), unique=True)
+ name = Column(String(100))
+ code = Column(String(50), unique=True)
+ created_on = Column(DateTime, default=datetime.datetime.utcnow)
+
+
+class ExampleModelDAO(BaseDAO[ExampleModel]):
+ model_cls = ExampleModel
+ id_column_name = "id"
+ base_filter = None
+
+
+class MockModel:
+ def __init__(self, id=1, name="test"):
+ self.id = id
+ self.name = name
+
+
+class TestDAO(BaseDAO[MockModel]):
+ model_cls = MockModel
+
+
+@pytest.fixture(autouse=True)
+def mock_g_user(app_context):
+ """Mock the flask g.user for security context."""
+ # Within app context, we can safely mock g
+ from flask import g
+
+ mock_user = User()
+ mock_user.id = 1
+ mock_user.username = "test_user"
+
+ # Set g.user directly instead of patching
+ g.user = mock_user
+ yield
+
+ # Clean up
+ if hasattr(g, "user"):
+ delattr(g, "user")
+
+
+# =============================================================================
+# Integration Tests - These tests use the actual database
+# =============================================================================
+
+
+def test_column_operator_enum_complete_coverage(user_with_data: Session) -> None:
+ """
+ Test that every single ColumnOperatorEnum operator is covered by tests.
+ This ensures we have comprehensive test coverage for all operators.
+ """
+ # Simply verify that we can create queries with all operators
+ for operator in ColumnOperatorEnum:
+ column_operator = ColumnOperator(
+ col="username", opr=operator, value="test_value"
+ )
+ # Just check it doesn't raise an error
+ assert column_operator.opr == operator
+
+
+def test_find_by_id_with_default_column(app_context: Session) -> None:
+ """Test find_by_id with default 'id' column."""
+ # Create a user to test with
+ user = User(
+ username="test_find_by_id",
+ first_name="Test",
+ last_name="User",
+ email="test@example.com",
+ active=True,
+ )
+ db.session.add(user)
+ db.session.commit()
+
+ # Find by numeric id
+ found = UserDAO.find_by_id(user.id, skip_base_filter=True)
+ assert found is not None
+ assert found.id == user.id
+ assert found.username == "test_find_by_id"
+
+ # Test with non-existent id
+ not_found = UserDAO.find_by_id(999999, skip_base_filter=True)
+ assert not_found is None
+
+
+def test_find_by_id_with_uuid_column(app_context: Session) -> None:
+ """Test find_by_id with custom uuid column."""
+ # Create a dashboard with uuid
+ dashboard = Dashboard(
+ dashboard_title="Test UUID Dashboard",
+ slug="test-uuid-dashboard",
+ published=True,
+ )
+ db.session.add(dashboard)
+ db.session.commit()
+
+ # Find by uuid string using the uuid column
+ found = DashboardDAO.find_by_id(
+ str(dashboard.uuid), id_column="uuid", skip_base_filter=True
+ )
+ assert found is not None
+ assert found.uuid == dashboard.uuid
+ assert found.dashboard_title == "Test UUID Dashboard"
+
+ # Find by numeric id (should still work)
+ found_by_id = DashboardDAO.find_by_id(dashboard.id, skip_base_filter=True)
+ assert found_by_id is not None
+ assert found_by_id.id == dashboard.id
+
+ # Test with non-existent uuid
+ not_found = DashboardDAO.find_by_id(str(uuid.uuid4()), skip_base_filter=True)
+ assert not_found is None
+
+
+def test_find_by_id_with_slug_column(app_context: Session) -> None:
+ """Test find_by_id with slug column fallback."""
+ # Create a dashboard with slug
+ dashboard = Dashboard(
+ dashboard_title="Test Slug Dashboard",
+ slug="test-slug-dashboard",
+ published=True,
+ )
+ db.session.add(dashboard)
+ db.session.commit()
+
+ # Find by slug using the slug column
+ found = DashboardDAO.find_by_id(
+ "test-slug-dashboard", id_column="slug", skip_base_filter=True
+ )
+ assert found is not None
+ assert found.slug == "test-slug-dashboard"
+ assert found.dashboard_title == "Test Slug Dashboard"
+
+ # Test with non-existent slug
+ not_found = DashboardDAO.find_by_id("non-existent-slug", skip_base_filter=True)
+ assert not_found is None
+
+
+def test_find_by_id_with_invalid_column(app_context: Session) -> None:
+ """Test find_by_id returns None when column doesn't exist."""
+ # This should return None gracefully
+ result = UserDAO.find_by_id("not_a_valid_id", skip_base_filter=True)
+ assert result is None
+
+
+def test_find_by_id_skip_base_filter(app_context: Session) -> None:
+ """Test find_by_id with skip_base_filter parameter."""
+ # Create users with different active states
+ active_user = User(
+ username="active_user",
+ first_name="Active",
+ last_name="User",
+ email="active@example.com",
+ active=True,
+ )
+ inactive_user = User(
+ username="inactive_user",
+ first_name="Inactive",
+ last_name="User",
+ email="inactive@example.com",
+ active=False,
+ )
+ db.session.add_all([active_user, inactive_user])
+ db.session.commit()
+
+ # Without skipping base filter (if one exists)
+ found_active = UserDAO.find_by_id(active_user.id, skip_base_filter=False)
+ assert found_active is not None
+
+ # With skipping base filter
+ found_active_skip = UserDAO.find_by_id(active_user.id, skip_base_filter=True)
+ assert found_active_skip is not None
+
+ # Both should find the user since UserDAO might not have a base filter
+ assert found_active.id == active_user.id
+ assert found_active_skip.id == active_user.id
+
+
+def test_find_by_ids_with_default_column(app_context: Session) -> None:
+ """Test find_by_ids with default 'id' column."""
+ # Create multiple users
+ users = []
+ for i in range(3):
+ user = User(
+ username=f"test_find_by_ids_{i}",
+ first_name=f"Test{i}",
+ last_name="User",
+ email=f"test{i}@example.com",
+ active=True,
+ )
+ users.append(user)
+ db.session.add(user)
+ db.session.commit()
+
+ # Find by multiple ids
+ ids = [user.id for user in users]
+ found = UserDAO.find_by_ids(ids, skip_base_filter=True)
+ assert len(found) == 3
+ found_ids = [u.id for u in found]
+ assert set(found_ids) == set(ids)
+
+ # Test with mix of existent and non-existent ids
+ mixed_ids = [users[0].id, 999999, users[1].id]
+ found_mixed = UserDAO.find_by_ids(mixed_ids, skip_base_filter=True)
+ assert len(found_mixed) == 2
+
+ # Test with empty list
+ found_empty = UserDAO.find_by_ids([], skip_base_filter=True)
+ assert found_empty == []
+
+
+def test_find_by_ids_with_uuid_column(app_context: Session) -> None:
+ """Test find_by_ids with uuid column."""
+ # Create multiple dashboards
+ dashboards = []
+ for i in range(3):
+ dashboard = Dashboard(
+ dashboard_title=f"Test UUID Dashboard {i}",
+ slug=f"test-uuid-dashboard-{i}",
+ published=True,
+ )
+ dashboards.append(dashboard)
+ db.session.add(dashboard)
+ db.session.commit()
+
+ # Find by multiple uuids
+ uuids = [str(dashboard.uuid) for dashboard in dashboards]
+ found = DashboardDAO.find_by_ids(uuids, id_column="uuid", skip_base_filter=True)
+ assert len(found) == 3
+ found_uuids = [str(d.uuid) for d in found]
+ assert set(found_uuids) == set(uuids)
+
+ # Test with mix of ids and uuids - search separately by column
+ found_by_id = DashboardDAO.find_by_ids([dashboards[0].id], skip_base_filter=True)
+ found_by_uuid = DashboardDAO.find_by_ids(
+ [str(dashboards[1].uuid)], id_column="uuid", skip_base_filter=True
+ )
+ assert len(found_by_id) == 1
+ assert len(found_by_uuid) == 1
+
+
+def test_find_by_ids_with_slug_column(app_context: Session) -> None:
+ """Test find_by_ids with slug column."""
+ # Create multiple dashboards
+ dashboards = []
+ for i in range(3):
+ dashboard = Dashboard(
+ dashboard_title=f"Test Slug Dashboard {i}",
+ slug=f"test-slug-dashboard-{i}",
+ published=True,
+ )
+ dashboards.append(dashboard)
+ db.session.add(dashboard)
+ db.session.commit()
+
+ # Find by multiple slugs
+ slugs = [dashboard.slug for dashboard in dashboards]
+ found = DashboardDAO.find_by_ids(slugs, id_column="slug", skip_base_filter=True)
+ assert len(found) == 3
+ found_slugs = [d.slug for d in found]
+ assert set(found_slugs) == set(slugs)
+
+
+def test_find_by_ids_with_invalid_column(app_context: Session) -> None:
+ """Test find_by_ids returns empty list when column doesn't exist."""
+ # This should return empty list gracefully
+ result = UserDAO.find_by_ids(["not_a_valid_id"], skip_base_filter=True)
+ assert result == []
+
+
+def test_find_by_ids_skip_base_filter(app_context: Session) -> None:
+ """Test find_by_ids with skip_base_filter parameter."""
+ # Create users
+ users = []
+ for i in range(3):
+ user = User(
+ username=f"test_skip_filter_{i}",
+ first_name=f"Test{i}",
+ last_name="User",
+ email=f"test{i}@example.com",
+ active=True,
+ )
+ users.append(user)
+ db.session.add(user)
+ db.session.commit()
+
+ ids = [user.id for user in users]
+
+ # Without skipping base filter
+ found_no_skip = UserDAO.find_by_ids(ids, skip_base_filter=False)
+ assert len(found_no_skip) == 3
+
+ # With skipping base filter
+ found_skip = UserDAO.find_by_ids(ids, skip_base_filter=True)
+ assert len(found_skip) == 3
+
+
+def test_base_dao_create_with_item(app_context: Session) -> None:
+ """Test BaseDAO.create with an item parameter."""
+ # Create a user item
+ user = User(
+ username="created_with_item",
+ first_name="Created",
+ last_name="Item",
+ email="created@example.com",
+ active=True,
+ )
+
+ # Create using the item
+ created = UserDAO.create(item=user)
+ assert created is not None
+ assert created.username == "created_with_item"
+ assert created.first_name == "Created"
+
+ # Verify it's in the session
+ assert created in db.session
+
+ # Commit and verify it persists
+ db.session.commit()
+
+ # Find it again to ensure it was saved
+ found = UserDAO.find_by_id(created.id, skip_base_filter=True)
+ assert found is not None
+ assert found.username == "created_with_item"
+
+
+def test_base_dao_create_with_attributes(app_context: Session) -> None:
+ """Test BaseDAO.create with attributes parameter."""
+ # Create using attributes dict
+ attributes = {
+ "username": "created_with_attrs",
+ "first_name": "Created",
+ "last_name": "Attrs",
+ "email": "attrs@example.com",
+ "active": True,
+ }
+
+ created = UserDAO.create(attributes=attributes)
+ assert created is not None
+ assert created.username == "created_with_attrs"
+ assert created.email == "attrs@example.com"
+
+ # Commit and verify
+ db.session.commit()
+ found = UserDAO.find_by_id(created.id, skip_base_filter=True)
+ assert found is not None
+ assert found.username == "created_with_attrs"
+
+
+def test_base_dao_create_with_both_item_and_attributes(app_context: Session) -> None:
+ """Test BaseDAO.create with both item and attributes (override behavior)."""
+ # Create a user item
+ user = User(
+ username="item_username",
+ first_name="Item",
+ last_name="User",
+ email="item@example.com",
+ active=False,
+ )
+
+ # Override some attributes
+ attributes = {
+ "username": "override_username",
+ "active": True,
+ }
+
+ created = UserDAO.create(item=user, attributes=attributes)
+ assert created is not None
+ assert created.username == "override_username" # Should be overridden
+ assert created.active is True # Should be overridden
+ assert created.first_name == "Item" # Should keep original
+ assert created.last_name == "User" # Should keep original
+
+ db.session.commit()
+
+
+def test_base_dao_update_with_item(app_context: Session) -> None:
+ """Test BaseDAO.update with an item parameter."""
+ # Create a user first
+ user = User(
+ username="update_test",
+ first_name="Original",
+ last_name="User",
+ email="original@example.com",
+ active=True,
+ )
+ db.session.add(user)
+ db.session.commit()
+
+ # Update the user
+ user.first_name = "Updated"
+ updated = UserDAO.update(item=user)
+ assert updated is not None
+ assert updated.first_name == "Updated"
+
+ db.session.commit()
+
+ # Verify the update persisted
+ found = UserDAO.find_by_id(user.id, skip_base_filter=True)
+ assert found is not None
+ assert found.first_name == "Updated"
+
+
+def test_base_dao_update_with_attributes(app_context: Session) -> None:
+ """Test BaseDAO.update with attributes parameter."""
+ # Create a user first
+ user = User(
+ username="update_attrs_test",
+ first_name="Original",
+ last_name="User",
+ email="original@example.com",
+ active=True,
+ )
+ db.session.add(user)
+ db.session.commit()
+
+ # Update using attributes
+ attributes = {"first_name": "Updated", "last_name": "Attr User"}
+ updated = UserDAO.update(item=user, attributes=attributes)
+ assert updated is not None
+ assert updated.first_name == "Updated"
+ assert updated.last_name == "Attr User"
+
+ db.session.commit()
+
+
+def test_base_dao_update_detached_item(app_context: Session) -> None:
+ """Test BaseDAO.update with a detached item."""
+ # Create a user first
+ user = User(
+ username="detached_test",
+ first_name="Original",
+ last_name="User",
+ email="detached@example.com",
+ active=True,
+ )
+ db.session.add(user)
+ db.session.commit()
+
+ user_id = user.id
+
+ # Expunge to detach from session
+ db.session.expunge(user)
Review Comment:
interesting test!
##########
superset/daos/base.py:
##########
@@ -32,6 +51,100 @@
T = TypeVar("T", bound=Model)
+class ColumnOperatorEnum(str, Enum):
+ eq = "eq"
+ ne = "ne"
+ sw = "sw"
+ ew = "ew"
+ in_ = "in"
+ nin = "nin"
+ gt = "gt"
+ gte = "gte"
+ lt = "lt"
+ lte = "lte"
+ like = "like"
+ ilike = "ilike"
+ is_null = "is_null"
+ is_not_null = "is_not_null"
+
+ @classmethod
+ def operator_map(cls) -> Dict[ColumnOperatorEnum, Any]:
+ return {
+ cls.eq: lambda col, val: col == val,
+ cls.ne: lambda col, val: col != val,
+ cls.sw: lambda col, val: col.like(f"{val}%"),
+ cls.ew: lambda col, val: col.like(f"%{val}"),
+ cls.in_: lambda col, val: col.in_(
+ val if isinstance(val, (list, tuple)) else [val]
+ ),
+ cls.nin: lambda col, val: ~col.in_(
+ val if isinstance(val, (list, tuple)) else [val]
+ ),
+ cls.gt: lambda col, val: col > val,
+ cls.gte: lambda col, val: col >= val,
+ cls.lt: lambda col, val: col < val,
+ cls.lte: lambda col, val: col <= val,
+ cls.like: lambda col, val: col.like(f"%{val}%"),
+ cls.ilike: lambda col, val: col.ilike(f"%{val}%"),
+ cls.is_null: lambda col, _: col.is_(None),
+ cls.is_not_null: lambda col, _: col.isnot(None),
+ }
Review Comment:
WDYT about just making this a dict, would be slightly faster since we would avoid a function call
##########
superset/daos/base.py:
##########
@@ -83,39 +196,138 @@ def find_by_id_or_uuid(
return None
@classmethod
- def find_by_id(
- cls,
- model_id: str | int,
- skip_base_filter: bool = False,
- ) -> T | None:
+ def _apply_base_filter(
+ cls, query: Any, skip_base_filter: bool = False, data_model: Any = None
+ ) -> Any:
"""
- Find a model by id, if defined applies `base_filter`
+ Apply the base_filter to the query if it exists and skip_base_filter is False.
"""
- query = db.session.query(cls.model_cls)
if cls.base_filter and not skip_base_filter:
- data_model = SQLAInterface(cls.model_cls, db.session)
+ if data_model is None:
+ data_model = SQLAInterface(cls.model_cls, db.session)
query = cls.base_filter( # pylint: disable=not-callable
cls.id_column_name, data_model
).apply(query, None)
- id_column = getattr(cls.model_cls, cls.id_column_name)
+ return query
+
+ @classmethod
+ def _convert_value_for_column(cls, column: Any, value: Any) -> Any:
+ """
+ Convert a value to the appropriate type for a given SQLAlchemy column.
+
+ Args:
+ column: SQLAlchemy column object
+ value: Value to convert
+
+ Returns:
+ Converted value or None if conversion fails
+ """
+ if (
+ hasattr(column.type, "python_type")
+ and column.type.python_type == uuid_lib.UUID
+ ):
+ if isinstance(value, str):
+ try:
+ return uuid_lib.UUID(value)
+ except (ValueError, AttributeError):
+ return None
+ return value
+
+ @classmethod
+ def _find_by_column(
+ cls,
+ column_name: str,
+ value: str | int,
+ skip_base_filter: bool = False,
+ ) -> T | None:
+ """
+ Private method to find a model by any column value.
+
+ Args:
+ column_name: Name of the column to search by
+ value: Value to search for
+ skip_base_filter: Whether to skip base filtering
+
+ Returns:
+ Model instance or None if not found
+ """
+ query = db.session.query(cls.model_cls)
+ query = cls._apply_base_filter(query, skip_base_filter)
+
+ if not hasattr(cls.model_cls, column_name):
+ return None
+
+ column = getattr(cls.model_cls, column_name)
+ converted_value = cls._convert_value_for_column(column, value)
+ if converted_value is None:
+ return None
+
try:
- return query.filter(id_column == model_id).one_or_none()
+ return query.filter(column == converted_value).one_or_none()
except StatementError:
# can happen if int is passed instead of a string or similar
return None
+ @classmethod
+ def find_by_id(
+ cls,
+ model_id: str | int,
+ skip_base_filter: bool = False,
+ id_column: str | None = None,
+ ) -> T | None:
+ """
+ Find a model by ID using specified or default ID column.
+
+ Args:
+ model_id: ID value to search for
+ skip_base_filter: Whether to skip base filtering
+ id_column: Column name to use (defaults to cls.id_column_name)
+
+ Returns:
+ Model instance or None if not found
+ """
+ column = id_column or cls.id_column_name
+ return cls._find_by_column(column, model_id, skip_base_filter)
+
@classmethod
def find_by_ids(
cls,
- model_ids: list[str] | list[int],
+ model_ids: Sequence[str | int],
skip_base_filter: bool = False,
+ id_column: str | None = None,
) -> list[T]:
"""
Find a List of models by a list of ids, if defined applies `base_filter`
+
+ :param model_ids: List of IDs to find
+ :param skip_base_filter: If true, skip applying the base filter
+ :param id_column: Optional column name to use for ID lookup
+ (defaults to id_column_name)
"""
- id_col = getattr(cls.model_cls, cls.id_column_name, None)
+ column = id_column or cls.id_column_name
+ id_col = getattr(cls.model_cls, column, None)
if id_col is None or not model_ids:
return []
+
+ # Convert IDs to appropriate types based on column type
+ converted_ids: list[str | int | uuid_lib.UUID] = []
+ for id_val in model_ids:
+ converted_value = cls._convert_value_for_column(id_col, id_val)
+ if converted_value is not None:
+ converted_ids.append(converted_value)
Review Comment:
nit: it seems we are assuming that all values are of the same type, should we make sure this is always true?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] Feat/dao updates [superset]
Posted by "korbit-ai[bot] (via GitHub)" <gi...@apache.org>.
korbit-ai[bot] commented on PR #35018:
URL: https://github.com/apache/superset/pull/35018#issuecomment-3254616730
Based on your review schedule, I'll hold off on reviewing this PR until it's marked as ready for review. If you'd like me to take a look now, comment `/korbit-review`.
> Your admin can change your review schedule in the [Korbit Console](https://app.korbit.ai/settings)
[](04d34913-477b-4b15-b714-6be7daa43864)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat: Add BaseDAO improvements and test reorganization [superset]
Posted by "aminghadersohi (via GitHub)" <gi...@apache.org>.
aminghadersohi commented on code in PR #35018:
URL: https://github.com/apache/superset/pull/35018#discussion_r2352929110
##########
superset/daos/base.py:
##########
@@ -83,52 +197,145 @@ def find_by_id_or_uuid(
return None
@classmethod
- def find_by_id(
- cls,
- model_id: str | int,
- skip_base_filter: bool = False,
- ) -> T | None:
+ def _apply_base_filter(
+ cls, query: Any, skip_base_filter: bool = False, data_model: Any = None
+ ) -> Any:
"""
- Find a model by id, if defined applies `base_filter`
+ Apply the base_filter to the query if it exists and skip_base_filter is False.
"""
- query = db.session.query(cls.model_cls)
if cls.base_filter and not skip_base_filter:
- data_model = SQLAInterface(cls.model_cls, db.session)
+ if data_model is None:
+ data_model = SQLAInterface(cls.model_cls, db.session)
query = cls.base_filter( # pylint: disable=not-callable
cls.id_column_name, data_model
).apply(query, None)
- id_column = getattr(cls.model_cls, cls.id_column_name)
+ return query
+
+ @classmethod
+ def _convert_value_for_column(cls, column: Any, value: Any) -> Any:
+ """
+ Convert a value to the appropriate type for a given SQLAlchemy column.
+
+ Args:
+ column: SQLAlchemy column object
+ value: Value to convert
+
+ Returns:
+ Converted value or None if conversion fails
+ """
+ if (
+ hasattr(column.type, "python_type")
+ and column.type.python_type == uuid_lib.UUID
+ ):
+ if isinstance(value, str):
+ try:
+ return uuid_lib.UUID(value)
+ except (ValueError, AttributeError):
+ return None
+ return value
+
+ @classmethod
+ def _find_by_column(
+ cls,
+ column_name: str,
+ value: str | int,
+ skip_base_filter: bool = False,
+ ) -> T | None:
+ """
+ Private method to find a model by any column value.
+
+ Args:
+ column_name: Name of the column to search by
+ value: Value to search for
+ skip_base_filter: Whether to skip base filtering
+
+ Returns:
+ Model instance or None if not found
+ """
+ query = db.session.query(cls.model_cls)
+ query = cls._apply_base_filter(query, skip_base_filter)
+
+ if not hasattr(cls.model_cls, column_name):
+ return None
+
+ column = getattr(cls.model_cls, column_name)
+ converted_value = cls._convert_value_for_column(column, value)
+ if converted_value is None:
+ return None
+
try:
- return query.filter(id_column == model_id).one_or_none()
+ return query.filter(column == converted_value).one_or_none()
except StatementError:
# can happen if int is passed instead of a string or similar
return None
+ @classmethod
+ def find_by_id(
+ cls,
+ model_id: str | int,
+ skip_base_filter: bool = False,
+ id_column: str | None = None,
+ ) -> T | None:
+ """
+ Find a model by ID using specified or default ID column.
+
+ Args:
+ model_id: ID value to search for
+ skip_base_filter: Whether to skip base filtering
+ id_column: Column name to use (defaults to cls.id_column_name)
+
+ Returns:
+ Model instance or None if not found
+ """
+ column = id_column or cls.id_column_name
+ return cls._find_by_column(column, model_id, skip_base_filter)
+
@classmethod
def find_by_ids(
cls,
- model_ids: list[str] | list[int],
+ model_ids: Sequence[str | int],
skip_base_filter: bool = False,
+ id_column: str | None = None,
) -> list[T]:
"""
Find a List of models by a list of ids, if defined applies `base_filter`
+
+ :param model_ids: List of IDs to find
+ :param skip_base_filter: If true, skip applying the base filter
+ :param id_column: Optional column name to use for ID lookup
+ (defaults to id_column_name)
"""
- id_col = getattr(cls.model_cls, cls.id_column_name, None)
+ column = id_column or cls.id_column_name
+ id_col = getattr(cls.model_cls, column, None)
if id_col is None or not model_ids:
return []
- query = db.session.query(cls.model_cls).filter(id_col.in_(model_ids))
- if cls.base_filter and not skip_base_filter:
- data_model = SQLAInterface(cls.model_cls, db.session)
- query = cls.base_filter( # pylint: disable=not-callable
- cls.id_column_name, data_model
- ).apply(query, None)
+
+ # Convert IDs to appropriate types based on column type
+ converted_ids: list[str | int | uuid_lib.UUID] = []
+ for id_val in model_ids:
+ converted_value = cls._convert_value_for_column(id_col, id_val)
+ if converted_value is not None:
+ converted_ids.append(converted_value)
+
+ # Validate type consistency for better error handling
+ if len(converted_ids) > 1:
+ types_found = set(map(type, converted_ids))
+ if len(types_found) > 1:
+ logger.warning(
+ "Mixed ID types detected for %s: %s",
+ cls.model_cls.__name__ if cls.model_cls else "Unknown",
+ [t.__name__ for t in types_found],
+ )
Review Comment:
I like the idea of failing if the types aren't consistent but worried about the risk of these calls sites expecting it to work with mixed ids.
1. API Endpoints (Most likely sources of mixed IDs):
- /charts/api.py - bulk operations accepting arrays of IDs
- /dashboards/api.py - bulk operations accepting arrays of IDs
- /tags/api.py - bulk operations accepting arrays of IDs
- /row_level_security/api.py - bulk operations
2. Command Classes (Delete operations):
- All delete commands that accept lists of IDs from API endpoints
- Examples: ChartDeleteCommand, DashboardDeleteCommand, etc.
3. Tag Operations:
- TagDAO.find_by_ids() when retrieving tagged objects
- Could mix integer and UUID IDs if different object types use
different ID schemes
If you think i shouldn't worry about that and we would prefer fail than warning i can add this:
```
if len(types_found) > 1:
error_msg = f"Mixed ID types detected for {cls.model_cls.__name__ if
cls.model_cls else 'Unknown'}: {[t.__name__ for t in types_found]}"
logger.error(error_msg)
raise ValueError(error_msg)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat: Add BaseDAO improvements and test reorganization [superset]
Posted by "aminghadersohi (via GitHub)" <gi...@apache.org>.
aminghadersohi commented on PR #35018:
URL: https://github.com/apache/superset/pull/35018#issuecomment-3299701012
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat: Add BaseDAO improvements and test reorganization [superset]
Posted by "aminghadersohi (via GitHub)" <gi...@apache.org>.
aminghadersohi commented on code in PR #35018:
URL: https://github.com/apache/superset/pull/35018#discussion_r2334516712
##########
superset/daos/base.py:
##########
@@ -83,39 +196,138 @@ def find_by_id_or_uuid(
return None
@classmethod
- def find_by_id(
- cls,
- model_id: str | int,
- skip_base_filter: bool = False,
- ) -> T | None:
+ def _apply_base_filter(
+ cls, query: Any, skip_base_filter: bool = False, data_model: Any = None
+ ) -> Any:
"""
- Find a model by id, if defined applies `base_filter`
+ Apply the base_filter to the query if it exists and skip_base_filter is False.
"""
- query = db.session.query(cls.model_cls)
if cls.base_filter and not skip_base_filter:
- data_model = SQLAInterface(cls.model_cls, db.session)
+ if data_model is None:
+ data_model = SQLAInterface(cls.model_cls, db.session)
query = cls.base_filter( # pylint: disable=not-callable
cls.id_column_name, data_model
).apply(query, None)
- id_column = getattr(cls.model_cls, cls.id_column_name)
+ return query
+
+ @classmethod
+ def _convert_value_for_column(cls, column: Any, value: Any) -> Any:
+ """
+ Convert a value to the appropriate type for a given SQLAlchemy column.
+
+ Args:
+ column: SQLAlchemy column object
+ value: Value to convert
+
+ Returns:
+ Converted value or None if conversion fails
+ """
+ if (
+ hasattr(column.type, "python_type")
+ and column.type.python_type == uuid_lib.UUID
+ ):
+ if isinstance(value, str):
+ try:
+ return uuid_lib.UUID(value)
+ except (ValueError, AttributeError):
+ return None
+ return value
+
+ @classmethod
+ def _find_by_column(
+ cls,
+ column_name: str,
+ value: str | int,
+ skip_base_filter: bool = False,
+ ) -> T | None:
+ """
+ Private method to find a model by any column value.
+
+ Args:
+ column_name: Name of the column to search by
+ value: Value to search for
+ skip_base_filter: Whether to skip base filtering
+
+ Returns:
+ Model instance or None if not found
+ """
+ query = db.session.query(cls.model_cls)
+ query = cls._apply_base_filter(query, skip_base_filter)
+
+ if not hasattr(cls.model_cls, column_name):
+ return None
+
+ column = getattr(cls.model_cls, column_name)
+ converted_value = cls._convert_value_for_column(column, value)
+ if converted_value is None:
+ return None
+
try:
- return query.filter(id_column == model_id).one_or_none()
+ return query.filter(column == converted_value).one_or_none()
except StatementError:
# can happen if int is passed instead of a string or similar
return None
+ @classmethod
+ def find_by_id(
+ cls,
+ model_id: str | int,
+ skip_base_filter: bool = False,
+ id_column: str | None = None,
+ ) -> T | None:
+ """
+ Find a model by ID using specified or default ID column.
+
+ Args:
+ model_id: ID value to search for
+ skip_base_filter: Whether to skip base filtering
+ id_column: Column name to use (defaults to cls.id_column_name)
+
+ Returns:
+ Model instance or None if not found
+ """
+ column = id_column or cls.id_column_name
+ return cls._find_by_column(column, model_id, skip_base_filter)
+
@classmethod
def find_by_ids(
cls,
- model_ids: list[str] | list[int],
+ model_ids: Sequence[str | int],
skip_base_filter: bool = False,
+ id_column: str | None = None,
) -> list[T]:
"""
Find a List of models by a list of ids, if defined applies `base_filter`
+
+ :param model_ids: List of IDs to find
+ :param skip_base_filter: If true, skip applying the base filter
+ :param id_column: Optional column name to use for ID lookup
+ (defaults to id_column_name)
"""
- id_col = getattr(cls.model_cls, cls.id_column_name, None)
+ column = id_column or cls.id_column_name
+ id_col = getattr(cls.model_cls, column, None)
if id_col is None or not model_ids:
return []
+
+ # Convert IDs to appropriate types based on column type
+ converted_ids: list[str | int | uuid_lib.UUID] = []
+ for id_val in model_ids:
+ converted_value = cls._convert_value_for_column(id_col, id_val)
+ if converted_value is not None:
+ converted_ids.append(converted_value)
+
+ query = db.session.query(cls.model_cls).filter(id_col.in_(converted_ids))
+ query = cls._apply_base_filter(query, skip_base_filter)
+
+ try:
+ results = query.all()
+ except SQLAlchemyError as ex:
+ model_name = cls.model_cls.__name__ if cls.model_cls else "Unknown"
+ raise DAOFindFailedError(
+ f"Failed to find {model_name} with ids: {model_ids}"
+ ) from ex
+
+ return results
Review Comment:
this was my mistake.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat: Add BaseDAO improvements and test reorganization [superset]
Posted by "aminghadersohi (via GitHub)" <gi...@apache.org>.
aminghadersohi commented on code in PR #35018:
URL: https://github.com/apache/superset/pull/35018#discussion_r2334689726
##########
superset/daos/base.py:
##########
@@ -251,3 +455,205 @@ def filter_by(cls, **filter_by: Any) -> list[T]:
cls.id_column_name, data_model
).apply(query, None)
return query.filter_by(**filter_by).all()
+
+ @classmethod
+ def apply_column_operators(
+ cls, query: Any, column_operators: Optional[List[ColumnOperator]] = None
+ ) -> Any:
+ """
+ Apply column operators (list of ColumnOperator) to the query using
+ ColumnOperatorEnum logic. Raises ValueError if a filter references a
+ non-existent column.
+ """
+ if not column_operators:
+ return query
+ for c in column_operators:
+ if not isinstance(c, ColumnOperator):
+ continue
+ col = c.col
+ opr = c.opr
+ value = c.value
+ if not col or not hasattr(cls.model_cls, col):
+ model_name = cls.model_cls.__name__ if cls.model_cls else "Unknown"
+ logging.error(
+ f"Invalid filter: column '{col}' does not exist on {model_name}"
+ )
+ raise ValueError(
+ f"Invalid filter: column '{col}' does not exist on {model_name}"
+ )
+ column = getattr(cls.model_cls, col)
+ try:
+ # Always use ColumnOperatorEnum's apply method
+ operator_enum = ColumnOperatorEnum(opr)
+ query = query.filter(operator_enum.apply(column, value))
+ except Exception as e:
+ logging.error(f"Error applying filter on column '{col}': {e}")
+ raise
+ return query
+
+ @classmethod
+ def get_filterable_columns_and_operators(cls) -> Dict[str, List[str]]:
+ """
+ Returns a dict mapping filterable columns (including hybrid/computed fields if
+ present) to their supported operators. Used by MCP tools to dynamically expose
+ filter options. Custom fields supported by the DAO but not present on the model
+ should be documented here.
+ """
+
+ mapper = inspect(cls.model_cls)
+ columns = {c.key: c for c in mapper.columns}
+ # Add hybrid properties
+ hybrids = {
+ name: attr
+ for name, attr in vars(cls.model_cls).items()
+ if isinstance(attr, hybrid_property)
+ }
+ # You may add custom fields here, e.g.:
+ # custom_fields = {"tags": ["eq", "in_", "like"], ...}
+ custom_fields: Dict[str, List[str]] = {}
+
+ filterable = {}
+ for name, col in columns.items():
+ if isinstance(col.type, (sa.String, sa.Text)):
+ filterable[name] = TYPE_OPERATOR_MAP["string"]
+ elif isinstance(col.type, (sa.Boolean,)):
+ filterable[name] = TYPE_OPERATOR_MAP["boolean"]
+ elif isinstance(col.type, (sa.Integer, sa.Float, sa.Numeric)):
+ filterable[name] = TYPE_OPERATOR_MAP["number"]
+ elif isinstance(col.type, (sa.DateTime, sa.Date, sa.Time)):
+ filterable[name] = TYPE_OPERATOR_MAP["datetime"]
+ else:
+ # Fallback to eq/ne/null
+ filterable[name] = ["eq", "ne", "is_null", "is_not_null"]
+ # Add hybrid properties as string fields by default
+ for name in hybrids:
+ filterable[name] = TYPE_OPERATOR_MAP["string"]
+ # Add custom fields
+ filterable.update(custom_fields)
+ return filterable
+
+ @classmethod
+ def _build_query(
+ cls,
+ column_operators: Optional[List[ColumnOperator]] = None,
+ search: Optional[str] = None,
+ search_columns: Optional[List[str]] = None,
+ custom_filters: Optional[Dict[str, BaseFilter]] = None,
+ skip_base_filter: bool = False,
+ data_model: Optional[SQLAInterface] = None,
+ ) -> Any:
+ """
+ Build a SQLAlchemy query with base filter, column operators, search, and
+ custom filters.
+ """
+ if data_model is None:
+ data_model = SQLAInterface(cls.model_cls, db.session)
+ query = data_model.session.query(cls.model_cls)
+ query = cls._apply_base_filter(
+ query, skip_base_filter=skip_base_filter, data_model=data_model
+ )
+ if search and search_columns:
+ search_filters = []
+ for column_name in search_columns:
+ if hasattr(cls.model_cls, column_name):
+ column = getattr(cls.model_cls, column_name)
+ search_filters.append(cast(column, Text).ilike(f"%{search}%"))
+ if search_filters:
+ query = query.filter(or_(*search_filters))
+ if custom_filters:
+ for filter_class in custom_filters.values():
+ query = filter_class.apply(query, None)
+ if column_operators:
+ query = cls.apply_column_operators(query, column_operators)
+ return query
+
+ @classmethod
+ def list( # noqa: C901
+ cls,
+ column_operators: Optional[List[ColumnOperator]] = None,
+ order_column: str = "changed_on",
+ order_direction: str = "desc",
+ page: int = 0,
+ page_size: int = 100,
+ search: Optional[str] = None,
+ search_columns: Optional[List[str]] = None,
+ custom_filters: Optional[Dict[str, BaseFilter]] = None,
+ columns: Optional[List[str]] = None,
+ ) -> Tuple[List[Any], int]:
+ """
+ Generic list method for filtered, sorted, and paginated results.
+ If columns is specified, returns a list of tuples (one per row),
+ otherwise returns model instances.
+ """
+ data_model = SQLAInterface(cls.model_cls, db.session)
+
+ column_attrs = []
+ relationship_loads = []
+ if columns is None:
+ columns = []
+ for name in columns:
+ attr = getattr(cls.model_cls, name, None)
+ if attr is None:
+ continue
+ prop = getattr(attr, "property", None)
+ if isinstance(prop, ColumnProperty):
+ column_attrs.append(attr)
+ elif isinstance(prop, RelationshipProperty):
+ relationship_loads.append(joinedload(attr))
Review Comment:
Good catch! You're right - joinedload with many-to-many relationships
would break pagination. I've fixed it by counting before adding the
relationship joins. This ensures:
- Accurate counts (not inflated by joins)
- Correct pagination (LIMIT/OFFSET on entities, not joined rows)
We don't have tests for relationship loading in the list method - that
should be added to prevent regression.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat: Add BaseDAO improvements and test reorganization [superset]
Posted by "dpgaspar (via GitHub)" <gi...@apache.org>.
dpgaspar commented on code in PR #35018:
URL: https://github.com/apache/superset/pull/35018#discussion_r2336222130
##########
tests/integration_tests/dao/base_dao_test.py:
##########
@@ -0,0 +1,1397 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""
+Integration tests for BaseDAO functionality.
+
+This module contains comprehensive integration tests for the BaseDAO class and its
+subclasses, covering database operations, CRUD methods, flexible column support,
+column operators, and error handling.
+
+Tests use an in-memory SQLite database for isolation and to replicate the unit test
+environment behavior. User model deletions are avoided due to circular dependency
+constraints with self-referential foreign keys.
+"""
+
+import datetime
+import time
+import uuid
+
+import pytest
+from flask_appbuilder.models.filters import BaseFilter
+from flask_appbuilder.security.sqla.models import User
+from sqlalchemy import Column, DateTime, Integer, String
+from sqlalchemy.ext.declarative import declarative_base
+from sqlalchemy.orm.session import Session
+
+from superset.daos.base import BaseDAO, ColumnOperator, ColumnOperatorEnum
+from superset.daos.chart import ChartDAO
+from superset.daos.dashboard import DashboardDAO
+from superset.daos.database import DatabaseDAO
+from superset.daos.user import UserDAO
+from superset.extensions import db
+from superset.models.core import Database
+from superset.models.dashboard import Dashboard
+from superset.models.slice import Slice
+
+# Create a test model for comprehensive testing
+Base = declarative_base()
+
+
+class ExampleModel(Base): # type: ignore
+ __tablename__ = "example_model"
+ id = Column(Integer, primary_key=True)
+ uuid = Column(String(36), unique=True, nullable=False)
+ slug = Column(String(100), unique=True)
+ name = Column(String(100))
+ code = Column(String(50), unique=True)
+ created_on = Column(DateTime, default=datetime.datetime.utcnow)
+
+
+class ExampleModelDAO(BaseDAO[ExampleModel]):
+ model_cls = ExampleModel
+ id_column_name = "id"
+ base_filter = None
+
+
+class MockModel:
+ def __init__(self, id=1, name="test"):
+ self.id = id
+ self.name = name
+
+
+class TestDAO(BaseDAO[MockModel]):
+ model_cls = MockModel
+
+
+@pytest.fixture(autouse=True)
+def mock_g_user(app_context):
+ """Mock the flask g.user for security context."""
+ # Within app context, we can safely mock g
+ from flask import g
+
+ mock_user = User()
+ mock_user.id = 1
+ mock_user.username = "test_user"
+
+ # Set g.user directly instead of patching
+ g.user = mock_user
+ yield
+
+ # Clean up
+ if hasattr(g, "user"):
+ delattr(g, "user")
+
+
+# =============================================================================
+# Integration Tests - These tests use the actual database
+# =============================================================================
+
+
+def test_column_operator_enum_complete_coverage(user_with_data: Session) -> None:
+ """
+ Test that every single ColumnOperatorEnum operator is covered by tests.
+ This ensures we have comprehensive test coverage for all operators.
+ """
+ # Simply verify that we can create queries with all operators
+ for operator in ColumnOperatorEnum:
+ column_operator = ColumnOperator(
+ col="username", opr=operator, value="test_value"
+ )
+ # Just check it doesn't raise an error
+ assert column_operator.opr == operator
+
+
+def test_find_by_id_with_default_column(app_context: Session) -> None:
+ """Test find_by_id with default 'id' column."""
+ # Create a user to test with
+ user = User(
+ username="test_find_by_id",
+ first_name="Test",
+ last_name="User",
+ email="test@example.com",
+ active=True,
+ )
+ db.session.add(user)
+ db.session.commit()
+
+ # Find by numeric id
+ found = UserDAO.find_by_id(user.id, skip_base_filter=True)
+ assert found is not None
+ assert found.id == user.id
+ assert found.username == "test_find_by_id"
+
+ # Test with non-existent id
+ not_found = UserDAO.find_by_id(999999, skip_base_filter=True)
+ assert not_found is None
+
+
+def test_find_by_id_with_uuid_column(app_context: Session) -> None:
+ """Test find_by_id with custom uuid column."""
+ # Create a dashboard with uuid
+ dashboard = Dashboard(
+ dashboard_title="Test UUID Dashboard",
+ slug="test-uuid-dashboard",
+ published=True,
+ )
+ db.session.add(dashboard)
+ db.session.commit()
+
+ # Find by uuid string using the uuid column
+ found = DashboardDAO.find_by_id(
+ str(dashboard.uuid), id_column="uuid", skip_base_filter=True
+ )
+ assert found is not None
+ assert found.uuid == dashboard.uuid
+ assert found.dashboard_title == "Test UUID Dashboard"
+
+ # Find by numeric id (should still work)
+ found_by_id = DashboardDAO.find_by_id(dashboard.id, skip_base_filter=True)
+ assert found_by_id is not None
+ assert found_by_id.id == dashboard.id
+
+ # Test with non-existent uuid
+ not_found = DashboardDAO.find_by_id(str(uuid.uuid4()), skip_base_filter=True)
+ assert not_found is None
+
+
+def test_find_by_id_with_slug_column(app_context: Session) -> None:
+ """Test find_by_id with slug column fallback."""
+ # Create a dashboard with slug
+ dashboard = Dashboard(
+ dashboard_title="Test Slug Dashboard",
+ slug="test-slug-dashboard",
+ published=True,
+ )
+ db.session.add(dashboard)
+ db.session.commit()
+
+ # Find by slug using the slug column
+ found = DashboardDAO.find_by_id(
+ "test-slug-dashboard", id_column="slug", skip_base_filter=True
+ )
+ assert found is not None
+ assert found.slug == "test-slug-dashboard"
+ assert found.dashboard_title == "Test Slug Dashboard"
+
+ # Test with non-existent slug
+ not_found = DashboardDAO.find_by_id("non-existent-slug", skip_base_filter=True)
+ assert not_found is None
+
+
+def test_find_by_id_with_invalid_column(app_context: Session) -> None:
+ """Test find_by_id returns None when column doesn't exist."""
+ # This should return None gracefully
+ result = UserDAO.find_by_id("not_a_valid_id", skip_base_filter=True)
+ assert result is None
+
+
+def test_find_by_id_skip_base_filter(app_context: Session) -> None:
+ """Test find_by_id with skip_base_filter parameter."""
+ # Create users with different active states
+ active_user = User(
+ username="active_user",
+ first_name="Active",
+ last_name="User",
+ email="active@example.com",
+ active=True,
+ )
+ inactive_user = User(
+ username="inactive_user",
+ first_name="Inactive",
+ last_name="User",
+ email="inactive@example.com",
+ active=False,
+ )
+ db.session.add_all([active_user, inactive_user])
+ db.session.commit()
+
+ # Without skipping base filter (if one exists)
+ found_active = UserDAO.find_by_id(active_user.id, skip_base_filter=False)
+ assert found_active is not None
+
+ # With skipping base filter
+ found_active_skip = UserDAO.find_by_id(active_user.id, skip_base_filter=True)
+ assert found_active_skip is not None
+
+ # Both should find the user since UserDAO might not have a base filter
+ assert found_active.id == active_user.id
+ assert found_active_skip.id == active_user.id
+
+
+def test_find_by_ids_with_default_column(app_context: Session) -> None:
+ """Test find_by_ids with default 'id' column."""
+ # Create multiple users
+ users = []
+ for i in range(3):
+ user = User(
+ username=f"test_find_by_ids_{i}",
+ first_name=f"Test{i}",
+ last_name="User",
+ email=f"test{i}@example.com",
+ active=True,
+ )
+ users.append(user)
+ db.session.add(user)
+ db.session.commit()
+
+ # Find by multiple ids
+ ids = [user.id for user in users]
+ found = UserDAO.find_by_ids(ids, skip_base_filter=True)
+ assert len(found) == 3
+ found_ids = [u.id for u in found]
+ assert set(found_ids) == set(ids)
+
+ # Test with mix of existent and non-existent ids
+ mixed_ids = [users[0].id, 999999, users[1].id]
+ found_mixed = UserDAO.find_by_ids(mixed_ids, skip_base_filter=True)
+ assert len(found_mixed) == 2
+
+ # Test with empty list
+ found_empty = UserDAO.find_by_ids([], skip_base_filter=True)
+ assert found_empty == []
+
+
+def test_find_by_ids_with_uuid_column(app_context: Session) -> None:
+ """Test find_by_ids with uuid column."""
+ # Create multiple dashboards
+ dashboards = []
+ for i in range(3):
+ dashboard = Dashboard(
+ dashboard_title=f"Test UUID Dashboard {i}",
+ slug=f"test-uuid-dashboard-{i}",
+ published=True,
+ )
+ dashboards.append(dashboard)
+ db.session.add(dashboard)
+ db.session.commit()
+
+ # Find by multiple uuids
+ uuids = [str(dashboard.uuid) for dashboard in dashboards]
+ found = DashboardDAO.find_by_ids(uuids, id_column="uuid", skip_base_filter=True)
+ assert len(found) == 3
+ found_uuids = [str(d.uuid) for d in found]
+ assert set(found_uuids) == set(uuids)
+
+ # Test with mix of ids and uuids - search separately by column
+ found_by_id = DashboardDAO.find_by_ids([dashboards[0].id], skip_base_filter=True)
+ found_by_uuid = DashboardDAO.find_by_ids(
+ [str(dashboards[1].uuid)], id_column="uuid", skip_base_filter=True
+ )
+ assert len(found_by_id) == 1
+ assert len(found_by_uuid) == 1
+
+
+def test_find_by_ids_with_slug_column(app_context: Session) -> None:
+ """Test find_by_ids with slug column."""
+ # Create multiple dashboards
+ dashboards = []
+ for i in range(3):
+ dashboard = Dashboard(
+ dashboard_title=f"Test Slug Dashboard {i}",
+ slug=f"test-slug-dashboard-{i}",
+ published=True,
+ )
+ dashboards.append(dashboard)
+ db.session.add(dashboard)
+ db.session.commit()
+
+ # Find by multiple slugs
+ slugs = [dashboard.slug for dashboard in dashboards]
+ found = DashboardDAO.find_by_ids(slugs, id_column="slug", skip_base_filter=True)
+ assert len(found) == 3
+ found_slugs = [d.slug for d in found]
+ assert set(found_slugs) == set(slugs)
+
+
+def test_find_by_ids_with_invalid_column(app_context: Session) -> None:
+ """Test find_by_ids returns empty list when column doesn't exist."""
+ # This should return empty list gracefully
+ result = UserDAO.find_by_ids(["not_a_valid_id"], skip_base_filter=True)
+ assert result == []
+
+
+def test_find_by_ids_skip_base_filter(app_context: Session) -> None:
+ """Test find_by_ids with skip_base_filter parameter."""
+ # Create users
+ users = []
+ for i in range(3):
+ user = User(
+ username=f"test_skip_filter_{i}",
+ first_name=f"Test{i}",
+ last_name="User",
+ email=f"test{i}@example.com",
+ active=True,
+ )
+ users.append(user)
+ db.session.add(user)
+ db.session.commit()
+
+ ids = [user.id for user in users]
+
+ # Without skipping base filter
+ found_no_skip = UserDAO.find_by_ids(ids, skip_base_filter=False)
+ assert len(found_no_skip) == 3
+
+ # With skipping base filter
+ found_skip = UserDAO.find_by_ids(ids, skip_base_filter=True)
+ assert len(found_skip) == 3
+
+
+def test_base_dao_create_with_item(app_context: Session) -> None:
+ """Test BaseDAO.create with an item parameter."""
+ # Create a user item
+ user = User(
+ username="created_with_item",
+ first_name="Created",
+ last_name="Item",
+ email="created@example.com",
+ active=True,
+ )
+
+ # Create using the item
+ created = UserDAO.create(item=user)
+ assert created is not None
+ assert created.username == "created_with_item"
+ assert created.first_name == "Created"
+
+ # Verify it's in the session
+ assert created in db.session
+
+ # Commit and verify it persists
+ db.session.commit()
+
+ # Find it again to ensure it was saved
+ found = UserDAO.find_by_id(created.id, skip_base_filter=True)
+ assert found is not None
+ assert found.username == "created_with_item"
+
+
+def test_base_dao_create_with_attributes(app_context: Session) -> None:
+ """Test BaseDAO.create with attributes parameter."""
+ # Create using attributes dict
+ attributes = {
+ "username": "created_with_attrs",
+ "first_name": "Created",
+ "last_name": "Attrs",
+ "email": "attrs@example.com",
+ "active": True,
+ }
+
+ created = UserDAO.create(attributes=attributes)
+ assert created is not None
+ assert created.username == "created_with_attrs"
+ assert created.email == "attrs@example.com"
+
+ # Commit and verify
+ db.session.commit()
+ found = UserDAO.find_by_id(created.id, skip_base_filter=True)
+ assert found is not None
+ assert found.username == "created_with_attrs"
+
+
+def test_base_dao_create_with_both_item_and_attributes(app_context: Session) -> None:
+ """Test BaseDAO.create with both item and attributes (override behavior)."""
+ # Create a user item
+ user = User(
+ username="item_username",
+ first_name="Item",
+ last_name="User",
+ email="item@example.com",
+ active=False,
+ )
+
+ # Override some attributes
+ attributes = {
+ "username": "override_username",
+ "active": True,
+ }
+
+ created = UserDAO.create(item=user, attributes=attributes)
+ assert created is not None
+ assert created.username == "override_username" # Should be overridden
+ assert created.active is True # Should be overridden
+ assert created.first_name == "Item" # Should keep original
+ assert created.last_name == "User" # Should keep original
+
+ db.session.commit()
+
+
+def test_base_dao_update_with_item(app_context: Session) -> None:
+ """Test BaseDAO.update with an item parameter."""
+ # Create a user first
+ user = User(
+ username="update_test",
+ first_name="Original",
+ last_name="User",
+ email="original@example.com",
+ active=True,
+ )
+ db.session.add(user)
+ db.session.commit()
+
+ # Update the user
+ user.first_name = "Updated"
+ updated = UserDAO.update(item=user)
+ assert updated is not None
+ assert updated.first_name == "Updated"
+
+ db.session.commit()
+
+ # Verify the update persisted
+ found = UserDAO.find_by_id(user.id, skip_base_filter=True)
+ assert found is not None
+ assert found.first_name == "Updated"
+
+
+def test_base_dao_update_with_attributes(app_context: Session) -> None:
+ """Test BaseDAO.update with attributes parameter."""
+ # Create a user first
+ user = User(
+ username="update_attrs_test",
+ first_name="Original",
+ last_name="User",
+ email="original@example.com",
+ active=True,
+ )
+ db.session.add(user)
+ db.session.commit()
+
+ # Update using attributes
+ attributes = {"first_name": "Updated", "last_name": "Attr User"}
+ updated = UserDAO.update(item=user, attributes=attributes)
+ assert updated is not None
+ assert updated.first_name == "Updated"
+ assert updated.last_name == "Attr User"
+
+ db.session.commit()
+
+
+def test_base_dao_update_detached_item(app_context: Session) -> None:
+ """Test BaseDAO.update with a detached item."""
+ # Create a user first
+ user = User(
+ username="detached_test",
+ first_name="Original",
+ last_name="User",
+ email="detached@example.com",
+ active=True,
+ )
+ db.session.add(user)
+ db.session.commit()
+
+ user_id = user.id
+
+ # Expunge to detach from session
+ db.session.expunge(user)
Review Comment:
I like it!!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] Feat/dao updates [superset]
Posted by "bito-code-review[bot] (via GitHub)" <gi...@apache.org>.
bito-code-review[bot] commented on PR #35018:
URL: https://github.com/apache/superset/pull/35018#issuecomment-3264174788
**Bito Automatic Review Skipped - Draft PR**<br/><br/>Bito didn't auto-review because this pull request is in draft status.<br/>No action is needed if you didn't intend for the agent to review it. Otherwise, to manually trigger a review, type `/review` in a comment and save.<br/>You can change draft PR review settings [here](https://alpha.bito.ai/home/ai-agents/code-review-agent), or contact your Bito workspace admin at [evan@preset.io](mailto:evan@preset.io).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] Feat/dao updates [superset]
Posted by "aminghadersohi (via GitHub)" <gi...@apache.org>.
aminghadersohi commented on code in PR #35018:
URL: https://github.com/apache/superset/pull/35018#discussion_r2328828963
##########
tests/unit_tests/dao/base_dao_test.py:
##########
@@ -37,6 +79,1924 @@ class TestDAOWithNoneModel(BaseDAO[MockModel]):
model_cls = None
+@pytest.fixture(autouse=True)
+def mock_g_user():
+ """Mock the flask g.user for security context."""
+ with patch("superset.security.manager.g") as mock_g:
+ mock_user = User()
+ mock_user.id = 1
+ mock_user.username = "test_user"
+ mock_g.user = mock_user
+ yield mock_g
+
+
+# =============================================================================
+# Column Operator Enum Tests
+# =============================================================================
+
+
+def test_column_operator_enum_apply_method() -> None: # noqa: C901
+ """
+ Test that the apply method works correctly for each operator.
+ This verifies the actual SQL generation for each operator.
+ """
+ Base_test = declarative_base() # noqa: N806
+
+ class TestModel(Base_test): # type: ignore
+ __tablename__ = "test_model"
+ id = Column(Integer, primary_key=True)
+ name = Column(String(50))
+ age = Column(Integer)
+ active = Column(Boolean)
+
+ # Test each operator's apply method
+ test_cases = [
+ # (operator, column, value, expected_sql_fragment)
+ (ColumnOperatorEnum.eq, TestModel.name, "test", "name = 'test'"),
+ (ColumnOperatorEnum.ne, TestModel.name, "test", "name != 'test'"),
+ (ColumnOperatorEnum.sw, TestModel.name, "test", "name LIKE 'test%'"),
+ (ColumnOperatorEnum.ew, TestModel.name, "test", "name LIKE '%test'"),
+ (ColumnOperatorEnum.in_, TestModel.id, [1, 2, 3], "id IN (1, 2, 3)"),
+ (ColumnOperatorEnum.nin, TestModel.id, [1, 2, 3], "id NOT IN (1, 2, 3)"),
+ (ColumnOperatorEnum.gt, TestModel.age, 25, "age > 25"),
+ (ColumnOperatorEnum.gte, TestModel.age, 25, "age >= 25"),
+ (ColumnOperatorEnum.lt, TestModel.age, 25, "age < 25"),
+ (ColumnOperatorEnum.lte, TestModel.age, 25, "age <= 25"),
+ (ColumnOperatorEnum.like, TestModel.name, "test", "name LIKE '%test%'"),
+ (ColumnOperatorEnum.ilike, TestModel.name, "test", "name ILIKE '%test%'"),
+ (ColumnOperatorEnum.is_null, TestModel.name, None, "name IS NULL"),
+ (ColumnOperatorEnum.is_not_null, TestModel.name, None, "name IS NOT NULL"),
+ ]
+
+ for operator, column, value, expected_sql_fragment in test_cases:
+ # Apply the operator
+ result = operator.apply(column, value)
+
+ # Convert to string to check SQL generation
+ sql_str = str(result.compile(compile_kwargs={"literal_binds": True}))
+
+ # Verify the SQL contains the expected fragment
+ # Note: SQLAlchemy might format this differently, so we check for key parts
+ if "=" in expected_sql_fragment:
+ assert "=" in sql_str
+ elif "!=" in expected_sql_fragment:
+ assert "!=" in sql_str
+ elif "LIKE" in expected_sql_fragment:
+ assert "LIKE" in sql_str
+ elif "ILIKE" in expected_sql_fragment:
+ assert "ILIKE" in sql_str
+ elif "IN" in expected_sql_fragment:
+ assert "IN" in sql_str
+ elif "NOT IN" in expected_sql_fragment:
+ assert "NOT IN" in sql_str
+ elif ">" in expected_sql_fragment:
+ assert ">" in sql_str
+ elif ">=" in expected_sql_fragment:
+ assert ">=" in sql_str
+ elif "<" in expected_sql_fragment:
+ assert "<" in sql_str
+ elif "<=" in expected_sql_fragment:
+ assert "<=" in sql_str
+ elif "IS NULL" in expected_sql_fragment:
+ assert "IS NULL" in sql_str
+ elif "IS NOT NULL" in expected_sql_fragment:
+ assert "IS NOT NULL" in sql_str
+
+ # Test that all operators are covered
+ all_operators = set(ColumnOperatorEnum)
+ tested_operators = {
+ ColumnOperatorEnum.eq,
+ ColumnOperatorEnum.ne,
+ ColumnOperatorEnum.sw,
+ ColumnOperatorEnum.ew,
+ ColumnOperatorEnum.in_,
+ ColumnOperatorEnum.nin,
+ ColumnOperatorEnum.gt,
+ ColumnOperatorEnum.gte,
+ ColumnOperatorEnum.lt,
+ ColumnOperatorEnum.lte,
+ ColumnOperatorEnum.like,
+ ColumnOperatorEnum.ilike,
+ ColumnOperatorEnum.is_null,
+ ColumnOperatorEnum.is_not_null,
+ }
+
+ # Ensure we've tested all operators
+ assert tested_operators == all_operators, (
+ f"Missing operators: {all_operators - tested_operators}"
+ )
+
+
+def test_column_operator_enum_complete_coverage(user_with_data: Session) -> None:
+ """
+ Test that every single ColumnOperatorEnum operator is covered by tests.
+ This ensures we have comprehensive test coverage for all operators.
+ """
+ # Create test users for comprehensive operator testing
+ test_users = [
+ User(
+ id=900,
+ username="testuser1",
+ first_name="Test",
+ last_name="User1",
+ email="test1@example.com",
+ active=True,
+ ),
+ User(
+ id=901,
+ username="testuser2",
+ first_name="Test",
+ last_name="User2",
+ email="test2@example.com",
+ active=False,
+ ),
+ User(
+ id=902,
+ username="adminuser",
+ first_name="Admin",
+ last_name="User",
+ email="admin@example.com",
+ active=True,
+ ),
+ ]
+
+ for user in test_users:
+ user_with_data.add(user)
+ user_with_data.commit()
+
+ # Test eq (equals)
+ results, _ = UserDAO.list(
+ column_operators=[
+ ColumnOperator(col="username", opr=ColumnOperatorEnum.eq, value="testuser1")
+ ]
+ )
+ assert any(u.username == "testuser1" for u in results)
+
+ # Test ne (not equals)
+ results, _ = UserDAO.list(
+ column_operators=[
+ ColumnOperator(col="username", opr=ColumnOperatorEnum.ne, value="testuser1")
+ ]
+ )
+ assert all(u.username != "testuser1" for u in results)
+
+ # Test sw (starts with)
+ results, _ = UserDAO.list(
+ column_operators=[
+ ColumnOperator(col="username", opr=ColumnOperatorEnum.sw, value="test")
+ ]
+ )
+ assert any(u.username.startswith("test") for u in results)
+
+ # Test ew (ends with)
+ results, _ = UserDAO.list(
+ column_operators=[
+ ColumnOperator(col="username", opr=ColumnOperatorEnum.ew, value="user1")
+ ]
+ )
+ assert any(u.username.endswith("user1") for u in results)
+
+ # Test in_ (in list)
+ user_ids = [900, 901]
+ results, _ = UserDAO.list(
+ column_operators=[
+ ColumnOperator(col="id", opr=ColumnOperatorEnum.in_, value=user_ids)
+ ]
+ )
+ result_ids = [u.id for u in results]
+ assert all(uid in result_ids for uid in user_ids)
+
+ # Test nin (not in list)
+ results, _ = UserDAO.list(
+ column_operators=[
+ ColumnOperator(col="id", opr=ColumnOperatorEnum.nin, value=user_ids)
+ ]
+ )
+ result_ids = [u.id for u in results]
+ assert all(uid not in result_ids for uid in user_ids)
+
+ # Test gt (greater than)
+ results, _ = UserDAO.list(
+ column_operators=[
+ ColumnOperator(col="id", opr=ColumnOperatorEnum.gt, value=900)
+ ]
+ )
+ assert all(u.id > 900 for u in results)
+
+ # Test gte (greater than or equal)
+ results, _ = UserDAO.list(
+ column_operators=[
+ ColumnOperator(col="id", opr=ColumnOperatorEnum.gte, value=901)
+ ]
+ )
+ assert all(u.id >= 901 for u in results)
+
+ # Test lt (less than)
+ results, _ = UserDAO.list(
+ column_operators=[
+ ColumnOperator(col="id", opr=ColumnOperatorEnum.lt, value=902)
+ ]
+ )
+ assert all(u.id < 902 for u in results)
+
+ # Test lte (less than or equal)
+ results, _ = UserDAO.list(
+ column_operators=[
+ ColumnOperator(col="id", opr=ColumnOperatorEnum.lte, value=901)
+ ]
+ )
+ assert all(u.id <= 901 for u in results)
+
+ # Test like (like pattern)
+ results, _ = UserDAO.list(
+ column_operators=[
+ ColumnOperator(col="username", opr=ColumnOperatorEnum.like, value="test")
+ ]
+ )
+ assert any("test" in u.username for u in results)
+
+ # Test ilike (case insensitive like)
+ results, _ = UserDAO.list(
+ column_operators=[
+ ColumnOperator(col="username", opr=ColumnOperatorEnum.ilike, value="TEST")
+ ]
+ )
+ assert any("test" in u.username.lower() for u in results)
+
+ # Test is_null (is null)
+ # Create a user with a null field that allows nulls
+ user_with_data.add(
+ User(
+ id=903,
+ username="nulluser",
+ first_name="Test",
+ last_name="User",
+ email="nulluser@example.com",
+ active=True,
+ )
+ )
+ user_with_data.commit()
+
+ # Update to set a field that can be null (we'll use a field that allows nulls)
+ # For this test, we'll check if we can find users where a field is null
+ # Since we can't easily set first_name to null, we'll test the is_null operator
+ # by checking if it works with fields that might be null
+ results, _ = UserDAO.list(
+ column_operators=[
+ ColumnOperator(col="first_name", opr=ColumnOperatorEnum.is_null, value=None)
+ ]
+ )
+ # This should return empty since all users have first_name set
+ assert len(results) == 0
+
+ # Test is_not_null (is not null)
+ results, _ = UserDAO.list(
+ column_operators=[
+ ColumnOperator(
+ col="first_name", opr=ColumnOperatorEnum.is_not_null, value=None
+ )
+ ]
+ )
+ # This should return all users since all have first_name set
+ assert len(results) > 0
+
+ # Test boolean operators
+ results, _ = UserDAO.list(
+ column_operators=[
+ ColumnOperator(col="active", opr=ColumnOperatorEnum.eq, value=True)
+ ]
+ )
+ assert all(u.active for u in results)
+
+ results, _ = UserDAO.list(
+ column_operators=[
+ ColumnOperator(col="active", opr=ColumnOperatorEnum.eq, value=False)
+ ]
+ )
+ assert all(not u.active for u in results)
+
+ # Test that all operators are covered
+ all_operators = set(ColumnOperatorEnum)
+ tested_operators = {
+ ColumnOperatorEnum.eq,
+ ColumnOperatorEnum.ne,
+ ColumnOperatorEnum.sw,
+ ColumnOperatorEnum.ew,
+ ColumnOperatorEnum.in_,
+ ColumnOperatorEnum.nin,
+ ColumnOperatorEnum.gt,
+ ColumnOperatorEnum.gte,
+ ColumnOperatorEnum.lt,
+ ColumnOperatorEnum.lte,
+ ColumnOperatorEnum.like,
+ ColumnOperatorEnum.ilike,
+ ColumnOperatorEnum.is_null,
+ ColumnOperatorEnum.is_not_null,
+ }
+
+ # Ensure we've tested all operators
+ assert tested_operators == all_operators, (
+ f"Missing operators: {all_operators - tested_operators}"
+ )
+
+
+# =============================================================================
+# Find by ID Tests - Testing flexible column support
+# =============================================================================
+
+
+def test_find_by_id_with_default_column(app_context: None) -> None:
+ """Test find_by_id using the default id column."""
+ # Create test data
+ user = User(
+ id=1000,
+ username="test_find_user",
+ first_name="Test",
+ last_name="User",
+ email="testfind@example.com",
+ active=True,
+ )
+ db.session.add(user)
+ db.session.commit()
+
+ # Find by ID using default column
+ found_user = UserDAO.find_by_id(1000)
+ assert found_user is not None
+ assert found_user.id == 1000
+ assert found_user.username == "test_find_user"
+
+ # Test not found
+ not_found = UserDAO.find_by_id(9999)
+ assert not_found is None
+
+ # Cleanup
+ db.session.delete(user)
+ db.session.commit()
+
+
+def test_find_by_id_with_uuid_column(app_context: None) -> None:
+ """Test find_by_id using UUID column."""
+ # Create dashboard with UUID
+ dashboard = Dashboard(
+ dashboard_title="Test UUID Dashboard",
+ slug="test-uuid-dashboard",
+ published=True,
+ )
+ db.session.add(dashboard)
+ db.session.commit()
+
+ # Verify dashboard exists (skip base filter for test)
+ found_by_id = DashboardDAO.find_by_id(dashboard.id, skip_base_filter=True)
+ assert found_by_id is not None
+
+ # Get the UUID as string for lookup
+ dashboard_uuid_str = str(dashboard.uuid)
+
+ # Find by UUID (skip base filter)
+ found_dashboard = DashboardDAO.find_by_id(
+ dashboard_uuid_str, id_column="uuid", skip_base_filter=True
+ )
+ assert found_dashboard is not None
+ assert str(found_dashboard.uuid) == dashboard_uuid_str
+ assert found_dashboard.dashboard_title == "Test UUID Dashboard"
+
+ # Test not found with invalid UUID
+ not_found = DashboardDAO.find_by_id("invalid-uuid", id_column="uuid")
+ assert not_found is None
+
+ # Cleanup
+ db.session.delete(dashboard)
+ db.session.commit()
+
+
+def test_find_by_id_with_slug_column(app_context: None) -> None:
+ """Test find_by_id using slug column."""
+ # Create test dashboard with slug
+ dashboard = Dashboard(
+ dashboard_title="Test Slug Dashboard",
+ slug="unique-test-slug",
+ published=True,
+ )
+ db.session.add(dashboard)
+ db.session.commit()
+
+ # Find by slug (skip base filter)
+ found_dashboard = DashboardDAO.find_by_id(
+ "unique-test-slug", id_column="slug", skip_base_filter=True
+ )
+ assert found_dashboard is not None
+ assert found_dashboard.slug == "unique-test-slug"
+ assert found_dashboard.dashboard_title == "Test Slug Dashboard"
+
+ # Test not found
+ not_found = DashboardDAO.find_by_id(
+ "non-existent-slug", id_column="slug", skip_base_filter=True
+ )
+ assert not_found is None
+
+ # Cleanup
+ db.session.delete(dashboard)
+ db.session.commit()
+
+
+def test_find_by_id_with_invalid_column(app_context: None) -> None:
+ """Test find_by_id with invalid column name."""
+ # Should return None when column doesn't exist
+ result = UserDAO.find_by_id(1, id_column="non_existent_column")
+ assert result is None
+
+
+def test_find_by_id_skip_base_filter(app_context: None) -> None:
+ """Test find_by_id with skip_base_filter option."""
+ # Create two users - one active, one inactive
+ active_user = User(
+ id=1001,
+ username="active_user",
+ first_name="Active",
+ last_name="User",
+ email="active@example.com",
+ active=True,
+ )
+ inactive_user = User(
+ id=1002,
+ username="inactive_user",
+ first_name="Inactive",
+ last_name="User",
+ email="inactive@example.com",
+ active=False,
+ )
+ db.session.add_all([active_user, inactive_user])
+ db.session.commit()
+
+ # Test basic finding without base filter
+ found_active = UserDAO.find_by_id(1001)
+ assert found_active is not None
+ assert found_active.id == 1001
+
+ found_inactive = UserDAO.find_by_id(1002)
+ assert found_inactive is not None
+ assert found_inactive.id == 1002
+
+ # Test skip_base_filter parameter works (should have same result for UserDAO)
+ found_with_skip = UserDAO.find_by_id(1001, skip_base_filter=True)
+ assert found_with_skip is not None
+ assert found_with_skip.id == 1001
+
+ # Cleanup
+ db.session.delete(active_user)
+ db.session.delete(inactive_user)
+ db.session.commit()
+
+
+# =============================================================================
+# Find by IDs Tests - Testing bulk operations with flexible columns
+# =============================================================================
+
+
+def test_find_by_ids_with_default_column(app_context: None) -> None:
+ """Test find_by_ids using the default id column."""
+ # Create multiple test users
+ users = []
+ for i in range(1100, 1105):
+ user = User(
+ id=i,
+ username=f"bulk_user_{i}",
+ first_name="Bulk",
+ last_name=f"User{i}",
+ email=f"bulk{i}@example.com",
+ active=True,
+ )
+ users.append(user)
+ db.session.add(user)
+ db.session.commit()
+
+ # Find multiple by IDs
+ ids_to_find = [1100, 1102, 1104]
+ found_users = UserDAO.find_by_ids(ids_to_find)
+ assert len(found_users) == 3
+ found_ids = [u.id for u in found_users]
+ assert set(found_ids) == set(ids_to_find)
+
+ # Test with some non-existent IDs
+ mixed_ids = [1101, 1103, 9999, 8888]
+ found_users = UserDAO.find_by_ids(mixed_ids)
+ assert len(found_users) == 2
+ found_ids = [u.id for u in found_users]
+ assert set(found_ids) == {1101, 1103}
+
+ # Test empty list
+ found_users = UserDAO.find_by_ids([])
+ assert found_users == []
+
+ # Cleanup
+ for user in users:
+ db.session.delete(user)
+ db.session.commit()
+
+
+def test_find_by_ids_with_uuid_column(app_context: None) -> None:
+ """Test find_by_ids using UUID column."""
+ # Create multiple charts with UUIDs
+ charts = []
+ chart_uuids = []
+ for i in range(3):
+ chart_uuid = str(uuid.uuid4())
+ chart_uuids.append(chart_uuid)
+ chart = Slice(
+ slice_name=f"Test Chart {i}",
+ viz_type="table",
+ datasource_id=1,
+ datasource_type="table",
+ params="{}",
+ uuid=chart_uuid,
+ )
+ charts.append(chart)
+ db.session.add(chart)
+ db.session.commit()
+
+ # Find multiple by UUIDs (skip base filter)
+ found_charts = ChartDAO.find_by_ids(
+ chart_uuids[:2], id_column="uuid", skip_base_filter=True
+ )
+ assert len(found_charts) == 2
+ found_uuids = [str(c.uuid) for c in found_charts]
+ assert set(found_uuids) == set(chart_uuids[:2])
+
+ # Test with mixed valid and invalid UUIDs (skip base filter)
+ mixed_uuids = [chart_uuids[0], "invalid-uuid", chart_uuids[2]]
+ found_charts = ChartDAO.find_by_ids(
+ mixed_uuids, id_column="uuid", skip_base_filter=True
+ )
+ assert len(found_charts) == 2
+ found_uuids = [str(c.uuid) for c in found_charts]
+ assert set(found_uuids) == {chart_uuids[0], chart_uuids[2]}
+
+ # Cleanup
+ for chart in charts:
+ db.session.delete(chart)
+ db.session.commit()
+
+
+def test_find_by_ids_with_slug_column(app_context: None) -> None:
+ """Test find_by_ids using slug column."""
+ # Create multiple dashboards with slugs
+ dashboards = []
+ slugs = []
+ for i in range(3):
+ slug = f"test-dashboard-{i}"
+ slugs.append(slug)
+ dashboard = Dashboard(
+ dashboard_title=f"Test Dashboard {i}",
+ slug=slug,
+ published=True,
+ )
+ dashboards.append(dashboard)
+ db.session.add(dashboard)
+ db.session.commit()
+
+ # Find multiple by slugs (skip base filter)
+ found_dashboards = DashboardDAO.find_by_ids(
+ slugs[:2], id_column="slug", skip_base_filter=True
+ )
+ assert len(found_dashboards) == 2
+ found_slugs = [d.slug for d in found_dashboards]
+ assert set(found_slugs) == set(slugs[:2])
+
+ # Test with non-existent slugs
+ mixed_slugs = [slugs[0], "non-existent-slug", slugs[2]]
+ found_dashboards = DashboardDAO.find_by_ids(
+ mixed_slugs, id_column="slug", skip_base_filter=True
+ )
+ assert len(found_dashboards) == 2
+ found_slugs = [d.slug for d in found_dashboards]
+ assert set(found_slugs) == {slugs[0], slugs[2]}
+
+ # Cleanup
+ for dashboard in dashboards:
+ db.session.delete(dashboard)
+ db.session.commit()
+
+
+def test_find_by_ids_with_invalid_column(app_context: None) -> None:
+ """Test find_by_ids with invalid column name."""
+ # Should return empty list when column doesn't exist
+ result = UserDAO.find_by_ids([1, 2, 3], id_column="non_existent_column")
+ assert result == []
+
+
+def test_find_by_ids_skip_base_filter(app_context: None) -> None:
+ """Test find_by_ids with skip_base_filter option."""
+ # Create mix of active and inactive users
+ users = []
+ user_ids = []
+ for i, active in enumerate([True, False, True, False]):
+ user_id = 1200 + i
+ user_ids.append(user_id)
+ user = User(
+ id=user_id,
+ username=f"filter_user_{user_id}",
+ first_name="Filter",
+ last_name=f"User{user_id}",
+ email=f"filter{user_id}@example.com",
+ active=active,
+ )
+ users.append(user)
+ db.session.add(user)
+ db.session.commit()
+
+ # Test finding all users
+ found_users = UserDAO.find_by_ids(user_ids)
+ assert len(found_users) == 4
+ found_ids = [u.id for u in found_users]
+ assert set(found_ids) == set(user_ids)
+
+ # Test skip_base_filter parameter works (should have same result for UserDAO)
+ found_users_skip = UserDAO.find_by_ids(user_ids, skip_base_filter=True)
+ assert len(found_users_skip) == 4
+ found_ids_skip = [u.id for u in found_users_skip]
+ assert set(found_ids_skip) == set(user_ids)
+
+ # Cleanup
+ for user in users:
+ db.session.delete(user)
+ db.session.commit()
+
+
+# =============================================================================
+# CRUD Operations Tests - Create, Read, Update, Delete
+# =============================================================================
+
+
+def test_base_dao_create_with_item(app_context: None) -> None:
+ """Test BaseDAO create method with an item."""
+ # Create a new user
+ user = User(
+ id=5000,
+ username="test_create_user",
+ first_name="Test",
+ last_name="Create",
+ email="testcreate@example.com",
+ active=True,
+ )
+ created = UserDAO.create(item=user)
+
+ # Verify it's added to session
+ assert created is user
+ assert created in db.session
+
+ # Commit and verify it persists
+ db.session.commit()
+ assert created.id is not None
+
+ # Verify we can find it
+ found = UserDAO.find_by_id(created.id)
+ assert found is not None
+ assert found.username == "test_create_user"
+ assert found.first_name == "Test"
+
+ # Cleanup
+ db.session.delete(created)
+ db.session.commit()
+
+
+def test_base_dao_create_with_attributes(app_context: None) -> None:
+ """Test BaseDAO create method with attributes dict."""
+ # Create with attributes
+ attributes = {
+ "id": 5001,
+ "username": "attr_user",
+ "first_name": "Attr",
+ "last_name": "User",
+ "email": "attr@example.com",
+ "active": True,
+ }
+ created = UserDAO.create(attributes=attributes)
+
+ # Verify attributes were set
+ assert created.username == "attr_user"
+ assert created.first_name == "Attr"
+ assert created in db.session
+
+ # Commit and verify
+ db.session.commit()
+ assert created.id is not None
+
+ # Cleanup
+ db.session.delete(created)
+ db.session.commit()
+
+
+def test_base_dao_create_with_both_item_and_attributes(app_context: None) -> None:
+ """Test BaseDAO create with both item and attributes."""
+ # Create user with some values
+ user = User(
+ id=5002,
+ username="initial_user",
+ first_name="Initial",
+ last_name="User",
+ email="initial@example.com",
+ active=True,
+ )
+
+ # Override with attributes
+ attributes = {
+ "username": "updated_user",
+ "first_name": "Updated",
+ }
+ created = UserDAO.create(item=user, attributes=attributes)
+
+ # Attributes should override item values
+ assert created.username == "updated_user"
+ assert created.first_name == "Updated"
+ assert created.last_name == "User" # Should keep original
+
+ db.session.commit()
+
+ # Cleanup
+ db.session.delete(created)
+ db.session.commit()
+
+
+def test_base_dao_update_with_item(app_context: None) -> None:
+ """Test BaseDAO update method with an item."""
+ # Create a user first
+ user = User(
+ id=5003,
+ username="original_user",
+ first_name="Original",
+ last_name="User",
+ email="original@example.com",
+ active=True,
+ )
+ db.session.add(user)
+ db.session.commit()
+
+ # Update it
+ user.first_name = "Updated"
+ updated = UserDAO.update(item=user)
+
+ assert updated is user
+ assert updated.first_name == "Updated"
+
+ db.session.commit()
+
+ # Verify update persisted
+ found = UserDAO.find_by_id(user.id)
+ assert found is not None
+ assert found.first_name == "Updated"
+
+ # Cleanup
+ db.session.delete(user)
+ db.session.commit()
+
+
+def test_base_dao_update_with_attributes(app_context: None) -> None:
+ """Test BaseDAO update method with attributes."""
+ # Create a user first
+ user = User(
+ id=5004,
+ username="attr_update_user",
+ first_name="Original",
+ last_name="User",
+ email="attrupdate@example.com",
+ active=True,
+ )
+ db.session.add(user)
+ db.session.commit()
+
+ # Update with attributes
+ attributes = {
+ "first_name": "Attr Updated",
+ "last_name": "Attr User",
+ }
+ updated = UserDAO.update(item=user, attributes=attributes)
+
+ assert updated.first_name == "Attr Updated"
+ assert updated.last_name == "Attr User"
+
+ db.session.commit()
+
+ # Cleanup
+ db.session.delete(user)
+ db.session.commit()
+
+
+def test_base_dao_update_detached_item(app_context: None) -> None:
+ """Test BaseDAO update with a detached item."""
+ # Create and commit a user
+ user = User(
+ id=5005,
+ username="detached_user",
+ first_name="Detached",
+ last_name="User",
+ email="detached@example.com",
+ active=True,
+ )
+ db.session.add(user)
+ db.session.commit()
+ user_id = user.id
+
+ # Detach it from session
+ db.session.expunge(user)
+
+ # Update the detached user
+ user.first_name = "Updated Detached"
+ updated = UserDAO.update(item=user)
+
+ # Should merge it back into session
+ assert updated in db.session
+ assert updated.first_name == "Updated Detached"
+
+ db.session.commit()
+
+ # Verify update
+ found = UserDAO.find_by_id(user_id)
+ assert found is not None
+ assert found.first_name == "Updated Detached"
+
+ # Cleanup
+ db.session.delete(found)
+ db.session.commit()
+
+
+def test_base_dao_delete_single_item(app_context: None) -> None:
+ """Test BaseDAO delete method with single item."""
+ # Create a user
+ user = User(
+ id=5006,
+ username="delete_user",
+ first_name="Delete",
+ last_name="User",
+ email="delete@example.com",
+ active=True,
+ )
+ db.session.add(user)
+ db.session.commit()
+ user_id = user.id
+
+ # Delete it
+ UserDAO.delete([user])
+ db.session.commit()
+
+ # Verify it's gone
+ found = UserDAO.find_by_id(user_id)
+ assert found is None
+
+
+def test_base_dao_delete_multiple_items(app_context: None) -> None:
+ """Test BaseDAO delete method with multiple items."""
+ # Create multiple users
+ users = []
+ for i in range(3):
+ user = User(
+ id=5007 + i,
+ username=f"delete_user_{i}",
+ first_name="Delete",
+ last_name=f"User{i}",
+ email=f"delete{i}@example.com",
+ active=True,
+ )
+ users.append(user)
+ db.session.add(user)
+ db.session.commit()
+
+ user_ids = [user.id for user in users]
+
+ # Delete all at once
+ UserDAO.delete(users)
+ db.session.commit()
+
+ # Verify all are gone
+ for user_id in user_ids:
+ found = UserDAO.find_by_id(user_id)
+ assert found is None
+
+
+def test_base_dao_delete_empty_list(app_context: None) -> None:
+ """Test BaseDAO delete with empty list."""
+ # Should not raise any errors
+ UserDAO.delete([])
+ db.session.commit()
+
+
+def test_base_dao_find_all(app_context: None) -> None:
+ """Test BaseDAO find_all method."""
+ # Create test users
+ users = []
+ for i in range(3):
+ user = User(
+ id=5010 + i,
+ username=f"find_all_user_{i}",
+ first_name="FindAll",
+ last_name=f"User{i}",
+ email=f"findall{i}@example.com",
+ active=True,
+ )
+ users.append(user)
+ db.session.add(user)
+ db.session.commit()
+
+ # Find all
+ all_users = UserDAO.find_all()
+
+ # Should include our created users
+ usernames = [user.username for user in all_users]
+ for i in range(3):
+ assert f"find_all_user_{i}" in usernames
+
+ # Cleanup
+ for user in users:
+ db.session.delete(user)
+ db.session.commit()
+
+
+def test_base_dao_find_one_or_none(app_context: None) -> None:
+ """Test BaseDAO find_one_or_none method."""
+ # Create test users
+ user1 = User(
+ id=5013,
+ username="unique_user",
+ first_name="Unique",
+ last_name="User",
+ email="unique@example.com",
+ active=True,
+ )
+ user2 = User(
+ id=5014,
+ username="duplicate_user_1",
+ first_name="Duplicate",
+ last_name="First",
+ email="dup1@example.com",
+ active=True,
+ )
+ user3 = User(
+ id=5015,
+ username="duplicate_user_2",
+ first_name="Duplicate",
+ last_name="Second",
+ email="dup2@example.com",
+ active=True,
+ )
+ db.session.add_all([user1, user2, user3])
+ db.session.commit()
+
+ # Find unique user
+ found = UserDAO.find_one_or_none(username="unique_user")
+ assert found is not None
+ assert found.first_name == "Unique"
+
+ # Find non-existent user
+ not_found = UserDAO.find_one_or_none(username="does_not_exist")
+ assert not_found is None
+
+ # Cleanup
+ db.session.delete(user1)
+ db.session.delete(user2)
+ db.session.delete(user3)
+ db.session.commit()
+
+
+# =============================================================================
+# List and Filtering Tests
+# =============================================================================
+
+
+def test_base_dao_list_returns_results(user_with_data: Session) -> None:
+ results, total = UserDAO.list()
+ assert total >= 1
+ assert any(u.username == "testuser" for u in results)
+
+
+def test_base_dao_list_with_column_operators(user_with_data: Session) -> None:
+ results, total = UserDAO.list(
+ column_operators=[
+ ColumnOperator(col="username", opr=ColumnOperatorEnum.eq, value="testuser")
+ ]
+ )
+ assert total >= 1
+ assert all(u.username == "testuser" for u in results)
+
+
+def test_base_dao_list_with_non_matching_column_operator(
+ user_with_data: Session,
+) -> None:
+ results, total = UserDAO.list(
+ column_operators=[
+ ColumnOperator(
+ col="username", opr=ColumnOperatorEnum.eq, value="doesnotexist"
+ )
+ ]
+ )
+ assert total == 0
+ assert results == []
+
+
+def test_base_dao_count_returns_value(user_with_data: Session) -> None:
+ count = UserDAO.count()
+ assert count >= 1
+
+
+def test_base_dao_count_with_column_operators(user_with_data: Session) -> None:
+ count = UserDAO.count(
+ column_operators=[
+ ColumnOperator(col="username", opr=ColumnOperatorEnum.eq, value="testuser")
+ ]
+ )
+ assert count >= 1
+ count = UserDAO.count(
+ column_operators=[
+ ColumnOperator(
+ col="username", opr=ColumnOperatorEnum.eq, value="doesnotexist"
+ )
+ ]
+ )
+ assert count == 0
+
+
+def test_base_dao_list_and_count_skip_base_filter(user_with_data: Session) -> None:
+ results, total = UserDAO.list()
+ results_skip, total_skip = UserDAO.list()
+ assert total == total_skip
+ count = UserDAO.count()
+ count_skip = UserDAO.count(skip_base_filter=True)
+ assert count_skip == count
+
+
+def test_base_dao_list_ordering(user_with_data: Session) -> None:
+ user_with_data.add_all(
+ [
+ User(
+ id=201,
+ username="buser",
+ first_name="B",
+ last_name="User",
+ email="buser@example.com",
+ active=True,
+ ),
+ User(
+ id=202,
+ username="auser",
+ first_name="A",
+ last_name="User",
+ email="auser@example.com",
+ active=True,
+ ),
+ User(
+ id=203,
+ username="zuser",
+ first_name="Z",
+ last_name="User",
+ email="zuser@example.com",
+ active=True,
+ ),
+ ]
+ )
+ user_with_data.commit()
+ results, _ = UserDAO.list(order_column="username", order_direction="asc")
+ usernames = [u.username for u in results]
+ assert usernames == sorted(usernames)
+ results, _ = UserDAO.list(order_column="id", order_direction="desc")
+ ids = [u.id for u in results]
+ assert ids == sorted(ids, reverse=True)
+
+
+def test_base_dao_list_paging(user_with_data: Session) -> None:
+ users = [
+ User(
+ id=300 + i,
+ username=f"user{i}",
+ first_name=f"F{i}",
+ last_name="User",
+ email=f"user{i}@example.com",
+ active=True,
+ )
+ for i in range(10)
+ ]
+ user_with_data.add_all(users)
+ user_with_data.commit()
+ results, total = UserDAO.list(
+ order_column="id", order_direction="asc", page=0, page_size=5
+ )
+ ids = [u.id for u in results]
+ all_results, _ = UserDAO.list(order_column="id", order_direction="asc")
+ all_ids = [u.id for u in all_results]
+ assert ids == all_ids[:5]
+ assert total == len(all_ids)
+ results, total = UserDAO.list(
+ order_column="id", order_direction="asc", page=1, page_size=5
+ )
+ ids = [u.id for u in results]
+ assert ids == all_ids[5:10]
+ results, total = UserDAO.list(
+ order_column="id", order_direction="asc", page=10, page_size=5
+ )
+ assert results == []
+
+
+def test_base_dao_list_search(user_with_data: Session) -> None:
+ user_with_data.add_all(
+ [
+ User(
+ id=400,
+ username="searchuser1",
+ first_name="Alice",
+ last_name="Wonderland",
+ email="alice@example.com",
+ active=True,
+ ),
+ User(
+ id=401,
+ username="searchuser2",
+ first_name="Bob",
+ last_name="Builder",
+ email="bob@example.com",
+ active=True,
+ ),
+ User(
+ id=402,
+ username="searchuser3",
+ first_name="Charlie",
+ last_name="Chocolate",
+ email="charlie@example.com",
+ active=True,
+ ),
+ ]
+ )
+ user_with_data.commit()
+ results, _ = UserDAO.list(search="Alice", search_columns=["first_name"])
+ assert any(u.first_name == "Alice" for u in results)
+ results, _ = UserDAO.list(search="Builder", search_columns=["last_name"])
+ assert any(u.last_name == "Builder" for u in results)
+ results, _ = UserDAO.list(search="ar", search_columns=["first_name", "last_name"])
+ names = [(u.first_name, u.last_name) for u in results]
+ assert ("Charlie", "Chocolate") in names or ("Bob", "Builder") in names
+
+
+def test_base_dao_list_custom_filter(user_with_data: Session) -> None:
+ datamodel = SQLAInterface(User, user_with_data)
+
+ class EmailDomainFilter(BaseFilter):
+ def apply(self, query, value):
+ return query.filter(User.email.like(f"%@{value}"))
+
+ user_with_data.add_all(
+ [
+ User(
+ id=500,
+ username="customuser1",
+ first_name="Dom",
+ last_name="Ain",
+ email="dom@domain.com",
+ active=True,
+ ),
+ User(
+ id=501,
+ username="customuser2",
+ first_name="Jane",
+ last_name="Doe",
+ email="jane@other.com",
+ active=True,
+ ),
+ ]
+ )
+ user_with_data.commit()
+ results, _ = UserDAO.list(
+ column_operators=None,
+ custom_filters={"email_domain": EmailDomainFilter("email_domain", datamodel)},
+ )
+ assert all(u.email.endswith("@domain.com") for u in results)
+
+
+def test_base_dao_list_base_filter(user_with_data: Session) -> None:
+ class OnlyActiveFilter(BaseFilter):
+ def apply(self, query, value):
+ return query.filter(User.active)
+
+ class ActiveUserDAO(UserDAO):
+ base_filter = OnlyActiveFilter
+
+ user_with_data.add_all(
+ [
+ User(
+ id=600,
+ username="inactiveuser",
+ first_name="Inactive",
+ last_name="User",
+ email="inactive@example.com",
+ active=False,
+ ),
+ User(
+ id=601,
+ username="activeuser",
+ first_name="Active",
+ last_name="User",
+ email="active@example.com",
+ active=True,
+ ),
+ ]
+ )
+ user_with_data.commit()
+ results, _ = ActiveUserDAO.list()
+ assert all(u.active for u in results)
+
+
+def test_base_dao_list_edge_cases(user_with_data: Session) -> None:
+ user_with_data.add_all(
+ [
+ User(
+ id=700,
+ username="booluser1",
+ first_name="Bool",
+ last_name="User",
+ email="bool1@example.com",
+ active=True,
+ ),
+ User(
+ id=701,
+ username="booluser2",
+ first_name="Bool",
+ last_name="User",
+ email="bool2@example.com",
+ active=False,
+ ),
+ ]
+ )
+ user_with_data.commit()
+ results, _ = UserDAO.list(
+ column_operators=[
+ ColumnOperator(col="active", opr=ColumnOperatorEnum.eq, value=True)
+ ]
+ )
+ assert all(u.active for u in results)
+ results, _ = UserDAO.list(
+ column_operators=[
+ ColumnOperator(col="active", opr=ColumnOperatorEnum.eq, value=False)
+ ]
+ )
+ assert all(not u.active for u in results)
+ results, _ = UserDAO.list(
+ column_operators=[
+ ColumnOperator(col="first_name", opr=ColumnOperatorEnum.eq, value=None)
+ ]
+ )
+ ids = [700, 701]
+ results, _ = UserDAO.list(
+ column_operators=[
+ ColumnOperator(col="id", opr=ColumnOperatorEnum.in_, value=ids)
+ ]
+ )
+ result_ids = [u.id for u in results]
+ assert set(result_ids) == set(ids)
+ results, _ = UserDAO.list(page=100, page_size=10)
+ assert results == []
+
+
+def test_base_dao_list_with_select_columns(user_with_data: Session) -> None:
+ # Add a user to ensure at least one exists
+ user_with_data.add(
+ User(
+ id=900,
+ username="coluser",
+ first_name="Col",
+ last_name="User",
+ email="coluser@example.com",
+ active=True,
+ )
+ )
+ user_with_data.commit()
+ # Request only username and email columns
+ results, total = UserDAO.list(columns=["username", "email"])
+ assert total >= 1
+ # Should return Row objects with correct values
+ found = False
+ for row in results:
+ # SQLAlchemy Row supports both index and key access
+ if row[0] == "coluser" and row[1] == "coluser@example.com":
+ found = True
+ assert found
+ # Request only id column
+ results, total = UserDAO.list(columns=["id"])
+ found = False
+ for row in results:
+ if row[0] == 900:
+ found = True
+ assert found
+
+
+def test_base_dao_list_with_default_columns(user_with_data: Session) -> None:
+ user_with_data.add(
+ User(
+ id=901,
+ username="defaultuser",
+ first_name="Default",
+ last_name="User",
+ email="defaultuser@example.com",
+ active=True,
+ )
+ )
+ user_with_data.commit()
+ results, total = UserDAO.list()
+ assert total >= 1
+ # Should return model instances
+ found = False
+ for user in results:
+ if hasattr(user, "id") and hasattr(user, "username") and hasattr(user, "email"):
+ if (
+ user.id == 901
+ and user.username == "defaultuser"
+ and user.email == "defaultuser@example.com"
+ ):
+ found = True
+ assert found
+
+
+# =============================================================================
+# Advanced Features Tests - Filtering, Operators, Performance
+# =============================================================================
+
+
+def test_base_dao_apply_column_operators_invalid_column(app_context: None) -> None:
+ """Test apply_column_operators with invalid column."""
+ # Create a query
+ query = db.session.query(User)
+
+ # Apply operator with invalid column
+ with patch("superset.daos.base.logging.error") as mock_log:
+ column_operators = [
+ ColumnOperator(
+ col="non_existent_column", opr=ColumnOperatorEnum.eq, value="test"
+ )
+ ]
+
+ # Should raise ValueError
+ with pytest.raises(ValueError, match="Invalid filter") as exc_info:
+ UserDAO.apply_column_operators(query, column_operators)
+
+ assert "Invalid filter" in str(exc_info.value)
+ mock_log.assert_called_once()
+
+
+def test_base_dao_apply_column_operators_none_value(app_context: None) -> None:
+ """Test apply_column_operators with None col value."""
+ query = db.session.query(User)
+
+ # Operator with None column name
+ column_operators = [
+ ColumnOperator(
+ col="", # Empty string instead of None
+ opr=ColumnOperatorEnum.eq,
+ value="test",
+ )
+ ]
+
+ # Should raise ValueError
+ with pytest.raises(ValueError, match="Invalid filter") as exc_info:
+ UserDAO.apply_column_operators(query, column_operators)
+
+ assert "Invalid filter" in str(exc_info.value)
+
+
+def test_base_dao_list_with_invalid_operator(app_context: None) -> None:
+ """Test list method with invalid column operator type."""
+ # Pass a non-ColumnOperator object
+ invalid_operators = [{"col": "username", "opr": "eq", "value": "test"}]
+
+ # Should handle gracefully (skip invalid operators)
+ results, total = UserDAO.list(column_operators=invalid_operators) # type: ignore
+
+ # Should return results without applying invalid filter
+ assert isinstance(results, list)
+ assert isinstance(total, int)
+
+
+def test_base_dao_get_filterable_columns(app_context: None) -> None:
+ """Test get_filterable_columns_and_operators method."""
+ # This is typically implemented by FAB, but we can test the interface
+ filterable = UserDAO.get_filterable_columns_and_operators()
+
+ # Should return a dict
+ assert isinstance(filterable, dict)
+
+ # Common user columns should be filterable
+ if filterable: # Only test if FAB integration is available
+ common_columns = ["username", "email", "active", "first_name", "last_name"]
+ for col in common_columns:
+ if col in filterable:
+ # Should have operator information
+ assert isinstance(filterable[col], (list, tuple, set))
+
+
+def test_base_dao_count_with_filters(app_context: None) -> None:
+ """Test count method with various filters."""
+ # Create test users
+ users = []
+ for i in range(5):
+ user = User(
+ id=2000 + i,
+ username=f"count_user_{i}",
+ first_name="Count",
+ last_name=f"User{i}",
+ email=f"count{i}@example.com",
+ active=(i % 2 == 0), # Alternate active/inactive
+ )
+ users.append(user)
+ db.session.add(user)
+ db.session.commit()
+
+ # Count all
+ total_count = UserDAO.count()
+ assert total_count >= 5
+
+ # Count with filter
+ active_count = UserDAO.count(
+ column_operators=[
+ ColumnOperator(col="active", opr=ColumnOperatorEnum.eq, value=True)
+ ]
+ )
+ assert active_count >= 3 # We created 3 active users (0, 2, 4)
+
+ # Count with multiple filters
+ specific_count = UserDAO.count(
+ column_operators=[
+ ColumnOperator(col="first_name", opr=ColumnOperatorEnum.eq, value="Count"),
+ ColumnOperator(col="active", opr=ColumnOperatorEnum.eq, value=False),
+ ]
+ )
+ assert specific_count >= 2 # We created 2 inactive Count users (1, 3)
+
+ # Cleanup
+ for user in users:
+ db.session.delete(user)
+ db.session.commit()
+
+
+# =============================================================================
+# Edge Cases and Special Scenarios
+# =============================================================================
+
+
+def test_find_by_ids_preserves_order(app_context: None) -> None:
+ """Test that find_by_ids returns results correctly."""
+ # Create test users
+ users = []
+ for i in [1300, 1301, 1302, 1303]:
+ user = User(
+ id=i,
+ username=f"order_user_{i}",
+ first_name="Order",
+ last_name=f"User{i}",
+ email=f"order{i}@example.com",
+ active=True,
+ )
+ users.append(user)
+ db.session.add(user)
+ db.session.commit()
+
+ # Request in specific order
+ requested_ids = [1302, 1300, 1303, 1301]
+ found_users = UserDAO.find_by_ids(requested_ids)
+
+ # Note: find_by_ids doesn't guarantee order preservation
+ # but should return all requested items
+ found_ids = [u.id for u in found_users]
+ assert set(found_ids) == set(requested_ids)
+
+ # Cleanup
+ for user in users:
+ db.session.delete(user)
+ db.session.commit()
+
+
+def test_find_by_ids_with_mixed_types(app_context: None) -> None:
+ """Test find_by_ids with integer IDs."""
+ # Create test data
+ database = Database(
+ id=100,
+ database_name="Test DB 100",
+ sqlalchemy_uri="sqlite:///:memory:",
+ )
+ db.session.add(database)
+ db.session.commit()
+
+ # Should handle integer IDs (skip base filter)
+ found_dbs = DatabaseDAO.find_by_ids([100, 100], skip_base_filter=True)
+ # Should find the database
+ assert len(found_dbs) >= 1
+ assert any(d.id == 100 for d in found_dbs)
+
+ # Cleanup
+ db.session.delete(database)
+ db.session.commit()
+
+
+def test_find_by_column_helper_method(app_context: None) -> None:
+ """Test the internal _find_by_column helper method."""
+ # Create test user
+ user = User(
+ id=1400,
+ username="helper_test_user",
+ first_name="Helper",
+ last_name="Test",
+ email="helper@example.com",
+ active=True,
+ )
+ db.session.add(user)
+ db.session.commit()
+
+ # Test the helper method directly
+ found_user = UserDAO._find_by_column("username", "helper_test_user")
+ assert found_user is not None
+ assert found_user.id == 1400
+
+ # Test with non-existent value
+ not_found = UserDAO._find_by_column("username", "non_existent")
+ assert not_found is None
+
+ # Test with invalid column
+ not_found = UserDAO._find_by_column("invalid_column", "value")
+ assert not_found is None
+
+ # Cleanup
+ db.session.delete(user)
+ db.session.commit()
+
+
+def test_find_methods_with_special_characters(app_context: None) -> None:
+ """Test find methods with values containing special characters."""
+ # Create dashboard with special characters in slug
+ special_slug = "test-dashboard-with-special-@#$"
+ dashboard = Dashboard(
+ dashboard_title="Special Dashboard",
+ slug=special_slug,
+ published=True,
+ )
+ db.session.add(dashboard)
+ db.session.commit()
+
+ # Find by slug with special characters (skip base filter)
+ found = DashboardDAO.find_by_id(
+ special_slug, id_column="slug", skip_base_filter=True
+ )
+ assert found is not None
+ assert found.slug == special_slug
+
+ # Test find_by_ids with special characters (skip base filter)
+ found_list = DashboardDAO.find_by_ids(
+ [special_slug], id_column="slug", skip_base_filter=True
+ )
+ assert len(found_list) == 1
+ assert found_list[0].slug == special_slug
+
+ # Cleanup
+ db.session.delete(dashboard)
+ db.session.commit()
+
+
+def test_find_methods_case_sensitivity(app_context: None) -> None:
+ """Test find methods with case-sensitive columns."""
+ # Create users with similar usernames differing in case
+ user1 = User(
+ id=1500,
+ username="CaseSensitive",
+ first_name="Case",
+ last_name="Sensitive",
+ email="case1@example.com",
+ active=True,
+ )
+ user2 = User(
+ id=1501,
+ username="casesensitive",
+ first_name="Case",
+ last_name="Sensitive",
+ email="case2@example.com",
+ active=True,
+ )
+ db.session.add_all([user1, user2])
+
+ try:
+ db.session.commit()
+
+ # Find by exact case
+ found = UserDAO.find_by_id("CaseSensitive", id_column="username")
+ assert found is not None
+ assert found.id == 1500
+
+ found = UserDAO.find_by_id("casesensitive", id_column="username")
+ assert found is not None
+ assert found.id == 1501
+
+ # Cleanup
+ db.session.delete(user1)
+ db.session.delete(user2)
+ db.session.commit()
+ except Exception:
+ # Some databases may have case-insensitive unique constraints
+ db.session.rollback()
+ pytest.skip("Database has case-insensitive unique constraint on username")
+
+
+def test_find_by_ids_empty_and_none_handling(app_context: None) -> None:
+ """Test find_by_ids with empty lists."""
+ # Test with empty list
+ result = UserDAO.find_by_ids([])
+ assert result == []
+
+ # Create test user
+ user = User(
+ id=1600,
+ username="none_test_user",
+ first_name="None",
+ last_name="Test",
+ email="none@example.com",
+ active=True,
+ )
+ db.session.add(user)
+ db.session.commit()
+
+ # Find with valid IDs only
+ found = UserDAO.find_by_ids([1600, 1601])
+ assert len(found) == 1
+ assert found[0].id == 1600
+
+ # Cleanup
+ db.session.delete(user)
+ db.session.commit()
+
+
+def test_find_methods_performance_with_large_lists(app_context: None) -> None:
+ """Test find_by_ids performance with large ID lists."""
+ # Create a batch of users
+ users = []
+ user_ids = []
+ for i in range(1700, 1750): # 50 users
+ user_ids.append(i)
+ user = User(
+ id=i,
+ username=f"perf_user_{i}",
+ first_name="Perf",
+ last_name=f"User{i}",
+ email=f"perf{i}@example.com",
+ active=True,
+ )
+ users.append(user)
+ db.session.add(user)
+ db.session.commit()
+
+ # Find all at once
+ start_time = time.time()
+ found_users = UserDAO.find_by_ids(user_ids)
+ end_time = time.time()
+
+ assert len(found_users) == 50
+ assert (end_time - start_time) < 1.0 # Should complete within 1 second
+
+ # Cleanup
+ for user in users:
+ db.session.delete(user)
+ db.session.commit()
+
+
+# =============================================================================
+# DAO Properties and Configuration Tests
+# =============================================================================
+
+
+def test_base_dao_model_cls_property(app_context: None) -> None:
+ """Test that model_cls property is properly set."""
+ assert UserDAO.model_cls == User
+
+ # Test with a DAO that doesn't set model_cls
+ class InvalidDAO(BaseDAO[User]):
+ pass
+
+ # Should have model_cls attribute set by __init_subclass__
+ # Just verify it exists and is the expected type
+ assert hasattr(InvalidDAO, "model_cls")
+ assert InvalidDAO.model_cls == User # It inherits the generic type
+
+
+def test_base_dao_id_column_name_property(app_context: None) -> None:
+ """Test that id_column_name property is properly used."""
+ assert UserDAO.id_column_name == "id"
+
+ # Test custom id column name
+ class CustomIdDAO(BaseDAO[User]):
+ model_cls = User
+ id_column_name = "username" # Use username as ID column
+
+ # Create test user
+ user = User(
+ id=5016,
+ username="unique-username-id",
+ first_name="Custom",
+ last_name="User",
+ email="custom@example.com",
+ active=True,
+ )
+ db.session.add(user)
+ db.session.commit()
+
+ # Find by custom ID column
+ found = CustomIdDAO.find_by_id("unique-username-id")
+ assert found is not None
+ assert found.first_name == "Custom"
+
+ # Cleanup
+ db.session.delete(user)
+ db.session.commit()
+
+
+def test_base_dao_base_filter_integration(app_context: None) -> None:
+ """Test base_filter integration with list operations."""
+ # Create mix of users
+ users = []
+ for i, active in enumerate([True, False, True, False, True]):
+ user = User(
+ id=2100 + i,
+ username=f"base_filter_user_{i}",
+ first_name="BaseFilter",
+ last_name=f"User{i}",
+ email=f"basefilter{i}@example.com",
+ active=active,
+ )
+ users.append(user)
+ db.session.add(user)
+ db.session.commit()
+
+ # Test basic list functionality
+ results, total = UserDAO.list()
+ assert isinstance(results, list)
+ assert isinstance(total, int)
+ assert total >= 5 # Should include our 5 test users
+
+ # Test skip_base_filter functionality
+ results_skip, total_skip = UserDAO.list()
+ assert isinstance(results_skip, list)
+ assert isinstance(total_skip, int)
+
+ # Test count functionality
+ count_all = UserDAO.count()
+ assert count_all >= 5
+
+ count_skip = UserDAO.count(skip_base_filter=True)
+ assert count_skip >= 5
+
+ # Cleanup
+ for user in users:
+ db.session.delete(user)
+ db.session.commit()
+
+
+def test_base_dao_edge_cases(app_context: None) -> None:
+ """Test BaseDAO edge cases and error conditions."""
+ # Test create without item or attributes
+ created = UserDAO.create()
+ assert created is not None
+ # User model has required fields, so we expect them to be None
+ assert created.username is None
+
+ # Don't commit - would fail due to constraints
+ db.session.rollback()
Review Comment:
Crunching...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] Feat/dao updates [superset]
Posted by "aminghadersohi (via GitHub)" <gi...@apache.org>.
aminghadersohi commented on code in PR #35018:
URL: https://github.com/apache/superset/pull/35018#discussion_r2328826230
##########
superset/daos/base.py:
##########
@@ -251,3 +411,247 @@ def filter_by(cls, **filter_by: Any) -> list[T]:
cls.id_column_name, data_model
).apply(query, None)
return query.filter_by(**filter_by).all()
+
+ @classmethod
+ def apply_column_operators(
+ cls, query: Any, column_operators: Optional[List[ColumnOperator]] = None
+ ) -> Any:
+ """
+ Apply column operators (list of ColumnOperator) to the query using
+ ColumnOperatorEnum logic. Raises ValueError if a filter references a
+ non-existent column.
+ """
+ if not column_operators:
+ return query
+ for c in column_operators:
+ if not isinstance(c, ColumnOperator):
+ continue
+ col = c.col
+ opr = c.opr
+ value = c.value
+ if not col or not hasattr(cls.model_cls, col):
+ model_name = cls.model_cls.__name__ if cls.model_cls else "Unknown"
+ logging.error(
+ f"Invalid filter: column '{col}' does not exist on {model_name}"
+ )
+ raise ValueError(
+ f"Invalid filter: column '{col}' does not exist on {model_name}"
+ )
+ column = getattr(cls.model_cls, col)
+ try:
+ # Always use ColumnOperatorEnum's apply method
+ operator_enum = ColumnOperatorEnum(opr)
+ query = query.filter(operator_enum.apply(column, value))
+ except Exception as e:
+ logging.error(f"Error applying filter on column '{col}': {e}")
+ raise
+ return query
+
+ @classmethod
+ def get_filterable_columns_and_operators(cls) -> Dict[str, List[str]]:
+ """
+ Returns a dict mapping filterable columns (including hybrid/computed fields if
+ present) to their supported operators. Used by MCP tools to dynamically expose
+ filter options. Custom fields supported by the DAO but not present on the model
+ should be documented here.
+ """
+ from sqlalchemy.ext.hybrid import hybrid_property
+
+ mapper = inspect(cls.model_cls)
+ columns = {c.key: c for c in mapper.columns}
+ # Add hybrid properties
+ hybrids = {
+ name: attr
+ for name, attr in vars(cls.model_cls).items()
+ if isinstance(attr, hybrid_property)
+ }
+ # You may add custom fields here, e.g.:
+ # custom_fields = {"tags": ["eq", "in_", "like"], ...}
+ custom_fields: Dict[str, List[str]] = {}
+ # Map SQLAlchemy types to supported operators
+ type_operator_map = {
Review Comment:
you're right. better defined once at module load time rather than being recreated every time the get_filterable_columns_and_operators method is called.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] Feat/dao updates [superset]
Posted by "mistercrunch (via GitHub)" <gi...@apache.org>.
mistercrunch commented on code in PR #35018:
URL: https://github.com/apache/superset/pull/35018#discussion_r2326354212
##########
superset/daos/base.py:
##########
@@ -251,3 +411,247 @@ def filter_by(cls, **filter_by: Any) -> list[T]:
cls.id_column_name, data_model
).apply(query, None)
return query.filter_by(**filter_by).all()
+
+ @classmethod
+ def apply_column_operators(
+ cls, query: Any, column_operators: Optional[List[ColumnOperator]] = None
+ ) -> Any:
+ """
+ Apply column operators (list of ColumnOperator) to the query using
+ ColumnOperatorEnum logic. Raises ValueError if a filter references a
+ non-existent column.
+ """
+ if not column_operators:
+ return query
+ for c in column_operators:
+ if not isinstance(c, ColumnOperator):
+ continue
+ col = c.col
+ opr = c.opr
+ value = c.value
+ if not col or not hasattr(cls.model_cls, col):
+ model_name = cls.model_cls.__name__ if cls.model_cls else "Unknown"
+ logging.error(
+ f"Invalid filter: column '{col}' does not exist on {model_name}"
+ )
+ raise ValueError(
+ f"Invalid filter: column '{col}' does not exist on {model_name}"
+ )
+ column = getattr(cls.model_cls, col)
+ try:
+ # Always use ColumnOperatorEnum's apply method
+ operator_enum = ColumnOperatorEnum(opr)
+ query = query.filter(operator_enum.apply(column, value))
+ except Exception as e:
+ logging.error(f"Error applying filter on column '{col}': {e}")
+ raise
+ return query
+
+ @classmethod
+ def get_filterable_columns_and_operators(cls) -> Dict[str, List[str]]:
+ """
+ Returns a dict mapping filterable columns (including hybrid/computed fields if
+ present) to their supported operators. Used by MCP tools to dynamically expose
+ filter options. Custom fields supported by the DAO but not present on the model
+ should be documented here.
+ """
+ from sqlalchemy.ext.hybrid import hybrid_property
Review Comment:
NIT: i think top-of-module imports are generally better, except for some exceptions, I keep catching claude doing this as I think it means fewer edits for it. [I think] exception are around potential circular deps or importing heavy modules
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] Feat/dao updates [superset]
Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #35018:
URL: https://github.com/apache/superset/pull/35018#issuecomment-3264144568
## [Codecov](https://app.codecov.io/gh/apache/superset/pull/35018?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `38.35616%` with `135 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 51.00%. Comparing base ([`0fce5ec`](https://app.codecov.io/gh/apache/superset/commit/0fce5ecfa5295f9978f873f2930ca972b868cfe5?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`80989a1`](https://app.codecov.io/gh/apache/superset/commit/80989a15359d6b798327f6280c06ee42bc4c7a48?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
| [Files with missing lines](https://app.codecov.io/gh/apache/superset/pull/35018?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [superset/daos/base.py](https://app.codecov.io/gh/apache/superset/pull/35018?src=pr&el=tree&filepath=superset%2Fdaos%2Fbase.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c3VwZXJzZXQvZGFvcy9iYXNlLnB5) | 36.36% | [121 Missing and 5 partials :warning: ](https://app.codecov.io/gh/apache/superset/pull/35018?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [superset/daos/chart.py](https://app.codecov.io/gh/apache/superset/pull/35018?src=pr&el=tree&filepath=superset%2Fdaos%2Fchart.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c3VwZXJzZXQvZGFvcy9jaGFydC5weQ==) | 57.14% | [3 Missing :warning: ](https://app.codecov.io/gh/apache/superset/pull/35018?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [superset/daos/dashboard.py](https://app.codecov.io/gh/apache/superset/pull/35018?src=pr&el=tree&filepath=superset%2Fdaos%2Fdashboard.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c3VwZXJzZXQvZGFvcy9kYXNoYm9hcmQucHk=) | 57.14% | [3 Missing :warning: ](https://app.codecov.io/gh/apache/superset/pull/35018?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [superset/daos/dataset.py](https://app.codecov.io/gh/apache/superset/pull/35018?src=pr&el=tree&filepath=superset%2Fdaos%2Fdataset.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c3VwZXJzZXQvZGFvcy9kYXRhc2V0LnB5) | 57.14% | [3 Missing :warning: ](https://app.codecov.io/gh/apache/superset/pull/35018?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #35018 +/- ##
===========================================
+ Coverage 0 51.00% +51.00%
===========================================
Files 0 586 +586
Lines 0 43043 +43043
Branches 0 4574 +4574
===========================================
+ Hits 0 21956 +21956
- Misses 0 20478 +20478
- Partials 0 609 +609
```
| [Flag](https://app.codecov.io/gh/apache/superset/pull/35018/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [hive](https://app.codecov.io/gh/apache/superset/pull/35018/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `46.60% <33.33%> (?)` | |
| [presto](https://app.codecov.io/gh/apache/superset/pull/35018/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `50.26% <38.35%> (?)` | |
| [python](https://app.codecov.io/gh/apache/superset/pull/35018/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `51.00% <38.35%> (?)` | |
| [unit](https://app.codecov.io/gh/apache/superset/pull/35018/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `100.00% <ø> (?)` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
</details>
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/superset/pull/35018?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat: Add BaseDAO improvements and test reorganization [superset]
Posted by "aminghadersohi (via GitHub)" <gi...@apache.org>.
aminghadersohi commented on code in PR #35018:
URL: https://github.com/apache/superset/pull/35018#discussion_r2334077828
##########
requirements/development.in:
##########
@@ -16,5 +16,5 @@
# specific language governing permissions and limitations
# under the License.
#
--e .[development,bigquery,druid,duckdb,gevent,gsheets,mysql,postgres,presto,prophet,trino,thumbnails]
+-e .[development,bigquery,druid,duckdb,gevent,gsheets,mysql,postgres,presto,prophet,pydantic,trino,thumbnails]
Review Comment:
ah yeah that makes sense. pydantic gets pulled as a part of fastmcp. This PR doesn't have fastMCP dep requirement but it does add pydantic usage to base dao. It wasn't really possible to keep using marshmallow because the mcp service uses the pydantic models directly and/or extends them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat: Add BaseDAO improvements and test reorganization [superset]
Posted by "aminghadersohi (via GitHub)" <gi...@apache.org>.
aminghadersohi commented on code in PR #35018:
URL: https://github.com/apache/superset/pull/35018#discussion_r2334800565
##########
tests/unit_tests/dao/base_dao_test.py:
##########
@@ -37,6 +44,103 @@ class TestDAOWithNoneModel(BaseDAO[MockModel]):
model_cls = None
+# =============================================================================
+# Unit Tests - These tests use mocks and don't touch the database
+# =============================================================================
+
+
+def test_column_operator_enum_apply_method() -> None: # noqa: C901
+ """
+ Test that the apply method works correctly for each operator.
+ This verifies the actual SQL generation for each operator.
+ """
+ Base_test = declarative_base() # noqa: N806
+
+ class TestModel(Base_test): # type: ignore
+ __tablename__ = "test_model"
+ id = Column(Integer, primary_key=True)
+ name = Column(String(50))
+ age = Column(Integer)
+ active = Column(Boolean)
+
+ # Test each operator's apply method
+ test_cases = [
+ # (operator, column, value, expected_sql_fragment)
+ (ColumnOperatorEnum.eq, TestModel.name, "test", "name = 'test'"),
+ (ColumnOperatorEnum.ne, TestModel.name, "test", "name != 'test'"),
+ (ColumnOperatorEnum.sw, TestModel.name, "test", "name LIKE 'test%'"),
+ (ColumnOperatorEnum.ew, TestModel.name, "test", "name LIKE '%test'"),
+ (ColumnOperatorEnum.in_, TestModel.id, [1, 2, 3], "id IN (1, 2, 3)"),
+ (ColumnOperatorEnum.nin, TestModel.id, [1, 2, 3], "id NOT IN (1, 2, 3)"),
+ (ColumnOperatorEnum.gt, TestModel.age, 25, "age > 25"),
+ (ColumnOperatorEnum.gte, TestModel.age, 25, "age >= 25"),
+ (ColumnOperatorEnum.lt, TestModel.age, 25, "age < 25"),
+ (ColumnOperatorEnum.lte, TestModel.age, 25, "age <= 25"),
+ (ColumnOperatorEnum.like, TestModel.name, "test", "name LIKE '%test%'"),
+ (ColumnOperatorEnum.ilike, TestModel.name, "test", "name ILIKE '%test%'"),
+ (ColumnOperatorEnum.is_null, TestModel.name, None, "name IS NULL"),
+ (ColumnOperatorEnum.is_not_null, TestModel.name, None, "name IS NOT NULL"),
+ ]
+
+ for operator, column, value, expected_sql_fragment in test_cases:
+ # Apply the operator
+ result = operator.apply(column, value)
+
+ # Convert to string to check SQL generation
+ sql_str = str(result.compile(compile_kwargs={"literal_binds": True}))
+
+ # Verify the SQL contains the expected fragment
+ # Note: SQLAlchemy might format this differently, so we check for key parts
+ if "=" in expected_sql_fragment:
+ assert "=" in sql_str
+ elif "!=" in expected_sql_fragment:
+ assert "!=" in sql_str
+ elif "LIKE" in expected_sql_fragment:
+ assert "LIKE" in sql_str
+ elif "ILIKE" in expected_sql_fragment:
+ assert "ILIKE" in sql_str
+ elif "IN" in expected_sql_fragment:
+ assert "IN" in sql_str
+ elif "NOT IN" in expected_sql_fragment:
+ assert "NOT IN" in sql_str
+ elif ">" in expected_sql_fragment:
+ assert ">" in sql_str
+ elif ">=" in expected_sql_fragment:
+ assert ">=" in sql_str
+ elif "<" in expected_sql_fragment:
+ assert "<" in sql_str
+ elif "<=" in expected_sql_fragment:
+ assert "<=" in sql_str
+ elif "IS NULL" in expected_sql_fragment:
+ assert "IS NULL" in sql_str
+ elif "IS NOT NULL" in expected_sql_fragment:
+ assert "IS NOT NULL" in sql_str
Review Comment:
You're right - these assertions are way too weak. They just check if
certain characters exist anywhere in the SQL string, which could lead
to false positives. Let me rewrite this test to properly validate the
SQL generation
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat: Add BaseDAO improvements and test reorganization [superset]
Posted by "aminghadersohi (via GitHub)" <gi...@apache.org>.
aminghadersohi commented on PR #35018:
URL: https://github.com/apache/superset/pull/35018#issuecomment-3293389419
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat: Add BaseDAO improvements and test reorganization [superset]
Posted by "aminghadersohi (via GitHub)" <gi...@apache.org>.
aminghadersohi commented on PR #35018:
URL: https://github.com/apache/superset/pull/35018#issuecomment-3300178425
/resolve
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] Feat/dao updates [superset]
Posted by "aminghadersohi (via GitHub)" <gi...@apache.org>.
aminghadersohi commented on code in PR #35018:
URL: https://github.com/apache/superset/pull/35018#discussion_r2328819759
##########
superset/daos/base.py:
##########
@@ -251,3 +411,247 @@ def filter_by(cls, **filter_by: Any) -> list[T]:
cls.id_column_name, data_model
).apply(query, None)
return query.filter_by(**filter_by).all()
+
+ @classmethod
+ def apply_column_operators(
+ cls, query: Any, column_operators: Optional[List[ColumnOperator]] = None
+ ) -> Any:
+ """
+ Apply column operators (list of ColumnOperator) to the query using
+ ColumnOperatorEnum logic. Raises ValueError if a filter references a
+ non-existent column.
+ """
+ if not column_operators:
+ return query
+ for c in column_operators:
+ if not isinstance(c, ColumnOperator):
+ continue
+ col = c.col
+ opr = c.opr
+ value = c.value
+ if not col or not hasattr(cls.model_cls, col):
+ model_name = cls.model_cls.__name__ if cls.model_cls else "Unknown"
+ logging.error(
+ f"Invalid filter: column '{col}' does not exist on {model_name}"
+ )
+ raise ValueError(
+ f"Invalid filter: column '{col}' does not exist on {model_name}"
+ )
+ column = getattr(cls.model_cls, col)
+ try:
+ # Always use ColumnOperatorEnum's apply method
+ operator_enum = ColumnOperatorEnum(opr)
+ query = query.filter(operator_enum.apply(column, value))
+ except Exception as e:
+ logging.error(f"Error applying filter on column '{col}': {e}")
+ raise
+ return query
+
+ @classmethod
+ def get_filterable_columns_and_operators(cls) -> Dict[str, List[str]]:
+ """
+ Returns a dict mapping filterable columns (including hybrid/computed fields if
+ present) to their supported operators. Used by MCP tools to dynamically expose
+ filter options. Custom fields supported by the DAO but not present on the model
+ should be documented here.
+ """
+ from sqlalchemy.ext.hybrid import hybrid_property
Review Comment:
you're absolutely right. I
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat: Add BaseDAO improvements and test reorganization [superset]
Posted by "mistercrunch (via GitHub)" <gi...@apache.org>.
mistercrunch commented on code in PR #35018:
URL: https://github.com/apache/superset/pull/35018#discussion_r2335661088
##########
superset/daos/base.py:
##########
@@ -251,3 +455,205 @@ def filter_by(cls, **filter_by: Any) -> list[T]:
cls.id_column_name, data_model
).apply(query, None)
return query.filter_by(**filter_by).all()
+
+ @classmethod
+ def apply_column_operators(
+ cls, query: Any, column_operators: Optional[List[ColumnOperator]] = None
+ ) -> Any:
+ """
+ Apply column operators (list of ColumnOperator) to the query using
+ ColumnOperatorEnum logic. Raises ValueError if a filter references a
+ non-existent column.
+ """
+ if not column_operators:
+ return query
+ for c in column_operators:
+ if not isinstance(c, ColumnOperator):
+ continue
+ col = c.col
+ opr = c.opr
+ value = c.value
+ if not col or not hasattr(cls.model_cls, col):
+ model_name = cls.model_cls.__name__ if cls.model_cls else "Unknown"
+ logging.error(
+ f"Invalid filter: column '{col}' does not exist on {model_name}"
Review Comment:
https://github.com/apache/superset/pull/35081
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat: Add BaseDAO improvements and test reorganization [superset]
Posted by "mistercrunch (via GitHub)" <gi...@apache.org>.
mistercrunch commented on code in PR #35018:
URL: https://github.com/apache/superset/pull/35018#discussion_r2334102906
##########
requirements/development.in:
##########
@@ -16,5 +16,5 @@
# specific language governing permissions and limitations
# under the License.
#
--e .[development,bigquery,druid,duckdb,gevent,gsheets,mysql,postgres,presto,prophet,trino,thumbnails]
+-e .[development,bigquery,druid,duckdb,gevent,gsheets,mysql,postgres,presto,prophet,pydantic,trino,thumbnails]
Review Comment:
it doesn't need to be referenced in this particular file as it's referenced in pyproject.toml core dependencies.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat: Add BaseDAO improvements and test reorganization [superset]
Posted by "mistercrunch (via GitHub)" <gi...@apache.org>.
mistercrunch commented on code in PR #35018:
URL: https://github.com/apache/superset/pull/35018#discussion_r2335181915
##########
superset/daos/base.py:
##########
@@ -251,3 +455,205 @@ def filter_by(cls, **filter_by: Any) -> list[T]:
cls.id_column_name, data_model
).apply(query, None)
return query.filter_by(**filter_by).all()
+
+ @classmethod
+ def apply_column_operators(
+ cls, query: Any, column_operators: Optional[List[ColumnOperator]] = None
+ ) -> Any:
+ """
+ Apply column operators (list of ColumnOperator) to the query using
+ ColumnOperatorEnum logic. Raises ValueError if a filter references a
+ non-existent column.
+ """
+ if not column_operators:
+ return query
+ for c in column_operators:
+ if not isinstance(c, ColumnOperator):
+ continue
+ col = c.col
+ opr = c.opr
+ value = c.value
+ if not col or not hasattr(cls.model_cls, col):
+ model_name = cls.model_cls.__name__ if cls.model_cls else "Unknown"
+ logging.error(
+ f"Invalid filter: column '{col}' does not exist on {model_name}"
Review Comment:
Oops:
```
Found 190 errors.
[*] 1 fixable with the `--fix` option.
```
alright, I just put 8 subagents on the task :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat: Add BaseDAO improvements and test reorganization [superset]
Posted by "mistercrunch (via GitHub)" <gi...@apache.org>.
mistercrunch commented on code in PR #35018:
URL: https://github.com/apache/superset/pull/35018#discussion_r2335177353
##########
superset/daos/base.py:
##########
@@ -251,3 +455,205 @@ def filter_by(cls, **filter_by: Any) -> list[T]:
cls.id_column_name, data_model
).apply(query, None)
return query.filter_by(**filter_by).all()
+
+ @classmethod
+ def apply_column_operators(
+ cls, query: Any, column_operators: Optional[List[ColumnOperator]] = None
+ ) -> Any:
+ """
+ Apply column operators (list of ColumnOperator) to the query using
+ ColumnOperatorEnum logic. Raises ValueError if a filter references a
+ non-existent column.
+ """
+ if not column_operators:
+ return query
+ for c in column_operators:
+ if not isinstance(c, ColumnOperator):
+ continue
+ col = c.col
+ opr = c.opr
+ value = c.value
+ if not col or not hasattr(cls.model_cls, col):
+ model_name = cls.model_cls.__name__ if cls.model_cls else "Unknown"
+ logging.error(
+ f"Invalid filter: column '{col}' does not exist on {model_name}"
Review Comment:
I'll put my best bot on it!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] Feat/dao updates [superset]
Posted by "bito-code-review[bot] (via GitHub)" <gi...@apache.org>.
bito-code-review[bot] commented on PR #35018:
URL: https://github.com/apache/superset/pull/35018#issuecomment-3254617413
**Bito Automatic Review Skipped - Draft PR**<br/><br/>Bito didn't auto-review because this pull request is in draft status.<br/>No action is needed if you didn't intend for the agent to review it. Otherwise, to manually trigger a review, type `/review` in a comment and save.<br/>You can change draft PR review settings [here](https://alpha.bito.ai/home/ai-agents/code-review-agent), or contact your Bito workspace admin at [evan@preset.io](mailto:evan@preset.io).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat: Add BaseDAO improvements and test reorganization [superset]
Posted by "mistercrunch (via GitHub)" <gi...@apache.org>.
mistercrunch merged PR #35018:
URL: https://github.com/apache/superset/pull/35018
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org