You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@sedona.apache.org by "paleolimbot (via GitHub)" <gi...@apache.org> on 2026/05/27 22:41:30 UTC
[PR] feat(python/sedonadb): Expose scalar and aggregate udfs from context registry [sedona-db]
paleolimbot opened a new pull request, #885:
URL: https://github.com/apache/sedona-db/pull/885
(no comment)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(python/sedonadb): Expose scalar and aggregate udfs from context registry [sedona-db]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot merged PR #885:
URL: https://github.com/apache/sedona-db/pull/885
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(python/sedonadb): Expose scalar and aggregate udfs from context registry [sedona-db]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #885:
URL: https://github.com/apache/sedona-db/pull/885#discussion_r3319384731
##########
python/sedonadb/python/sedonadb/functions/__init__.py:
##########
@@ -38,3 +41,35 @@ def table(self) -> "TableFunctions":
from sedonadb.functions.table import TableFunctions
return TableFunctions(self._ctx)
+
+ def __getattr__(self, name) -> Union["ScalarUdf", "AggregateUdf"]:
+ try:
+ return ScalarUdf(self._ctx._impl.scalar_udf(name))
+ except SedonaError:
+ pass
+
+ try:
+ return AggregateUdf(self._ctx._impl.aggregate_udf(name))
+ except SedonaError:
+ pass
+
+ return AttributeError(f"Can't find scalar or aggregate function '{name}'")
+
+ def __getitem__(self, key) -> Union["ScalarUdf", "AggregateUdf"]:
+ try:
+ return self.__getattr__(key)
+ except AttributeError:
+ KeyError(f"Can't find scalar or aggregate function '{key}'")
Review Comment:
`__getattr__` must raise `AttributeError` when a function is missing, not return an `AttributeError` instance. As written, `sd.funcs.not_a_function` evaluates to an exception object instead of failing, which also prevents `__getitem__` from detecting missing functions correctly.
##########
python/sedonadb/python/sedonadb/dataframe.py:
##########
@@ -194,6 +351,21 @@ def select(self, *exprs: Union[Expr, str, _SedonaLit]) -> "DataFrame":
f"select() expects str, Expr, or Literal arguments, "
f"got {type(e).__name__}"
)
+
+ # Process keyword arguments: alias each value with its key
+ for name, e in kwargs.items():
+ if isinstance(e, Expr):
+ coerced.append(e.alias(name)._impl)
+ elif isinstance(e, str):
+ coerced.append(_col(e).alias(name)._impl)
Review Comment:
The string-valued keyword path is introduced here but the added test that appears intended to cover it passes `col("x")` instead of the string `"x"`. Please add or fix a test for `df.select(renamed="x")` so this branch is exercised.
##########
python/sedonadb/python/sedonadb/functions/__init__.py:
##########
@@ -38,3 +41,35 @@ def table(self) -> "TableFunctions":
from sedonadb.functions.table import TableFunctions
return TableFunctions(self._ctx)
+
+ def __getattr__(self, name) -> Union["ScalarUdf", "AggregateUdf"]:
+ try:
+ return ScalarUdf(self._ctx._impl.scalar_udf(name))
+ except SedonaError:
+ pass
+
+ try:
+ return AggregateUdf(self._ctx._impl.aggregate_udf(name))
+ except SedonaError:
+ pass
+
+ return AttributeError(f"Can't find scalar or aggregate function '{name}'")
+
+ def __getitem__(self, key) -> Union["ScalarUdf", "AggregateUdf"]:
+ try:
+ return self.__getattr__(key)
+ except AttributeError:
+ KeyError(f"Can't find scalar or aggregate function '{key}'")
Review Comment:
This constructs a `KeyError` but does not raise it, so after `__getattr__` is fixed, `sd.funcs['missing']` will return `None` instead of raising. Raise the `KeyError` here so item lookup has the expected mapping semantics.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(python/sedonadb): Expose scalar and aggregate udfs from context registry [sedona-db]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #885:
URL: https://github.com/apache/sedona-db/pull/885#discussion_r3319870525
##########
python/sedonadb/python/sedonadb/dataframe.py:
##########
@@ -34,9 +34,112 @@
class DataFrame:
"""Representation of a (lazy) collection of columns
- This object is usually constructed from a
- SedonaContext][sedonadb.context.SedonaContext] by importing an object,
- reading a file, or executing SQL.
+ This object is usually constructed from `sd = sedona.db.connect()`
+ by importing an object with `sd.create_data_frame()`, reading a file
+ with `sd.read_parquet()`/`sd.read_pyogrio()`, or executing SQL with
+ `sd.sql()`. Once created, a DataFrame can be modified using the Python
+ API (e.g., `.select()`, `.filter()`, `.sort()`, `.limit()`) or by
+ creating a temporary view with `.to_view("name")` and querying the
+ resulting view using `sd.sql()`. The Python API aims to provide
+ a minimal subset of functionality derived primarily from Ibis and
+ DuckDB's relational APIs.
+
+ Examples:
+
+ >>> sd = sedona.db.connect()
+ >>> sd.options.interactive = True
+ >>> df = sd.sql("SELECT * FROM (VALUES (1, 'a'), (2, 'b'), (3, 'c')) AS t(x, y)")
+ >>> df.limit(2)
+ ┌───────┬──────┐
+ │ x ┆ y │
+ │ int64 ┆ utf8 │
+ ╞═══════╪══════╡
+ │ 1 ┆ a │
+ ├╌╌╌╌╌╌╌┼╌╌╌╌╌╌┤
+ │ 2 ┆ b │
+ └───────┴──────┘
+
+ Columns can be specified as part of the Python API in several ways:
+
+ >>> df.select("y", z=df.x + 1)
+ ┌──────┬───────┐
+ │ y ┆ z │
+ │ utf8 ┆ int64 │
+ ╞══════╪═══════╡
+ │ a ┆ 2 │
+ ├╌╌╌╌╌╌┼╌╌╌╌╌╌╌┤
+ │ b ┆ 3 │
+ ├╌╌╌╌╌╌┼╌╌╌╌╌╌╌┤
+ │ c ┆ 4 │
+ └──────┴───────┘
+ >>> df.select("y", z=df["x"] + 1)
+ ┌──────┬───────┐
+ │ y ┆ z │
+ │ utf8 ┆ int64 │
+ ╞══════╪═══════╡
+ │ a ┆ 2 │
+ ├╌╌╌╌╌╌┼╌╌╌╌╌╌╌┤
+ │ b ┆ 3 │
+ ├╌╌╌╌╌╌┼╌╌╌╌╌╌╌┤
+ │ c ┆ 4 │
+ └──────┴───────┘
+ >>> df.select("y", z=sd.col("x") + 1)
+ ┌──────┬───────┐
+ │ y ┆ z │
+ │ utf8 ┆ int64 │
+ ╞══════╪═══════╡
+ │ a ┆ 2 │
+ ├╌╌╌╌╌╌┼╌╌╌╌╌╌╌┤
+ │ b ┆ 3 │
+ ├╌╌╌╌╌╌┼╌╌╌╌╌╌╌┤
+ │ c ┆ 4 │
+ └──────┴───────┘
+
+ Literals can be specified explicitly using `sd.lit(obj)` but can also
+ be used directly in function calls or comparisons.
+
+ >>> df.filter(df.x > 1)
+ ┌───────┬──────┐
+ │ x ┆ y │
+ │ int64 ┆ utf8 │
+ ╞═══════╪══════╡
+ │ 2 ┆ b │
+ ├╌╌╌╌╌╌╌┼╌╌╌╌╌╌┤
+ │ 3 ┆ c │
+ └───────┴──────┘
+ >>> df.filter(df.x > sd.lit(1))
+ ┌───────┬──────┐
+ │ x ┆ y │
+ │ int64 ┆ utf8 │
+ ╞═══════╪══════╡
+ │ 2 ┆ b │
+ ├╌╌╌╌╌╌╌┼╌╌╌╌╌╌┤
+ │ 3 ┆ c │
+ └───────┴──────┘
+
+ Supported literals include any Python object supported by pyarrow
+ in addition to GeoPandas, Shapely, and PyProj objects for use in
+ geometry functions.
+
+ Functions are available from `sd.funcs`. Assigning this to a local
+ variable generally leads to better autocomplete but is not necessary.
+ In addition to standard DataFusion scalar and aggregate functions,
+ a number of spatial functions are provided. See the
+ [SedonaDB SQL Reference](https://sedona.apache.org/sedonadb/latest/reference/sql/)
+ for a list of supported functions.
+
+ >>> f = sd.funcs
+ >>> df.select(geometry=f.st_point(df.x, df.x + 1))
+ ┌────────────┐
+ │ geometry │
+ │ geometry │
+ ╞════════════╡
+ │ POINT(1 2) │
+ ├╌╌╌╌╌╌╌╌╌╌╌╌┤
+ │ POINT(2 3) │
+ ├╌╌╌╌╌╌╌╌╌╌╌╌┤
+ │ POINT(3 4) │
+ └────────────┘
"""
Review Comment:
This is the TL;DR of this PR. Also a good bit of text to point AI at so that it does a better job writing Python code.
##########
python/sedonadb/python/sedonadb/context.py:
##########
@@ -486,6 +491,53 @@ def funcs(self) -> Functions:
"""Access Python wrappers for SedonaDB functions"""
return Functions(self)
+ def col(self, name: str, qualifier: Optional[str] = None) -> Expr:
+ """Reference a column by name.
+
+ Args:
+ name: The column name to reference.
+ qualifier: An optional table qualifier (e.g. `"t"` for `t.x`). Useful
Review Comment:
I added `col()` and `lit()` to `sd.col()` and `sd.lit()` because otherwise they are tricky to include without an Apache Sedona Python release (to add the expr module to `sedona.db`). It also means that expressions are always created with the ability to reference a `SedonaContext` which will simplify one of the things I want to do next (chained accessors so we can do `sd.col("foo").geo.length()`)
##########
python/sedonadb/python/sedonadb/dataframe.py:
##########
@@ -114,40 +230,70 @@ def __getitem__(self, key: Union[str, int]) -> Expr:
Examples:
>>> sd = sedona.db.connect()
- >>> df = sd.sql("SELECT * FROM (VALUES (1, 10), (2, 20)) AS t(x, y)")
+ >>> df = sd.sql("SELECT * FROM (VALUES (1, 10), (2, 20)) AS t(x, y)").alias("t")
>>> df["x"]
- Expr(x)
+ Expr(t.x)
>>> df[1]
- Expr(y)
+ Expr(t.y)
>>> df[-1]
- Expr(y)
+ Expr(t.y)
"""
# `bool` is a subclass of `int`, so guard explicitly — otherwise
# `df[True]` would silently mean `df[1]`.
- if isinstance(key, bool) or not isinstance(key, (str, int)):
+ if isinstance(key, bool):
raise TypeError(
- f"DataFrame indexing is not supported for "
- f"{type(key).__name__}. Use df['name'] or df[i] for a "
- f"column expression; use df.select(*cols) for multi-column "
- f"projection and df.filter(expr) for row filtering."
+ "DataFrame indexing is not supported for bool. "
+ "Use df['name'] or df[i] for a column expression; "
+ "use df.select(*cols) for multi-column projection "
+ "and df.filter(expr) for row filtering."
)
-
- columns = self._impl.columns()
- if isinstance(key, str):
- if key not in columns:
- raise KeyError(
- f"Column {key!r} not found. Available columns: {columns}"
- )
- return _col(key)
- # int (and not bool, by the guard above)
- if not -len(columns) <= key < len(columns):
- raise IndexError(
- f"Column index {key} is out of range for DataFrame with "
- f"{len(columns)} columns"
+ if isinstance(key, list):
+ raise TypeError(
+ "DataFrame indexing with a list is not supported. "
+ "Use df.select(*cols) for multi-column projection."
)
- return _col(columns[key])
+ if isinstance(key, Expr):
+ raise TypeError(
+ "DataFrame indexing with an Expr is not supported. "
+ "Use df.filter(expr) for row filtering."
+ )
+ if isinstance(key, slice):
+ raise TypeError(
+ "DataFrame slicing is not supported. "
+ "Use df.limit(n) or df.limit(n, offset=k)."
+ )
+ inner_expr = self._impl.qualified_column_expr(key)
+ return Expr(inner_expr)
+
+ def __getattr__(self, name):
+ """Syntactic sugar for column access
+
+ Allows columns to be accessed like `t.geometry` for columns
+ that do not collide with existing attributes. Programmatic usage
+ should use `sd.col()` or `t["col"]`.
+ """
+ try:
+ return self[name]
+ except KeyError as e:
+ raise AttributeError(str(e))
+
+ def __dir__(self):
+ """List attributes of this object
+
+ This is primarily intended to power autocomplete, so columns
+ are placed first.
+ """
+ return self.columns + super().__dir__()
Review Comment:
This is a nice abbreviation (allowed by ibis). The `__dir__` implementation gives nice autocomplete here in notebooks:
<img width="350" height="151" alt="Image" src="https://github.com/user-attachments/assets/5a935d6f-bc60-492d-a6eb-4e5f3ccf2b2d" />
##########
python/sedonadb/python/sedonadb/dataframe.py:
##########
@@ -89,6 +192,19 @@ def head(self, n: int = 5) -> "DataFrame":
"""
return self.limit(n)
+ def alias(self, name: str) -> "DataFrame":
+ """Qualify all columns of this DataFrame with a given name
+
+ Returns a DataFrame where all columns are qualified to disambiguate
+ references in join expressions. This is the equivalent of aliasing a subquery
+ in SQL (`(SELECT * FROM df) AS name`).
+ """
Review Comment:
This and the change to `__getitem__` is needed for join expressions so that the right and left can be disambiguated. Something like
```python
df.inner_join(df2, f.st_intersects(df["geometry"], df2["geometry"]))`
```
The previous implementation would have created ambiguous references for this case. Usually on read the table is qualified with the filename, although in some cases it might be necessary to manually `.alias()` (e.g., a self join, or unqualified data frame from GeoPandas).
##########
python/sedonadb/python/sedonadb/dataframe.py:
##########
@@ -114,40 +230,70 @@ def __getitem__(self, key: Union[str, int]) -> Expr:
Examples:
>>> sd = sedona.db.connect()
- >>> df = sd.sql("SELECT * FROM (VALUES (1, 10), (2, 20)) AS t(x, y)")
+ >>> df = sd.sql("SELECT * FROM (VALUES (1, 10), (2, 20)) AS t(x, y)").alias("t")
>>> df["x"]
- Expr(x)
+ Expr(t.x)
>>> df[1]
- Expr(y)
+ Expr(t.y)
>>> df[-1]
- Expr(y)
+ Expr(t.y)
"""
# `bool` is a subclass of `int`, so guard explicitly — otherwise
# `df[True]` would silently mean `df[1]`.
- if isinstance(key, bool) or not isinstance(key, (str, int)):
+ if isinstance(key, bool):
raise TypeError(
- f"DataFrame indexing is not supported for "
- f"{type(key).__name__}. Use df['name'] or df[i] for a "
- f"column expression; use df.select(*cols) for multi-column "
- f"projection and df.filter(expr) for row filtering."
+ "DataFrame indexing is not supported for bool. "
+ "Use df['name'] or df[i] for a column expression; "
+ "use df.select(*cols) for multi-column projection "
+ "and df.filter(expr) for row filtering."
)
-
- columns = self._impl.columns()
- if isinstance(key, str):
- if key not in columns:
- raise KeyError(
- f"Column {key!r} not found. Available columns: {columns}"
- )
- return _col(key)
- # int (and not bool, by the guard above)
- if not -len(columns) <= key < len(columns):
- raise IndexError(
- f"Column index {key} is out of range for DataFrame with "
- f"{len(columns)} columns"
+ if isinstance(key, list):
+ raise TypeError(
+ "DataFrame indexing with a list is not supported. "
+ "Use df.select(*cols) for multi-column projection."
)
- return _col(columns[key])
+ if isinstance(key, Expr):
+ raise TypeError(
+ "DataFrame indexing with an Expr is not supported. "
+ "Use df.filter(expr) for row filtering."
+ )
+ if isinstance(key, slice):
+ raise TypeError(
+ "DataFrame slicing is not supported. "
+ "Use df.limit(n) or df.limit(n, offset=k)."
+ )
+ inner_expr = self._impl.qualified_column_expr(key)
+ return Expr(inner_expr)
+
+ def __getattr__(self, name):
+ """Syntactic sugar for column access
+
+ Allows columns to be accessed like `t.geometry` for columns
+ that do not collide with existing attributes. Programmatic usage
+ should use `sd.col()` or `t["col"]`.
+ """
+ try:
+ return self[name]
+ except KeyError as e:
+ raise AttributeError(str(e))
+
+ def __dir__(self):
+ """List attributes of this object
+
+ This is primarily intended to power autocomplete, so columns
+ are placed first.
+ """
+ return self.columns + super().__dir__()
+
+ def _ipython_key_completions_(self):
+ """Enable tab completion for df["col"] in IPython/Jupyter."""
+ return self.columns
Review Comment:
Not sure if this works elsewhere but it's nice in VSCode notebooks:
<img width="290" height="117" alt="Image" src="https://github.com/user-attachments/assets/0b63ee2e-8cd6-4bfe-9e92-213b14ef5a2d" />
##########
python/sedonadb/python/sedonadb/functions/__init__.py:
##########
@@ -38,3 +41,35 @@ def table(self) -> "TableFunctions":
from sedonadb.functions.table import TableFunctions
return TableFunctions(self._ctx)
+
+ def __getattr__(self, name) -> Union["ScalarUdf", "AggregateUdf"]:
+ try:
+ return ScalarUdf(self._ctx._impl.scalar_udf(name))
+ except SedonaError:
+ pass
+
+ try:
+ return AggregateUdf(self._ctx._impl.aggregate_udf(name))
+ except SedonaError:
+ pass
+
+ raise AttributeError(f"Can't find scalar or aggregate function '{name}'")
Review Comment:
This is a cheap way to be able to access *any* scalar or aggregate function without listing them all in code (i.e., automatically in sync without us having to do anything). We also might want to add additional accessors in the future that embed the documentation but this gets us most of the way there.
Both `__dir__` and `_ipython_key_completions_` are implemented here for help in the IDE, although you need to assign this to a variable first and execute which is not ideal and doesn't help in a non-notebook environment. (We can fix that with lazily imported accessors that inline docs but that's a follow-up).
<img width="302" height="266" alt="Image" src="https://github.com/user-attachments/assets/fa7075b4-e48b-404b-8163-cdd5d273b56e" />
<img width="337" height="345" alt="Image" src="https://github.com/user-attachments/assets/469b6d2f-18af-4e82-9050-d1222cd28d12" />
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org