You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@sedona.apache.org by "paleolimbot (via GitHub)" <gi...@apache.org> on 2026/06/09 20:51:23 UTC
[PR] feat(python/sedonadb): Ensure dataframes have meaningful default alias [sedona-db]
paleolimbot opened a new pull request, #935:
URL: https://github.com/apache/sedona-db/pull/935
This PR ensures all dataframes when created have a default alias that is probably unique and mostly meaningful (e.g., the filename being read). This ensures that joins don't require manual aliasing in most cases.
Motivating use case:
```python
import sedona.db
sd = sedona.db.connect()
pts = sd.read_parquet(
"https://github.com/geoarrow/geoarrow-data/releases/download/v0.2.0/ns-water_water-point.parquet"
)
ply = sd.read_parquet(
"https://github.com/geoarrow/geoarrow-data/releases/download/v0.2.0/ns-water_water-poly.parquet"
)
pts.join(ply, on=sd.funcs.st_intersects(pts.geometry, ply.geometry)).select(
pt_hid=pts.HID, pl_hid=ply.HID, geometry=pts.geometry
).show(5)
# ┌────────────────────────────────┬────────────────────────────────┬────────────────────────────────┐
# │ pt_hid ┆ pl_hid ┆ geometry │
# │ utf8 ┆ utf8 ┆ geometry │
# ╞════════════════════════════════╪════════════════════════════════╪════════════════════════════════╡
# │ FC9A29123BEF4A6588A2FD777B81A… ┆ 63E9CC5AE7144EEABEBEC2C95C91F… ┆ POINT Z(258498.62729999982 48… │
# ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
# │ 28E47E22D71549ED91472AD59A9D8… ┆ 63E9CC5AE7144EEABEBEC2C95C91F… ┆ POINT Z(258498.32830000017 48… │
# ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
# │ B10B26FA32FB4438925D86CD612BA… ┆ 63E9CC5AE7144EEABEBEC2C95C91F… ┆ POINT Z(258502.0273000002 481… │
# ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
# │ 6ACD71128B6B492C8177B11A5C2D2… ┆ 63E9CC5AE7144EEABEBEC2C95C91F… ┆ POINT Z(258498.92729999963 48… │
# ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
# │ 0A4BE2AB03D8459B8279FCC4BDE55… ┆ 63E9CC5AE7144EEABEBEC2C95C91F… ┆ POINT Z(258526.62729999982 48… │
# └────────────────────────────────┴────────────────────────────────┴────────────────────────────────┘
```
A downside here is that the explain plans get more verbose because they always print out the qualifier.
Closes #926.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(python/sedonadb): Ensure dataframes have meaningful default alias [sedona-db]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot closed pull request #935: feat(python/sedonadb): Ensure dataframes have meaningful default alias
URL: https://github.com/apache/sedona-db/pull/935
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(python/sedonadb): Ensure dataframes have meaningful default alias [sedona-db]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on PR #935:
URL: https://github.com/apache/sedona-db/pull/935#issuecomment-4692705211
@jiayuasu Do you have bandwidth to review this one since it's mostly for joins?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(python/sedonadb): Ensure dataframes have meaningful default alias [sedona-db]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #935:
URL: https://github.com/apache/sedona-db/pull/935#discussion_r3384183544
##########
python/sedonadb/python/sedonadb/context.py:
##########
@@ -290,13 +292,13 @@ def read_parquet(
return DataFrame(
self,
self._impl.read_parquet(
- [str(path) for path in table_paths],
+ table_paths_list,
options,
geometry_columns,
validate,
None if partitioning is None else list(partitioning),
),
- )
+ )._ensure_aliased(table_paths)
Review Comment:
`read_parquet()` computes `table_paths_list` by consuming `table_paths` when it's an iterable, but then uses the original `table_paths` object to derive the default alias. For generators (e.g. `Path.glob(...)`) this yields a non-meaningful alias like `generator_<id>` and can even reference an exhausted iterator; it also won't match the intention/tests of aliasing based on the file being read.
##########
python/sedonadb/python/sedonadb/context.py:
##########
@@ -435,11 +441,11 @@ def read_format(
self,
self._impl.read_external_format(
spec,
- [str(path) for path in table_paths],
+ table_paths_list,
check_extension,
None if partitioning is None else list(partitioning),
),
- )
+ )._ensure_aliased(table_paths)
Review Comment:
`read_format()` defines `table_paths_list` only in the `else` branch; for a single `str/Path` input this will raise `NameError` when calling `_impl.read_external_format(...)`. Also, the alias source should be derived from the resolved string path(s), not the original iterable (which may be exhausted).
##########
python/sedonadb/python/sedonadb/context.py:
##########
@@ -378,11 +382,11 @@ def read_pyogrio(
self,
self._impl.read_external_format(
spec,
- [str(path) for path in table_paths],
+ table_paths_list,
False,
None if partitioning is None else list(partitioning),
),
- )
+ )._ensure_aliased(table_paths)
Review Comment:
`read_pyogrio()` computes `table_paths_list` by consuming `table_paths` when it's an iterable, but then uses the original `table_paths` object to derive the default alias. For iterators (e.g. `Path.glob(...)`) this produces an unhelpful alias based on the iterator type/id rather than the file path(s).
##########
python/sedonadb/python/sedonadb/dataframe.py:
##########
@@ -1429,6 +1429,18 @@ def _out_width(self, width=None) -> int:
return width
+ def _ensure_aliased(self, src: Any) -> "DataFrame":
+ return self.alias(_default_alias_for_obj(src))
+
+
+def _default_alias_for_obj(obj: Any) -> str:
+ if isinstance(obj, str):
+ return obj
+ elif isinstance(obj, Path):
+ return str(obj)
+
+ return f"{type(obj).__name__}_{id(obj)}"
Review Comment:
`_default_alias_for_obj()` returns the full input string/path verbatim for `str`/`Path`, which (a) is often not very meaningful (full URL / full directory path) and (b) is not even "probably unique" (reading the same path twice yields identical aliases and can still cause ambiguous references in joins). Consider deriving the alias from the basename and adding a per-DataFrame suffix to avoid collisions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(python/sedonadb): Ensure dataframes have meaningful default alias [sedona-db]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #935:
URL: https://github.com/apache/sedona-db/pull/935#discussion_r3392852171
##########
python/sedonadb/python/sedonadb/dataframe.py:
##########
@@ -1455,7 +1477,7 @@ def _create_data_frame(ctx, obj, schema) -> DataFrame:
# __datafusion_table_provider__ or __arrow_c_stream__. For objects implementing
# __arrow_c_stream__, this currently will only work for a single scan (i.e.,
# the returned data frame can't be previewed before the query is computed).
- return _scan_default(ctx, obj, schema)
+ return _scan_default(ctx, obj, schema)._ensure_aliased(obj)
Review Comment:
_create_data_frame() only applies the new default aliasing to the generic scan path; objects handled via SPECIAL_CASED_SCANS (pyarrow/pandas/geopandas/polars) return a DataFrame without calling _ensure_aliased(). This means many common DataFrame creations will still end up with the anonymous/default qualifier (the original motivation for #926), so joins may remain ambiguous for these inputs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(python/sedonadb): Ensure dataframes have meaningful default alias [sedona-db]
Posted by "jiayuasu (via GitHub)" <gi...@apache.org>.
jiayuasu commented on PR #935:
URL: https://github.com/apache/sedona-db/pull/935#issuecomment-4697938990
Two things give me pause about this mechanism,
though:
1. It doesn't actually solve self-joins. Because the alias is baked from
the source object at create time, a self-join gets the same qualifier on
both sides. I tried the predicate path from your example:
```
df = sd.create_data_frame(...).alias("foo_deadbeef") # one default
alias
df.join(df, on=df["k"] == df["k"])
```
2. 2. The id()-based suffix is non-deterministic. the qualifier changes every run
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(python/sedonadb): Ensure dataframes have meaningful default alias [sedona-db]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on PR #935:
URL: https://github.com/apache/sedona-db/pull/935#issuecomment-4709164323
Got it...we can manually alias and/or add spatial join helpers later
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org