You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@sedona.apache.org by "jiayuasu (via GitHub)" <gi...@apache.org> on 2026/02/09 07:36:05 UTC
[PR] [GH-2407] Auto-detect raster columns in SedonaUtils.display_image [sedona]
jiayuasu opened a new pull request, #2633:
URL: https://github.com/apache/sedona/pull/2633
## Did you read the Contributor Guide?
- Yes, I have read the [Contributor Rules](https://sedona.apache.org/latest/community/rule/) and [Contributor Development Guide](https://sedona.apache.org/latest/community/develop/)
## Is this PR related to a ticket?
- Yes, and the PR name follows the format `[GH-XXX] my subject`. Closes #2407
## What changes were proposed in this PR?
`SedonaUtils.display_image()` hangs when passed a raw raster DataFrame (e.g., 1400x800) because `__convert_to_gdf_or_pdf__` attempts to Arrow-serialize the full GridCoverage2D object.
This PR adds raster UDT detection to `display_image()`:
- Detects raster columns via `typeName() == "rastertype"`
- Auto-applies `RS_AsImage()` to each raster column before rendering
- Preserves all non-raster columns in the DataFrame
- Falls through to the existing `__convert_to_gdf_or_pdf__` + `to_html()` path after conversion
Users can now pass a raw raster DataUsers can now pass a raw raster DataUsers can now pass a raw raster DataUsers this patch tested?
- Tested manually in Jupyter notebook inside the Sedona Docker container
- Verified with raw raster DataFrame (single and multi-column)
- Verified with pre-applied `RS_AsImage()` DataFrame (regression check)
- Existing test `test_sedonautils.py::test_display_image` continues to pass
## Did this PR include necessary documentation updates?
- Yes, I have updated the documentation. Updated `docs/api/sql/Raster-visualizer.md` to show both the new direct raster display (recommended) and the explicit `RS_AsImage` workflow.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [GH-2407] Auto-detect raster columns in SedonaUtils.display_image [sedona]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #2633:
URL: https://github.com/apache/sedona/pull/2633#discussion_r2781787884
##########
python/tests/raster_viz_utils/test_sedonautils.py:
##########
@@ -35,3 +35,28 @@ def test_display_image(self):
assert (
html_call is None
) # just test that this function was called and returned no output
+
+ def test_display_image_raw_raster(self):
+ """Test that display_image auto-detects raster columns and applies RS_AsImage."""
+ raster_bin_df = self.spark.read.format("binaryFile").load(
+ world_map_raster_input_location
+ )
+ raster_bin_df.createOrReplaceTempView("raster_binary_table")
Review Comment:
Both tests register the same temp view name (`raster_binary_table`). This can cause cross-test interference when running tests concurrently (or if other tests in the suite use the same view name). Prefer avoiding temp views here (use `selectExpr` directly on the DataFrame) or use a unique view name per test.
##########
python/sedona/spark/raster_utils/SedonaUtils.py:
##########
@@ -21,7 +21,40 @@
class SedonaUtils:
@classmethod
def display_image(cls, df):
+ """Display raster images in a Jupyter notebook.
+
+ Accepts DataFrames with either:
+ - A raster column (GridCoverage2D) — auto-applies RS_AsImage
+ - An HTML image column from RS_AsImage() — renders directly
+
+ Falls back to the SedonaMapUtils HTML table path for other DataFrames.
+ """
from IPython.display import HTML, display
+ schema = df.schema
+
+ # Detect raster UDT columns and auto-apply RS_AsImage.
+ # Without this, passing a raw raster DataFrame to the fallback path
+ # causes __convert_to_gdf_or_pdf__ to Arrow-serialize the full raster
+ # grid, which hangs for large rasters (e.g., 1400x800).
+ raster_cols = [
+ f.name
+ for f in schema.fields
+ if hasattr(f.dataType, "typeName") and f.dataType.typeName() == "rastertype"
+ ]
+ if raster_cols:
+ # Replace each raster column with its RS_AsImage() HTML representation,
+ # preserving all other columns in the DataFrame.
+ select_exprs = []
+ for f in schema.fields:
+ col_name_escaped = f.name.replace("`", "``")
+ if f.name in raster_cols:
+ select_exprs.append(
+ f"RS_AsImage(`{col_name_escaped}`) as `{col_name_escaped}`"
+ )
+ else:
+ select_exprs.append(f"`{col_name_escaped}`")
+ df = df.selectExpr(*select_exprs)
Review Comment:
`display_image` now unconditionally assumes a Spark DataFrame by accessing `df.schema` / `df.selectExpr`. If `display_image` is (or was) intended to accept non-Spark inputs that `SedonaMapUtils.__convert_to_gdf_or_pdf__` can handle, this introduces a breaking change (AttributeError before reaching the fallback). Consider guarding the raster auto-detection behind a Spark-DataFrame check (e.g., `hasattr(df, "schema") and hasattr(df, "selectExpr")`) and otherwise directly fall through to `__convert_to_gdf_or_pdf__`.
```suggestion
# Only attempt Spark-specific raster auto-detection if `df` looks like
# a Spark DataFrame. For other input types, fall through directly to
# SedonaMapUtils.__convert_to_gdf_or_pdf__.
if hasattr(df, "schema") and hasattr(df, "selectExpr"):
schema = df.schema
# Detect raster UDT columns and auto-apply RS_AsImage.
# Without this, passing a raw raster DataFrame to the fallback path
# causes __convert_to_gdf_or_pdf__ to Arrow-serialize the full raster
# grid, which hangs for large rasters (e.g., 1400x800).
raster_cols = [
f.name
for f in schema.fields
if hasattr(f.dataType, "typeName") and f.dataType.typeName() == "rastertype"
]
if raster_cols:
# Replace each raster column with its RS_AsImage() HTML representation,
# preserving all other columns in the DataFrame.
select_exprs = []
for f in schema.fields:
col_name_escaped = f.name.replace("`", "``")
if f.name in raster_cols:
select_exprs.append(
f"RS_AsImage(`{col_name_escaped}`) as `{col_name_escaped}`"
)
else:
select_exprs.append(f"`{col_name_escaped}`")
df = df.selectExpr(*select_exprs)
```
##########
python/tests/raster_viz_utils/test_sedonautils.py:
##########
@@ -35,3 +35,28 @@ def test_display_image(self):
assert (
html_call is None
) # just test that this function was called and returned no output
+
+ def test_display_image_raw_raster(self):
+ """Test that display_image auto-detects raster columns and applies RS_AsImage."""
+ raster_bin_df = self.spark.read.format("binaryFile").load(
+ world_map_raster_input_location
+ )
+ raster_bin_df.createOrReplaceTempView("raster_binary_table")
+ # Pass a raw raster DataFrame — no RS_AsImage applied
+ raster_df = self.spark.sql(
+ "SELECT RS_FromGeotiff(content) as raster from raster_binary_table"
+ )
+ html_call = SedonaUtils.display_image(raster_df)
+ assert html_call is None
+
+ def test_display_image_preserves_non_raster_columns(self):
+ """Test that non-raster columns are preserved alongside raster columns."""
+ raster_bin_df = self.spark.read.format("binaryFile").load(
+ world_map_raster_input_location
+ )
+ raster_bin_df.createOrReplaceTempView("raster_binary_table")
+ raster_df = self.spark.sql(
+ "SELECT path, RS_FromGeotiff(content) as raster from raster_binary_table"
+ )
+ html_call = SedonaUtils.display_image(raster_df)
+ assert html_call is None
Review Comment:
The new tests exercise the call path but don’t assert the new behavior: (1) that raster columns are actually converted via `RS_AsImage`, and (2) that non-raster columns are preserved. Since `display_image` returns `None`, consider asserting against an observable side effect (e.g., mock/spy `SedonaMapUtils.__convert_to_gdf_or_pdf__` to capture the DataFrame passed in and validate its schema/columns/types, or mock `IPython.display.display` and assert the rendered HTML contains an `<img` tag and includes the non-raster column value).
##########
python/tests/raster_viz_utils/test_sedonautils.py:
##########
@@ -35,3 +35,28 @@ def test_display_image(self):
assert (
html_call is None
) # just test that this function was called and returned no output
+
+ def test_display_image_raw_raster(self):
+ """Test that display_image auto-detects raster columns and applies RS_AsImage."""
+ raster_bin_df = self.spark.read.format("binaryFile").load(
+ world_map_raster_input_location
+ )
+ raster_bin_df.createOrReplaceTempView("raster_binary_table")
+ # Pass a raw raster DataFrame — no RS_AsImage applied
+ raster_df = self.spark.sql(
+ "SELECT RS_FromGeotiff(content) as raster from raster_binary_table"
+ )
+ html_call = SedonaUtils.display_image(raster_df)
+ assert html_call is None
+
+ def test_display_image_preserves_non_raster_columns(self):
+ """Test that non-raster columns are preserved alongside raster columns."""
+ raster_bin_df = self.spark.read.format("binaryFile").load(
+ world_map_raster_input_location
+ )
+ raster_bin_df.createOrReplaceTempView("raster_binary_table")
Review Comment:
Both tests register the same temp view name (`raster_binary_table`). This can cause cross-test interference when running tests concurrently (or if other tests in the suite use the same view name). Prefer avoiding temp views here (use `selectExpr` directly on the DataFrame) or use a unique view name per test.
##########
python/tests/raster_viz_utils/test_sedonautils.py:
##########
@@ -35,3 +35,28 @@ def test_display_image(self):
assert (
html_call is None
) # just test that this function was called and returned no output
+
+ def test_display_image_raw_raster(self):
+ """Test that display_image auto-detects raster columns and applies RS_AsImage."""
+ raster_bin_df = self.spark.read.format("binaryFile").load(
+ world_map_raster_input_location
+ )
+ raster_bin_df.createOrReplaceTempView("raster_binary_table")
+ # Pass a raw raster DataFrame — no RS_AsImage applied
+ raster_df = self.spark.sql(
+ "SELECT RS_FromGeotiff(content) as raster from raster_binary_table"
+ )
+ html_call = SedonaUtils.display_image(raster_df)
+ assert html_call is None
Review Comment:
The new tests exercise the call path but don’t assert the new behavior: (1) that raster columns are actually converted via `RS_AsImage`, and (2) that non-raster columns are preserved. Since `display_image` returns `None`, consider asserting against an observable side effect (e.g., mock/spy `SedonaMapUtils.__convert_to_gdf_or_pdf__` to capture the DataFrame passed in and validate its schema/columns/types, or mock `IPython.display.display` and assert the rendered HTML contains an `<img` tag and includes the non-raster column value).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [GH-2407] Auto-detect raster columns in SedonaUtils.display_image [sedona]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #2633:
URL: https://github.com/apache/sedona/pull/2633#discussion_r2781122146
##########
docs/tutorial/raster.md:
##########
@@ -472,6 +472,17 @@ The output looks like this:
+!!!tip
+ In a Jupyter notebook, use `SedonaUtils.display_image` to render rasters directly — no need to call RS_AsImage manually:
Review Comment:
The admonition syntax appears incorrect for MkDocs-style admonitions: it should be `!!! tip` (note the space) and typically uses a lowercase type for consistency. As written (`!!!tip`), this may render as plain text instead of a tip block. Please update this (and the other newly added `!!!tip` blocks in the docs) to the correct admonition syntax.
##########
python/sedona/spark/raster_utils/SedonaUtils.py:
##########
@@ -21,7 +21,39 @@
class SedonaUtils:
@classmethod
def display_image(cls, df):
+ """Display raster images in a Jupyter notebook.
+
+ Accepts DataFrames with either:
+ - A raster column (GridCoverage2D) — auto-applies RS_AsImage
+ - An HTML image column from RS_AsImage() — renders directly
+
+ Falls back to the SedonaMapUtils HTML table path for other DataFrames.
+ """
from IPython.display import HTML, display
+ schema = df.schema
+
+ # Detect raster UDT columns and auto-apply RS_AsImage.
+ # Without this, passing a raw raster DataFrame to the fallback path
+ # causes __convert_to_gdf_or_pdf__ to Arrow-serialize the full raster
+ # grid, which hangs for large rasters (e.g., 1400x800).
+ raster_cols = [
+ f.name
+ for f in schema.fields
+ if hasattr(f.dataType, "typeName") and f.dataType.typeName() == "rastertype"
+ ]
+ if raster_cols:
+ # Replace each raster column with its RS_AsImage() HTML representation,
+ # preserving all other columns in the DataFrame.
+ select_exprs = [
+ (
+ f"RS_AsImage(`{f.name}`) as `{f.name}`"
+ if f.name in raster_cols
+ else f"`{f.name}`"
+ )
+ for f in schema.fields
+ ]
+ df = df.selectExpr(*select_exprs)
Review Comment:
The new raster-column auto-detection and projection path isn’t covered by automated tests (the existing `test_display_image` passing suggests it covers the pre-`RS_AsImage`/HTML path). Please add a test that passes a DataFrame with a raster-typed column to `display_image()` and asserts the raster column is converted via `RS_AsImage` (and that non-raster columns are preserved).
##########
python/sedona/spark/raster_utils/SedonaUtils.py:
##########
@@ -21,7 +21,39 @@
class SedonaUtils:
@classmethod
def display_image(cls, df):
+ """Display raster images in a Jupyter notebook.
+
+ Accepts DataFrames with either:
+ - A raster column (GridCoverage2D) — auto-applies RS_AsImage
+ - An HTML image column from RS_AsImage() — renders directly
+
+ Falls back to the SedonaMapUtils HTML table path for other DataFrames.
+ """
from IPython.display import HTML, display
+ schema = df.schema
+
+ # Detect raster UDT columns and auto-apply RS_AsImage.
+ # Without this, passing a raw raster DataFrame to the fallback path
+ # causes __convert_to_gdf_or_pdf__ to Arrow-serialize the full raster
+ # grid, which hangs for large rasters (e.g., 1400x800).
+ raster_cols = [
+ f.name
+ for f in schema.fields
+ if hasattr(f.dataType, "typeName") and f.dataType.typeName() == "rastertype"
+ ]
+ if raster_cols:
+ # Replace each raster column with its RS_AsImage() HTML representation,
+ # preserving all other columns in the DataFrame.
+ select_exprs = [
+ (
+ f"RS_AsImage(`{f.name}`) as `{f.name}`"
+ if f.name in raster_cols
+ else f"`{f.name}`"
+ )
+ for f in schema.fields
+ ]
Review Comment:
Column names are interpolated into SQL strings for `selectExpr` and only wrapped in backticks. If a column name itself contains a backtick, Spark SQL parsing can break and (depending on context) this can become an injection vector. Please escape backticks in `f.name` (Spark uses doubled backticks) before embedding them in the expression strings.
```suggestion
select_exprs = []
for f in schema.fields:
col_name_escaped = f.name.replace("`", "``")
if f.name in raster_cols:
select_exprs.append(
f"RS_AsImage(`{col_name_escaped}`) as `{col_name_escaped}`"
)
else:
select_exprs.append(f"`{col_name_escaped}`")
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [GH-2407] Auto-detect raster columns in SedonaUtils.display_image [sedona]
Posted by "jiayuasu (via GitHub)" <gi...@apache.org>.
jiayuasu merged PR #2633:
URL: https://github.com/apache/sedona/pull/2633
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org