You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@sedona.apache.org by "james-willis (via GitHub)" <gi...@apache.org> on 2026/07/23 21:30:19 UTC
[PR] perf(rust/sedona-spatial-join-raster): pin raster operand to the probe side [sedona-db]
james-willis opened a new pull request, #1074:
URL: https://github.com/apache/sedona-db/pull/1074
Stacked on #1073 — the diff shows both branches' commits until #1073 merges, then reduces to this branch's single commit.
Pins the raster operand of a raster–vector spatial join to the probe (streamed) side so its BinaryView band buffers stay uncopied, short-lived, and shared across the rows a raster fans out to — instead of being buffered and gc-compacted on the build side.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(rust/sedona-spatial-join-raster): pin raster operand to the probe side [sedona-db]
Posted by "james-willis (via GitHub)" <gi...@apache.org>.
james-willis commented on PR #1074:
URL: https://github.com/apache/sedona-db/pull/1074#issuecomment-5065013371
> This is only necessary when we have RS_Intersects followed by a zonal states, right?
I guess you could have other operators like udf or AsRaster? I thought it would be fine to keep this simple and not condition the logic.
Would you rather leave the standard build-side logic in most cases and use some optimized rule to pipe in a probe side parameter to the join node? I feel that adds a lot of complexity and im not sure of the value
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(rust/sedona-spatial-join-raster): pin raster operand to the probe side [sedona-db]
Posted by "jiayuasu (via GitHub)" <gi...@apache.org>.
jiayuasu commented on PR #1074:
URL: https://github.com/apache/sedona-db/pull/1074#issuecomment-5066198110
Correct, I would rather to keep the current behavior as it uses statistics to detect the best side to be the probe side. I think I should correct our discussion today as follows:
When there is RS_ZonalStats(vec, ras) operator, assume we have input record batches, each of which is as follows (please adapt this to the DataFusion language).
```
Vec1, Ras1
Vec2, Ras2
Vec3, Ras3
Vec1, Ras2
Vec2, Ras1
```
If we just perform the operator in the normal order, you will need to either load Ras1 twice or pin Ras1 in memory cache until the end of the query.
Instead, we should have a logical / physical optimization rules (or a new algorithm?) to sort the records in the batch by Ras ID. We will have something as follows
```
Vec1, Ras1
Vec2, Ras1
Vec2, Ras2
Vec1, Ras2
Vec3, Ras3
```
I think this has nothing to do with spatial join. It is a pure zonal stats optimization. I think Kristin might have already done this in his Zonal stats implementation. Please double check.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(rust/sedona-spatial-join-raster): pin raster operand to the probe side [sedona-db]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on PR #1074:
URL: https://github.com/apache/sedona-db/pull/1074#issuecomment-5071352315
Sure, but it sounds like then we loose zero copy for (maybe tiled) numpy arrays that hit the zonal stats.
We also need better stats (in particular for pushdown between sides of a join, but also to possibly make a better decision on which side to build on), but that shouldn't block the best choice for the implementation we have today.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(rust/sedona-spatial-join-raster): pin raster operand to the probe side [sedona-db]
Posted by "jiayuasu (via GitHub)" <gi...@apache.org>.
jiayuasu commented on PR #1074:
URL: https://github.com/apache/sedona-db/pull/1074#issuecomment-5071303234
@paleolimbot Either way, the raster join only uses the convex hull of the raster and then it becomes a vector - vector join. Then we don't need to have any special treatment for raster. You can argue that raster data's hull is simpler but in that case, we need to have better stats in spatial join to fix that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(rust/sedona-spatial-join-raster): pin raster operand to the probe side [sedona-db]
Posted by "james-willis (via GitHub)" <gi...@apache.org>.
james-willis commented on PR #1074:
URL: https://github.com/apache/sedona-db/pull/1074#issuecomment-5072073564
We need special treatment for raster because we don't want to load the same image from the network many times.
Thats the idea of this PR:
1. first we load so when theres 0 dupes of rasters we've already created the binary view
2. then we join so when we copy the raster from the input to the output we reuse the backing buffer of the binary view. We use the probe side so rasters sharing a buffer backing their binary view tend to wind up in the same batch of downstream
3. finally run zonal stats. all copies of the same raster come through the same batch and thus all references to the same buffer are released. This keeps peak ram usage down.
The alternative is a cache.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(rust/sedona-spatial-join-raster): pin raster operand to the probe side [sedona-db]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on PR #1074:
URL: https://github.com/apache/sedona-db/pull/1074#issuecomment-5071038081
> I would rather keep the current behavior as it uses statistics to detect the best side to be the probe side.
I'll note that we've had a lot of trouble with our current heuristic, which only uses row count or byte size to determine the best probe side. We should use more advanced stats to choose this but we don't. In fact, the only non-committer contributions to the spatial join have been to add an option to disable our heuristic 🙂
The current heuristic is to pick the smallest side to index, on the bet that the largest side will have to spill and therefore be slower. On those grounds, we should always be indexing the raster side. A better heuristic is probably to index the geometry side, because those geometries are likely to be more complex and benefit from preparation. It sounds like indexing on the geometry side is also better for the case where we have materialized rasters.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(rust/sedona-spatial-join-raster): pin raster operand to the probe side [sedona-db]
Posted by "james-willis (via GitHub)" <gi...@apache.org>.
james-willis commented on PR #1074:
URL: https://github.com/apache/sedona-db/pull/1074#issuecomment-5066406423
Kristins work does not sort the input of the RS functions, but also that doesn't matter when we aren't using some kind of cache.
The reason we want the raster as the probe side is that so we can load before the join. We need to load before duplicating to avoid duplicate loads. If we dupe before load we dont share a BinaryView. if we load before we dupe theyll share the buffer backing the binary view
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(rust/sedona-spatial-join-raster): pin raster operand to the probe side [sedona-db]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on PR #1074:
URL: https://github.com/apache/sedona-db/pull/1074#issuecomment-5072513291
If I'm reading this correctly, there's danger that by doing that we're also loading a lot of unnecessary images (i.e., images that never touch a feature). If we keep them lazy there's maybe chance we can load some of them (and, crucially, drop the loaded versions of them) at a time?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(rust/sedona-spatial-join-raster): pin raster operand to the probe side [sedona-db]
Posted by "james-willis (via GitHub)" <gi...@apache.org>.
james-willis commented on PR #1074:
URL: https://github.com/apache/sedona-db/pull/1074#issuecomment-5073165730
@paleolimbot yes that is true.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(rust/sedona-spatial-join-raster): pin raster operand to the probe side [sedona-db]
Posted by "jiayuasu (via GitHub)" <gi...@apache.org>.
jiayuasu commented on PR #1074:
URL: https://github.com/apache/sedona-db/pull/1074#issuecomment-5066565435
In vector raster join, we never load the raster data, don't we? We just do the join using the convex hull of the raster metadata and pixels are never used
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(rust/sedona-spatial-join-raster): pin raster operand to the probe side [sedona-db]
Posted by "jiayuasu (via GitHub)" <gi...@apache.org>.
jiayuasu commented on PR #1074:
URL: https://github.com/apache/sedona-db/pull/1074#issuecomment-5064816741
This is only necessary when we have RS_Intersects followed by a zonal states, right?
The RS_Intersects only uses the metadata of raster so pinning raster to the probe side probably does not matter much?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(rust/sedona-spatial-join-raster): pin raster operand to the probe side [sedona-db]
Posted by "james-willis (via GitHub)" <gi...@apache.org>.
james-willis merged PR #1074:
URL: https://github.com/apache/sedona-db/pull/1074
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org