You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@sedona.apache.org by "paleolimbot (via GitHub)" <gi...@apache.org> on 2026/03/30 18:10:54 UTC

[I] rust/sedona-spatial-join: Improve memory estimation for EvaluatedGeometryArray [sedona-db]

paleolimbot opened a new issue, #745:
URL: https://github.com/apache/sedona-db/issues/745

   There is at least one place where we may be double counting a large chunk of memory in the EvaluatedBatch:
   
   ```rust
           // NOTE: sometimes `geom_array` will reuse the memory of `batch`, especially when
           // the expression for evaluating the geometry is a simple column reference. In this case,
           // the in_mem_size will be overestimated. It is a conservative estimation so there's no risk
           // of running out of memory because of underestimation.
           let record_batch_size = get_record_batch_memory_size(&self.batch)?;
           let geom_array_size = self.geom_array.in_mem_size()?;
           Ok(record_batch_size + geom_array_size)
   ```
   
   This might explain an issue we ran across when trying to enable this by default where we determined we'd need to set the memory pool size to more than twice as much memory as was required for a join used in the released post (my reading of that comment is that we would be reserving ~2x as much memory as was required for most joins but I have not investigated).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] rust/sedona-spatial-join: Improve memory estimation and serialization for EvaluatedGeometryArray [sedona-db]

Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot closed issue #745: rust/sedona-spatial-join: Improve memory estimation and serialization for EvaluatedGeometryArray
URL: https://github.com/apache/sedona-db/issues/745


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] rust/sedona-spatial-join: Improve memory estimation and serialization for EvaluatedGeometryArray [sedona-db]

Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on issue #745:
URL: https://github.com/apache/sedona-db/issues/745#issuecomment-4158226366

   Something else that came up in #737 is that we are potentially double serializing the geometry (in addition to the geometry f32 rect bounds, which may or may not be needed depending on the index implementation, or may or may not be faster to compute than to compress depending on the compression options / speed of bounding).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org