You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@sedona.apache.org by "dfischercodethoughts (via GitHub)" <gi...@apache.org> on 2023/06/05 16:09:37 UTC

[GitHub] [sedona] dfischercodethoughts opened a new issue, #849: Spatial Joins with a Prebuilt Index Loads All Results In Memory

dfischercodethoughts opened a new issue, #849:
URL: https://github.com/apache/sedona/issues/849

   ## Expected behavior
   Similar to the ```DynamicIndexLookupJudgement``` code, the prebuilt index ```Left/RightIndexLookupJudgement```s should operate on a new custom individual iterator which can ```loadNextBatch``` so an entire partition need not be read into executor memory.
   
   ## Actual behavior
   The prebuilt index IntersectionJoinJudgement loads all results as a list in memory before returning an iterator.
   
   ## Steps to reproduce the problem
   Create a geometry with 40k points and intersection join against 1m rows of medium/small geometries (~1kb each) with a prebuilt index. This will OOM on most default executors (i.e. ones with ~2-3 GB memory).
   
   ## Settings


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@sedona.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [sedona] jiayuasu closed issue #849: Spatial Joins with a Prebuilt Index Loads All Results In Memory

Posted by "jiayuasu (via GitHub)" <gi...@apache.org>.
jiayuasu closed issue #849: Spatial Joins with a Prebuilt Index Loads All Results In Memory
URL: https://github.com/apache/sedona/issues/849


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@sedona.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [sedona] jiayuasu commented on issue #849: Spatial Joins with a Prebuilt Index Loads All Results In Memory

Posted by "jiayuasu (via GitHub)" <gi...@apache.org>.
jiayuasu commented on issue #849:
URL: https://github.com/apache/sedona/issues/849#issuecomment-1578144959

   @dfischercodethoughts Thanks for catching this. Will fix this soon.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@sedona.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [sedona] jiayuasu commented on issue #849: Spatial Joins with a Prebuilt Index Loads All Results In Memory

Posted by "jiayuasu (via GitHub)" <gi...@apache.org>.
jiayuasu commented on issue #849:
URL: https://github.com/apache/sedona/issues/849#issuecomment-1580027669

   Fixed by #851 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@sedona.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org