[I] STAC Python client: multi-bbox search with max_items enumerates the entire collection [sedona]

["jiayuasu (via GitHub)" <gi...@apache.org>]

jiayuasu opened a new issue, #3286:
URL: https://github.com/apache/sedona/issues/3286

   ## Expected behavior
   
   ```python
   client = Client.open("https://earth-search.aws.element84.com/v1")
   items = client.search(
       collection_id="sentinel-2-c1-l2a",
       bbox=[[-10.0, 35.0, 5.0, 45.0], [100.0, -10.0, 120.0, 10.0]],
       datetime="2025",
       max_items=100,
   )
   ```
   
   should fetch roughly `max_items` results per bounding box from the STAC API and return quickly, the way a bounded search is expected to behave.
   
   ## Actual behavior
   
   GH-3283 added API-native capped searches, but only for at most **one** bbox. A search with multiple bounding boxes takes the Spark-filtered fallback: the driver sequentially enumerates **every** result page of the collection before Spark applies the bbox predicates and the final limit. The fallback also inherits the datasource's default page size of 10 Items per request, so a bounded multi-bbox query can issue tens of thousands of sequential HTTP requests and run into service throttling.
   
   ## Proposed fix
   
   Each bbox maps directly to a STAC API Collection Items request, and Sedona's list-of-bboxes extension represents their union, so a bounded multi-bbox search can be decomposed instead of falling back:
   
   - For a named HTTP collection with a positive `max_items`, one or more valid bboxes, at most one `datetime` interval, and no ID or geometry filter: issue one API-native capped search per bbox (`itemsLimitMax=max_items`, `itemsLimitPerRequest=min(200, max_items)`), union the per-bbox DataFrames, deduplicate by `(collection, id)`, and apply one final global `max_items` limit.
   - Each request may safely use the full `max_items` cap: if any bbox alone contains at least that many Items it can satisfy the limit; otherwise every bbox is fully represented, so deduplicate-then-limit yields the requested global result size.
   - Shapes that still require a Spark-side predicate (multiple datetime intervals, geometry filters, IDs, or an invalid bbox) keep the uncapped fallback so a raw fetch cap cannot starve residual predicates — but use 200-Item request pages to reduce pagination overhead.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] STAC Python client: multi-bbox search with max_items enumerates the entire collection [sedona]

["jiayuasu (via GitHub)" <gi...@apache.org>]

jiayuasu closed issue #3286: STAC Python client: multi-bbox search with max_items enumerates the entire collection
URL: https://github.com/apache/sedona/issues/3286


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org