You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@sedona.apache.org by "jiayuasu (via GitHub)" <gi...@apache.org> on 2026/08/23 07:02:45 UTC
[I] [BUG] STAC reader sends duplicate datetime parameters for constrained endpoints [sedona]
jiayuasu opened a new issue, #3277:
URL: https://github.com/apache/sedona/issues/3277
## Expected behavior
When a STAC items or search endpoint already contains a datetime query parameter, that parameter defines the source relation. Pushing a Spark temporal predicate must not add a second datetime parameter.
A safe fallback is to preserve the endpoint parameter and let Spark's retained residual predicate apply the pushed condition.
## Actual behavior
StacBatch.getItemLink preserves the endpoint query and then StacUtils.addFiltersToUrl unconditionally appends the pushed temporal filter.
On Sedona 1.9.1 and current master, this produces:
~~~text
...?datetime=<endpoint-range>&limit=10&datetime=<pushed-range>
~~~
Repeated scalar parameters are handled differently across STAC servers. A server may reject the request or honor only the first or last value. If it honors the pushed value and ignores the endpoint value, rows outside the source relation can survive Spark's residual filter.
The current call chain is:
- [StacBatch.getItemLink](https://github.com/apache/sedona/blob/ccee236e00184438708c4e88be59b47002759bea/spark/common/src/main/scala/org/apache/spark/sql/sedona_sql/io/stac/StacBatch.scala#L290-L320) preserves the endpoint query and calls addFiltersToUrl.
- [StacUtils.addFiltersToUrl](https://github.com/apache/sedona/blob/ccee236e00184438708c4e88be59b47002759bea/spark/common/src/main/scala/org/apache/spark/sql/sedona_sql/io/stac/StacUtils.scala#L477-L486) appends another datetime whenever a temporal filter is present.
## Steps to reproduce
~~~scala
import java.time.LocalDateTime
import org.apache.spark.sql.execution.datasource.stac.TemporalFilter
import org.apache.spark.sql.sedona_sql.io.stac.StacBatch
import org.apache.spark.sql.types.StructType
val batch = StacBatch(
null,
"",
"""{"links":[]}""",
StructType(Nil),
Map.empty,
None,
None,
None)
val pushedFilter = Some(
TemporalFilter.GreaterThanFilter(
"datetime",
LocalDateTime.parse("2025-03-06T00:00:00")))
val result = batch.getItemLink(
"https://example.test/items" +
"?datetime=2025-02-01T00:00:00Z/2025-02-28T23:59:59Z",
10,
None,
pushedFilter)
println(result)
~~~
Actual output:
~~~text
https://example.test/items?datetime=2025-02-01T00:00:00Z/2025-02-28T23:59:59Z&limit=10&datetime=2025-03-06T00:00:00.000000000Z/..
~~~
The endpoint range and pushed range are disjoint, so the correct result is empty. A server that honors only the final parameter can instead return March items, which satisfy the Spark residual predicate but violate the endpoint's February constraint.
## Affected versions
- Sedona 1.7.2 through 1.9.1
- Current master at ccee236e
In versions 1.7.2 through 1.9.0, existing queries are additionally followed by another ?limit=, producing a malformed form such as:
~~~text
...?datetime=<endpoint-range>?limit=10&datetime=<pushed-range>
~~~
Sedona 1.7.1 predates this filter-pushdown path.
## Impact
Depending on server behavior, scans can fail, lose remote pruning, or return rows outside the endpoint-defined source relation.
## Suggested fix
Before appending the pushed temporal filter:
1. Parse query parameters separately from the fragment and recognize percent-encoded parameter names.
2. If the endpoint already owns any datetime parameter, preserve it unchanged and do not append another one.
3. Let Spark's retained temporal predicate enforce the pushed condition.
Optionally, exactly one valid endpoint interval may be intersected with the pushed interval and serialized as one parameter. Repeated, empty, reversed, or unparsable endpoint values should remain unchanged rather than being removed or replaced.
Suggested regression cases:
- a valid endpoint interval plus a pushed predicate;
- malformed, empty, and repeated endpoint datetime values;
- existing query parameters and URL fragments;
- percent-encoded parameter names.
## Sedona version
1.9.1 and current master
## Apache Spark version
N/A; reproduces directly in shared URL construction
## API type
Scala
## Scala version
2.12 and 2.13
## JRE version
11+
## Environment
Local unit-level reproduction; runtime behavior affects any Spark STAC scan whose endpoint URL already contains datetime.
## Existing issues
- [x] I searched the existing issues and did not find a duplicate.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [BUG] STAC reader sends duplicate datetime parameters for constrained endpoints [sedona]
Posted by "jiayuasu (via GitHub)" <gi...@apache.org>.
jiayuasu closed issue #3277: [BUG] STAC reader sends duplicate datetime parameters for constrained endpoints
URL: https://github.com/apache/sedona/issues/3277
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [BUG] STAC reader sends duplicate datetime parameters for constrained endpoints [sedona]
Posted by "jiayuasu (via GitHub)" <gi...@apache.org>.
jiayuasu commented on issue #3277:
URL: https://github.com/apache/sedona/issues/3277#issuecomment-5389455933
@pttydou sounds good.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [BUG] STAC reader sends duplicate datetime parameters for constrained endpoints [sedona]
Posted by "pttydou (via GitHub)" <gi...@apache.org>.
pttydou commented on issue #3277:
URL: https://github.com/apache/sedona/issues/3277#issuecomment-5386959299
I'd like to work on this. On master at ccee236e, StacBatch.getItemLink preserves the endpoint query and StacUtils.addFiltersToUrl then appends the pushed datetime. I plan to keep any existing datetime parameter unchanged, suppress only the additional temporal parameter, preserve other query parameters and fragments, and add regression cases for encoded names, repeated or empty values, and coexistence with bbox.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org