You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "shauryachats (via GitHub)" <gi...@apache.org> on 2025/07/24 01:12:33 UTC
[I] Proposal for BrokerResponse compatible timeseries API [pinot]
shauryachats opened a new issue, #16415:
URL: https://github.com/apache/pinot/issues/16415
## Current Issues
The current timeseries API (`GET /timeseries/api/v1/query_range`) is compatible with Prometheus but is not Pinot-friendly for a couple of reasons:
- The query execution statistics cannot be accommodated in the current timeseries API.
- All the necessary params have to be encoded in a `GET` request, causing inadvertent query truncation (due to limited data length), URL exposure and other problems.
- The exception handling is limited by the current timeseries API and cannot make use of the detailed error codes for `BrokerResponse`.
- `BrokerResponse` supports features like cursor-based pagination and streaming output which are not directly supported in the current timeseries API.
- The timeseries system operates in complete isolation from standard query processing. By making a `BrokerResponse` compatible API for timeseries, it is a step forward towards consolidation of these different flows.
## Proposal
Due to the aforementioned reasons, a new timeseries API `POST /query/timeseries` is proposed which accepts:
```
{
"query": "fetch {table='installs_REALTIME'} | sum{country_id, device_id}",
"language": "m3ql"
"start_time": 1720527600,
"end_time": 1720527720
}
```
and returns a `BrokerResponse` with the introduction of a new optional `extensions` key.
```
{
"resultTable": {
"dataSchema": {
"columnNames": ["country_id", "device_id", "value"],
"columnDataTypes": ["INT", "INT", "LONG_ARRAY"]
},
"rows": [
["US", "iOS", [10, 20, 30]],
["US", "android", [30, 40, 50]],
...
]
},
"numGroupsLimitReached": true,
"extensions": {
"timeSeries": {
"timeBuckets": [1720527600, 1720527660, 1720527720]
}
}
}
```
The structure ensures that the existing `BrokerResponse` does not break for existing use-cases, and also accommodates a generic `extension` object which can extend `BrokerResponse` for timeseries and other future uses.
## Alternative Proposal
Another structure of the response can be:
```
{
"resultTable": {
"dataSchema": {
"columnNames": ["country_id", "device_id", "timebuckets", "value"],
"columnDataTypes": ["INT", "INT", "LONG_ARRAY", "LONG_ARRAY"]
},
"rows": [
["US", "iOS", [1720527600, 1720527660, 1720527720], [10, 20, 30]],
["US", "android", [1720527600, 1720527660, 1720527720], [30, 40, 50]],
...
]
},
"numGroupsLimitReached": true,
}
```
which fits into the existing `BrokerResponse` model but is suboptimal since it involves `timebuckets` to be repeated for each row, inflating the response size from the timeseries API.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Proposal for BrokerResponse compatible timeseries API [pinot]
Posted by "raghavyadav01 (via GitHub)" <gi...@apache.org>.
raghavyadav01 commented on issue #16415:
URL: https://github.com/apache/pinot/issues/16415#issuecomment-3130310103
@shauryachats Are we planning to remove the existing Prometheus-compatible API request/response? We would still need the existing API's request/response to support prometheus clients.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Proposal for BrokerResponse compatible timeseries API [pinot]
Posted by "shauryachats (via GitHub)" <gi...@apache.org>.
shauryachats commented on issue #16415:
URL: https://github.com/apache/pinot/issues/16415#issuecomment-3130459052
@raghavyadav01 We are not planning to deprecate the Prometheus-compatible API currently; that will still be kept for backward compatibility. The BrokerResponse compatible API would be the preferred API for timeseries queries, though (we would move the Controller UI to be powered from this to also highlight query stats, amongst other things).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Proposal for BrokerResponse compatible timeseries API [pinot]
Posted by "shauryachats (via GitHub)" <gi...@apache.org>.
shauryachats closed issue #16415: Proposal for BrokerResponse compatible timeseries API
URL: https://github.com/apache/pinot/issues/16415
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Proposal for BrokerResponse compatible timeseries API [pinot]
Posted by "ankitsultana (via GitHub)" <gi...@apache.org>.
ankitsultana commented on issue #16415:
URL: https://github.com/apache/pinot/issues/16415#issuecomment-3114353557
I was thinking again about this after our discussion yesterday, I think it might make sense to:
* Make the default format as the one you outlined in "Alternative Proposal"
* Add a URL queryParam to change response format to the one which uses extensions.
The reason for making "Alternative Proposal" the default is that it aligns exactly with Pinot's response format **and** it doesn't enforce that the number of time buckets will be the same for all series. Meaning in the future we can break away from this contract more easily and functionally we can do everything that the Prometheus API can do.
For better performance, users can use the queryParam to choose alternative formats like the extensions approach.
Also nit: instead of snake case in `start_time` use camelCase for consistency. But overall looks good to me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Proposal for BrokerResponse compatible timeseries API [pinot]
Posted by "raghavyadav01 (via GitHub)" <gi...@apache.org>.
raghavyadav01 commented on issue #16415:
URL: https://github.com/apache/pinot/issues/16415#issuecomment-3134193675
Thanks @shauryachats . I think the proposal looks good to me and make sense. I think the alternate proposal as Ankit suggested can be default and user can override with query_param.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org