[I] Prevent excessive dataset count queries when using large page sizes

["tenthe (via GitHub)" <gi...@apache.org>]

tenthe opened a new issue, #4567:
URL: https://github.com/apache/streampipes/issues/4567

   ### Body
   
   Prevent excessive dataset count queries when using large page sizes
   
   The maximum table page size was recently increased to 500. While this works well for most overview pages, it can cause performance issues in the dataset overview.
   
   The dataset overview automatically loads the **# Events (7d)** count for all datasets visible on the current page. When a page size of 500 is selected, opening the dataset overview can therefore trigger up to 500 count queries against the time-series storage.
   
   Currently, the dataset overview uses the globally configured table page size and automatically requests event counts for all visible datasets. This can create unnecessary load on the backend and the underlying storage system, especially in environments with many datasets.
   
   ### Expected Behavior
   
   Users should be able to browse large numbers of datasets without automatically triggering a large number of count queries.
   
   ### Possible Solution
   
   One possible solution would be to decouple automatic count loading from the selected page size.
   
   Potential approaches include:
   
   * Limiting automatic count loading to a smaller number of datasets (e.g. 25 or 50)
   * Loading counts on demand, like the option of **# Events (total)**
   * Users can manually load the number of events by clicking on the new button
   
   ### Discussion
   
   The proposed solution is only one possible approach. We should discuss whether there is a better solution to address the issue.
   
   Alternative approaches could include backend-side batching, execution limits, caching of count results, or other mechanisms to reduce the load caused by count requests while maintaining a good user experience.
   
   
   ### StreamPipes Committer
   
   I acknowledge that I am a maintainer/committer of the Apache StreamPipes project.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@streampipes.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Prevent excessive dataset count queries when using large page sizes

["SvenO3 (via GitHub)" <gi...@apache.org>]

SvenO3 closed issue #4567: Prevent excessive dataset count queries when using large page sizes
URL: https://github.com/apache/streampipes/issues/4567


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@streampipes.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org