[jira] [Commented] (AIRAVATA-3322) Add paging to experiment statistics

["ASF subversion and git services (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/AIRAVATA-3322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17362070#comment-17362070 ] 

ASF subversion and git services commented on AIRAVATA-3322:
-----------------------------------------------------------

Commit 2400943bf7673347eae7c23cecc81502775af4af in airavata's branch refs/heads/airavata-3322 from Marcus Christie
[ https://gitbox.apache.org/repos/asf?p=airavata.git;h=2400943 ]

AIRAVATA-3322 Separate queries for counts and data


> Add paging to experiment statistics
> -----------------------------------
>
>                 Key: AIRAVATA-3322
>                 URL: https://issues.apache.org/jira/browse/AIRAVATA-3322
>             Project: Airavata
>          Issue Type: New Feature
>          Components: Django Portal, Registry API
>            Reporter: Marcus Christie
>            Assignee: Marcus Christie
>            Priority: Major
>
> The getExperimentStatistics API method works well for small numbers of experiments but doesn't scale well to thousands of experiments, which some popular gateways like [seagrid.org|https://seagrid.org] seeing this level of activity. For example, for SEAGrid we had to up the Apache/Django timeout to 5 minutes to load statistics for a few months for one compute resource, and even then it didn't always load.
> I added slow query logging in the MariaDB but that turned up nothing so I don't believe the queries need to be tuned. I think the main culprit is that thousands of ExperimentSummary instances are getting populated and returned by the method. It would be better to page in experiments and only return the first, say, 50 experiments in each status category.
> Currently the ExperimentSummaries are loaded for each ExperimentState and they are counted. Instead, we'd need two queries per state, one to load the count as a scalar and another to load a page's worth of ExperimentSummaries.
> To preserve backwards compatibility, it might be best to just make a new method and then in time phase out the old one.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)