You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@allura.apache.org by Dave Brondsema <br...@users.sf.net> on 2014/08/28 16:24:31 UTC
[allura:tickets] #7644 Make /nf/admin/new_projects faster
---
** [tickets:#7644] Make /nf/admin/new_projects faster**
**Status:** open
**Milestone:** forge-sep-19
**Labels:** performance
**Created:** Thu Aug 28, 2014 02:24 PM UTC by Dave Brondsema
**Last Updated:** Thu Aug 28, 2014 02:24 PM UTC
**Owner:** nobody
/nf/admin/new_projects can take a very long time when showing hundreds of projects. I don't have timermiddleware stats, that'll probably have to be reproduced on a sandbox for detailed analysis of what is slow. I do have data from New Relic which indicates hundreds of mongo queries are being run and it looks like they are invoked from within the template. Perhaps we are doing separate queries for each project and can do a batch query instead?
---
Sent from sourceforge.net because dev@allura.apache.org is subscribed to https://sourceforge.net/p/allura/tickets/
To unsubscribe from further messages, a project admin can change settings at https://sourceforge.net/p/allura/admin/tickets/options. Or, if this is a mailing list, you can unsubscribe from the mailing list.
[allura:tickets] Re: #7644 Make /nf/admin/new_projects faster
Posted by Jacob Moorman <mo...@users.sf.net>.
Admins are needed as it allows correlation of username to project name, and
grouping of projects by username.
On Mon, Sep 22, 2014 at 5:23 AM, Igor Bondarenko <je...@users.sf.net>
wrote:
>
> - *status*: in-progress --> code-review
> - *Comment*:
>
> I've generated ~2200 projects with scripts/perf/generate-projects.py to
> test this.
>
> Closed #658. je/42cc_7644
>
> The bottleneck is call to get project admins in the template, which
> generates a lot of queries to lookup ProjectRoles. Roles are cached, but
> for every call to admins() there's actually a cache miss :)
>
> There are two reasons for that:
>
> 1. we ask for roles for each project exactly once within every
> request, and role cache is not preserved between different requests
> 2. users_with_named_role() (which Project.admins() uses) does not
> actually use a cache and always generates a new query
>
> I've added cache pre-population before rendering template, which generates
> only one query to get all roles for all given projects and fixed reason (2)
> above.
>
> On my test data response time improved from ~12-14sec to ~9-11sec.
>
> That's better, but not as much as I want it to be :)
>
> Removing calls to project.admins() entirely gives response time of
> ~4-5sec for the same data. So another option could be to initially render
> page without displaying admins. Instead we can add a button "Show admins"
> for each project which will make an ajax call to get admins or even just a
> link to a page, which already contains a list of admins (don't sure such
> page exists, though).
>
> Any thoughts?
> ------------------------------
>
> * [tickets:#7644] <http://sourceforge.net/p/allura/tickets/7644> Make
> /nf/admin/new_projects faster*
>
> *Status:* code-review
> *Milestone:* forge-sep-19
> *Labels:* performance 42cc
> *Created:* Thu Aug 28, 2014 02:24 PM UTC by Dave Brondsema
> *Last Updated:* Fri Sep 19, 2014 06:04 AM UTC
> *Owner:* Igor Bondarenko
>
> /nf/admin/new_projects can take a very long time when showing hundreds of
> projects. I don't have timermiddleware stats, that'll probably have to be
> reproduced on a sandbox for detailed analysis of what is slow. I do have
> data from New Relic which indicates hundreds of mongo queries are being run
> and it looks like they are invoked from within the template. Perhaps we are
> doing separate queries for each project and can do a batch query instead?
> ------------------------------
>
> Sent from sourceforge.net because you indicated interest in
> https://sourceforge.net/p/allura/tickets/7644/
>
> To unsubscribe from further messages, please visit
> https://sourceforge.net/auth/subscriptions/
>
---
** [tickets:#7644] Make /nf/admin/new_projects faster**
**Status:** code-review
**Milestone:** forge-sep-19
**Labels:** performance 42cc
**Created:** Thu Aug 28, 2014 02:24 PM UTC by Dave Brondsema
**Last Updated:** Mon Sep 22, 2014 09:23 AM UTC
**Owner:** Igor Bondarenko
/nf/admin/new_projects can take a very long time when showing hundreds of projects. I don't have timermiddleware stats, that'll probably have to be reproduced on a sandbox for detailed analysis of what is slow. I do have data from New Relic which indicates hundreds of mongo queries are being run and it looks like they are invoked from within the template. Perhaps we are doing separate queries for each project and can do a batch query instead?
---
Sent from sourceforge.net because dev@allura.apache.org is subscribed to http://sourceforge.net/p/allura/tickets/
To unsubscribe from further messages, a project admin can change settings at http://sourceforge.net/p/allura/admin/tickets/options. Or, if this is a mailing list, you can unsubscribe from the mailing list.
[allura:tickets] #7644 Make /nf/admin/new_projects faster
Posted by Igor Bondarenko <je...@users.sf.net>.
- **status**: in-progress --> code-review
- **Comment**:
I've generated ~2200 projects with `scripts/perf/generate-projects.py` to test this.
Closed #658. `je/42cc_7644`
The bottleneck is call to get project admins in the template, which generates a lot of queries to lookup `ProjectRoles`. Roles are cached, but for every call to `admins()` there's actually a cache miss :)
There are two reasons for that:
1. we ask for roles for each project exactly once within every request, and role cache is not preserved between different requests
2. `users_with_named_role()` (which `Project.admins()` uses) does not actually use a cache and always generates a new query
I've added cache pre-population before rendering template, which generates only one query to get all roles for all given projects and fixed reason (2) above.
On my test data response time improved from ~12-14sec to ~9-11sec.
That's better, but not as much as I want it to be :)
Removing calls to `project.admins()` entirely gives response time of ~4-5sec for the same data. So another option could be to initially render page without displaying admins. Instead we can add a button "Show admins" for each project which will make an ajax call to get admins or even just a link to a page, which already contains a list of admins (don't sure such page exists, though).
Any thoughts?
---
** [tickets:#7644] Make /nf/admin/new_projects faster**
**Status:** code-review
**Milestone:** forge-sep-19
**Labels:** performance 42cc
**Created:** Thu Aug 28, 2014 02:24 PM UTC by Dave Brondsema
**Last Updated:** Fri Sep 19, 2014 06:04 AM UTC
**Owner:** Igor Bondarenko
/nf/admin/new_projects can take a very long time when showing hundreds of projects. I don't have timermiddleware stats, that'll probably have to be reproduced on a sandbox for detailed analysis of what is slow. I do have data from New Relic which indicates hundreds of mongo queries are being run and it looks like they are invoked from within the template. Perhaps we are doing separate queries for each project and can do a batch query instead?
---
Sent from sourceforge.net because dev@allura.apache.org is subscribed to https://sourceforge.net/p/allura/tickets/
To unsubscribe from further messages, a project admin can change settings at https://sourceforge.net/p/allura/admin/tickets/options. Or, if this is a mailing list, you can unsubscribe from the mailing list.
[allura:tickets] #7644 Make /nf/admin/new_projects faster
Posted by Igor Bondarenko <je...@users.sf.net>.
- **labels**: performance --> performance, 42cc
- **status**: open --> in-progress
- **assigned_to**: Igor Bondarenko
---
** [tickets:#7644] Make /nf/admin/new_projects faster**
**Status:** in-progress
**Milestone:** forge-sep-19
**Labels:** performance 42cc
**Created:** Thu Aug 28, 2014 02:24 PM UTC by Dave Brondsema
**Last Updated:** Mon Sep 08, 2014 03:57 PM UTC
**Owner:** Igor Bondarenko
/nf/admin/new_projects can take a very long time when showing hundreds of projects. I don't have timermiddleware stats, that'll probably have to be reproduced on a sandbox for detailed analysis of what is slow. I do have data from New Relic which indicates hundreds of mongo queries are being run and it looks like they are invoked from within the template. Perhaps we are doing separate queries for each project and can do a batch query instead?
---
Sent from sourceforge.net because dev@allura.apache.org is subscribed to https://sourceforge.net/p/allura/tickets/
To unsubscribe from further messages, a project admin can change settings at https://sourceforge.net/p/allura/admin/tickets/options. Or, if this is a mailing list, you can unsubscribe from the mailing list.
[allura:tickets] #7644 Make /nf/admin/new_projects faster
Posted by Dave Brondsema <br...@users.sf.net>.
- **Size**: --> 4
---
** [tickets:#7644] Make /nf/admin/new_projects faster**
**Status:** open
**Milestone:** forge-sep-19
**Labels:** performance
**Created:** Thu Aug 28, 2014 02:24 PM UTC by Dave Brondsema
**Last Updated:** Thu Aug 28, 2014 02:24 PM UTC
**Owner:** nobody
/nf/admin/new_projects can take a very long time when showing hundreds of projects. I don't have timermiddleware stats, that'll probably have to be reproduced on a sandbox for detailed analysis of what is slow. I do have data from New Relic which indicates hundreds of mongo queries are being run and it looks like they are invoked from within the template. Perhaps we are doing separate queries for each project and can do a batch query instead?
---
Sent from sourceforge.net because dev@allura.apache.org is subscribed to https://sourceforge.net/p/allura/tickets/
To unsubscribe from further messages, a project admin can change settings at https://sourceforge.net/p/allura/admin/tickets/options. Or, if this is a mailing list, you can unsubscribe from the mailing list.
[allura:tickets] #7644 Make /nf/admin/new_projects faster
Posted by Dave Brondsema <br...@users.sf.net>.
- **status**: code-review --> closed
- **QA**: Dave Brondsema
---
** [tickets:#7644] Make /nf/admin/new_projects faster**
**Status:** closed
**Milestone:** forge-sep-19
**Labels:** performance 42cc
**Created:** Thu Aug 28, 2014 02:24 PM UTC by Dave Brondsema
**Last Updated:** Mon Sep 22, 2014 09:23 AM UTC
**Owner:** Igor Bondarenko
/nf/admin/new_projects can take a very long time when showing hundreds of projects. I don't have timermiddleware stats, that'll probably have to be reproduced on a sandbox for detailed analysis of what is slow. I do have data from New Relic which indicates hundreds of mongo queries are being run and it looks like they are invoked from within the template. Perhaps we are doing separate queries for each project and can do a batch query instead?
---
Sent from sourceforge.net because dev@allura.apache.org is subscribed to https://sourceforge.net/p/allura/tickets/
To unsubscribe from further messages, a project admin can change settings at https://sourceforge.net/p/allura/admin/tickets/options. Or, if this is a mailing list, you can unsubscribe from the mailing list.