JIRA scalability issues

[Marvin Humphrey <ma...@rectangular.com>]

On Fri, May 25, 2012 at 12:57 AM, Steve Loughran
<st...@gmail.com> wrote:
> It is becoming a bit of a SPOF, isn't it?

What has changed about our JIRA instance is both its size and its increasing
integration into the workflows of some projects.  It always was a SPOF, but
now we are feeling it more because it is getting harder to keep online, harder
to troubleshoot, and more sorely missed when it is unavailable.

Unfortunately, JIRA is not a distributed application.  There is one massive
database.  There is one process.

Resource utilization exceeding the capabilities of a single machine isn't a
problem yet.  Traffic isn't too heavy -- 3-4 hits a second on average, from
what I hear.

But when you have to reindex, it takes hours, and when you have to restart, it
takes several minutes.  That sluggishness severely impedes troubleshooting --
a problem that could be isolated in minutes with a smaller JIRA instance and
near-instantaneous restarts takes hours to solve with a database as big as
ours.

The JIRA project import pains which have affected the Flex podling are related
to this.  JIRA is a pain to upgrade because it is big and finicky about JREs
and operating systems, and because our infamous custom checkbox module makes
things tricky.  There was some question about whether version discrepancies
were aggravating the Flex import failures, but nobody wanted to go through the
upgrade to see if that helped.

Then, JIRA security vulnerabilities forced Infra's hand this week.  The
consequence was a lot of downtime and a lot of frustration and stress for
everybody while Infra has worked through the upgrade with lots of 5-minute
restart cycles.  It has been at least as painful as everyone had anticipated.

This list isn't the venue for solving these problems, but as JIRA scalability
issues are impacting podlings and projects, I think it's wise for us to take
the situation into account.

Marvin Humphrey

---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org

Re: JIRA scalability issues

[Steve Loughran <st...@gmail.com>]

On 26 May 2012 21:50, Marvin Humphrey <ma...@rectangular.com> wrote:

> On Fri, May 25, 2012 at 12:57 AM, Steve Loughran
> <st...@gmail.com> wrote:
> > It is becoming a bit of a SPOF, isn't it?
>
> What has changed about our JIRA instance is both its size and its
> increasing
> integration into the workflows of some projects.  It always was a SPOF, but
> now we are feeling it more because it is getting harder to keep online,
> harder
> to troubleshoot, and more sorely missed when it is unavailable.
>

I think it's the integration that's changed. Instead of the mail list being
the focus of collaboration, and the bug tracker a todo list, now the
tracker is the centre of collaboration and how things are shared.


>
> Unfortunately, JIRA is not a distributed application.  There is one massive
> database.  There is one process.
>

Oh, the irony of developing datacentre-scale nosql apps using Jira as a dev
tool overwhelms me.


>
> Resource utilization exceeding the capabilities of a single machine isn't a
> problem yet.  Traffic isn't too heavy -- 3-4 hits a second on average, from
> what I hear.
>
> But when you have to reindex, it takes hours, and when you have to
> restart, it
> takes several minutes.  That sluggishness severely impedes troubleshooting
> --
> a problem that could be isolated in minutes with a smaller JIRA instance
> and
> near-instantaneous restarts takes hours to solve with a database as big as
> ours.
>

I see the issue. You could shard, but that would increase the workload of
user management (unless you switch to LDAP-auth), plus it you cant
cross-link JIRA issues.


>
> The JIRA project import pains which have affected the Flex podling are
> related
> to this.  JIRA is a pain to upgrade because it is big and finicky about
> JREs
> and operating systems, and because our infamous custom checkbox module
> makes
> things tricky.  There was some question about whether version discrepancies
> were aggravating the Flex import failures, but nobody wanted to go through
> the
> upgrade to see if that helped.
>
> Then, JIRA security vulnerabilities forced Infra's hand this week.  The
> consequence was a lot of downtime and a lot of frustration and stress for
> everybody while Infra has worked through the upgrade with lots of 5-minute
> restart cycles.  It has been at least as painful as everyone had
> anticipated.
>
>
I see those security emails too, and given the way JIRA was used to gain
access to the apache infrastructure in the past, I understand precisely why
those updates are needed.