You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@archiva.apache.org by Sascha Vogt <sa...@gmail.com> on 2014/05/15 10:24:13 UTC
Degrading upload performance during day
Hi all,
we're experiencing a strange behavior in our Archiva instance (latest
2.1.0-SNAPSHOT):
During the day, the upload performance from Jenkins degrades heavily. We
have a job which uploads a few hundred artifacts and in the morning this
takes between 50 seconds and 1:10. During the day this gets slower and
slower (around 1 PM we're at 3-5 minutes, at 4 PM we're usually between
8-11 minutes). This is with no other uploads running in parallel and I
don't see any load on the Archiva server.
Then after a night has passed, we're back to 50 seconds and the same
starts over.
Is there anything which could cause this? I checked, the repo scanning
queue and index queue are 0 when I start my tests. I also don't see any
big network or IO load which could explain it. And as this is now for
over a week, ever day the same, I suspect it is something inside Archvia.
Are there any jobs running at night / once a day, which could explain
the behavior? Something within Jackrabbit? The index? Is it that we run
the repo crons at night (1 am) and they do some sort of cleanup?
Greetings
-Sascha-
Re: Degrading upload performance during day
Posted by Sascha Vogt <sa...@gmail.com>.
Hi,
please find 10 stacks from this morning (upload from Jenkins to Archiva
took 1 min) and 10 stacks from just now (upload from Jenkins to Archiva
took 5:42)
https://dl.dropboxusercontent.com/u/44225754/stacks.zip
Hope you find something valuable
Greetings
-Sascha-
Am 20.05.2014 09:15, schrieb Sascha Vogt:
> Hi,
>
> sure, I'll get a few this evening. And to note: Not the whole system is
> slow, just uploads via Maven from our Jenkins.
>
> Searches in the UI are "normal", also builds / downloads don't seem to
> be particularly slow.
>
> Maybe Jackrabbit does some cleanup jobs during the night / once a day? A
> colleague which works on a project also using Jackrabbit as datastore
> said something that there are very small index files which eventually
> get merged to some bigger index. Could that be it? I think our SAN
> behaves particularly bad when a lot of small files are involved.
>
> Anyway, I just counted the files in the index directory and took the
> first 10 thread dumps. Will do the same later
>
> Greetings
> -Sascha-
>
> Am 20.05.2014 01:52, schrieb Olivier Lamy:
>> Hi,
>> Really hard to know exactly what happen.
>> Do you have time to get a jstack while the system is slow?
>>
>> Olivier
>>
>> On 15 May 2014 18:24, Sascha Vogt <sa...@gmail.com> wrote:
>>> Hi all,
>>>
>>> we're experiencing a strange behavior in our Archiva instance (latest
>>> 2.1.0-SNAPSHOT):
>>>
>>> During the day, the upload performance from Jenkins degrades heavily. We
>>> have a job which uploads a few hundred artifacts and in the morning this
>>> takes between 50 seconds and 1:10. During the day this gets slower and
>>> slower (around 1 PM we're at 3-5 minutes, at 4 PM we're usually between
>>> 8-11 minutes). This is with no other uploads running in parallel and I
>>> don't see any load on the Archiva server.
>>>
>>> Then after a night has passed, we're back to 50 seconds and the same
>>> starts over.
>>>
>>> Is there anything which could cause this? I checked, the repo scanning
>>> queue and index queue are 0 when I start my tests. I also don't see any
>>> big network or IO load which could explain it. And as this is now for
>>> over a week, ever day the same, I suspect it is something inside Archvia.
>>>
>>> Are there any jobs running at night / once a day, which could explain
>>> the behavior? Something within Jackrabbit? The index? Is it that we run
>>> the repo crons at night (1 am) and they do some sort of cleanup?
>>>
>>> Greetings
>>> -Sascha-
Re: Degrading upload performance during day
Posted by Sascha Vogt <sa...@gmail.com>.
Hi,
sure, I'll get a few this evening. And to note: Not the whole system is
slow, just uploads via Maven from our Jenkins.
Searches in the UI are "normal", also builds / downloads don't seem to
be particularly slow.
Maybe Jackrabbit does some cleanup jobs during the night / once a day? A
colleague which works on a project also using Jackrabbit as datastore
said something that there are very small index files which eventually
get merged to some bigger index. Could that be it? I think our SAN
behaves particularly bad when a lot of small files are involved.
Anyway, I just counted the files in the index directory and took the
first 10 thread dumps. Will do the same later
Greetings
-Sascha-
Am 20.05.2014 01:52, schrieb Olivier Lamy:
> Hi,
> Really hard to know exactly what happen.
> Do you have time to get a jstack while the system is slow?
>
> Olivier
>
> On 15 May 2014 18:24, Sascha Vogt <sa...@gmail.com> wrote:
>> Hi all,
>>
>> we're experiencing a strange behavior in our Archiva instance (latest
>> 2.1.0-SNAPSHOT):
>>
>> During the day, the upload performance from Jenkins degrades heavily. We
>> have a job which uploads a few hundred artifacts and in the morning this
>> takes between 50 seconds and 1:10. During the day this gets slower and
>> slower (around 1 PM we're at 3-5 minutes, at 4 PM we're usually between
>> 8-11 minutes). This is with no other uploads running in parallel and I
>> don't see any load on the Archiva server.
>>
>> Then after a night has passed, we're back to 50 seconds and the same
>> starts over.
>>
>> Is there anything which could cause this? I checked, the repo scanning
>> queue and index queue are 0 when I start my tests. I also don't see any
>> big network or IO load which could explain it. And as this is now for
>> over a week, ever day the same, I suspect it is something inside Archvia.
>>
>> Are there any jobs running at night / once a day, which could explain
>> the behavior? Something within Jackrabbit? The index? Is it that we run
>> the repo crons at night (1 am) and they do some sort of cleanup?
>>
>> Greetings
>> -Sascha-
>
>
>
Re: Degrading upload performance during day
Posted by Olivier Lamy <ol...@apache.org>.
Hi,
Really hard to know exactly what happen.
Do you have time to get a jstack while the system is slow?
Olivier
On 15 May 2014 18:24, Sascha Vogt <sa...@gmail.com> wrote:
> Hi all,
>
> we're experiencing a strange behavior in our Archiva instance (latest
> 2.1.0-SNAPSHOT):
>
> During the day, the upload performance from Jenkins degrades heavily. We
> have a job which uploads a few hundred artifacts and in the morning this
> takes between 50 seconds and 1:10. During the day this gets slower and
> slower (around 1 PM we're at 3-5 minutes, at 4 PM we're usually between
> 8-11 minutes). This is with no other uploads running in parallel and I
> don't see any load on the Archiva server.
>
> Then after a night has passed, we're back to 50 seconds and the same
> starts over.
>
> Is there anything which could cause this? I checked, the repo scanning
> queue and index queue are 0 when I start my tests. I also don't see any
> big network or IO load which could explain it. And as this is now for
> over a week, ever day the same, I suspect it is something inside Archvia.
>
> Are there any jobs running at night / once a day, which could explain
> the behavior? Something within Jackrabbit? The index? Is it that we run
> the repo crons at night (1 am) and they do some sort of cleanup?
>
> Greetings
> -Sascha-
--
Olivier Lamy
Ecetera: http://ecetera.com.au
http://twitter.com/olamy | http://linkedin.com/in/olamy