You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@directory.apache.org by Emmanuel Lécharny <el...@gmail.com> on 2019/06/24 15:18:29 UTC
Question regarding pipeline performances
Hi !
I'm just wondering on which kind of machine are the pipelines running?
The tests are taking almost forever. For instance, a test were we add
1000 entries and search using an index takes 9 seconds on my computer
(ok, it's a fast one), and more 300 seconds on the pipeline..
...
[INFO] Running org.apache.directory.server.core.operations.search.SearchWithIndicesPersonIT
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 314.49 s - in org.apache.directory.server.core.operations.search.SearchWithIndicesPersonIT
...
That's more than 3O times slower !
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@directory.apache.org
For additional commands, e-mail: dev-help@directory.apache.org
Re: Question regarding pipeline performances
Posted by Emmanuel Lécharny <el...@gmail.com>.
Not too bad :-)
Linux Java 8 :
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 17:49 min
On 25/06/2019 12:41, Emmanuel Lécharny wrote:
> Ok, commited a big change: reverts are now processed within a global
> transaction, which saves a LOT of time. On my machine, building
> apacheds takes 12 mins vs 22 mins before this simple change.
>
> I'm waiting for the pipeline to be completed (probably after my lunch)
> but I expect it to be way faster.
>
>
> On 25/06/2019 10:13, Emmanuel Lécharny wrote:
>> Trying something more atm : adding txn around the revert operation.
>> That should speedup all the tests that uses the revert feature.
>>
>> On 25/06/2019 08:18, Emmanuel Lécharny wrote:
>>> A bit better (3x faster):
>>>
>>>
>>> [INFO] Running
>>> org.apache.directory.server.core.operations.search.SearchWithIndicesPersonIT
>>> [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time
>>> elapsed: 96.514 s - in
>>> org.apache.directory.server.core.operations.search.SearchWithIndicesPersonIT
>>>
>>>
>>> On 25/06/2019 03:05, Emmanuel Lécharny wrote:
>>>> Many thanks, Stefan.
>>>>
>>>> It makes sense now.
>>>>
>>>> I have modified the test that add 1000 entries before running so
>>>> that those additions are made within a single transaction. That
>>>> should speedup the addition considerably (on my laptop, the test
>>>> now takes 6 seconds instead of 10, and most of the time is spent
>>>> elsewhere - the entries addition takes less than 1 second compared
>>>> to 4 seconds -). I expect to see the test running in way less time
>>>> than 300 seconds now.
>>>>
>>>> I also have disabled the recording feature (we don't revert
>>>> anything), which brings another huge speedup: the test now runs in
>>>> 1.4 seconds on my computer
>>>>
>>>> Let's see what it gives in the pipeline result :-)
>>>>
>>>>
>>>> Side note: this is exactly what I want to do for the Fortress
>>>> tests: use external transaction to gather injections of data in
>>>> one single database update.
>>>>
>>>>
>>>> On 25/06/2019 00:04, Stefan Seelmann wrote:
>>>>> Hi Emmanuel,
>>>>>
>>>>> yes, the Jenkins machines are rather slow, especially the H2x and H3x
>>>>> nodes. I run lscpu and free -m on some, most show
>>>>>
>>>>> Intel(R) Xeon(R) CPU E5-2430 (from May 2012) or Intel(R) Xeon(R) CPU
>>>>> E5-2620 v2 (from Sept 2013) and 96 GB RAM
>>>>>
>>>>> But I assume those are highly overprovisioned VMs with spinning disks
>>>>> only. If you want more info you have to ask infra.
>>>>>
>>>>> See also the recent discussion with Shawn [1].
>>>>>
>>>>> But I don't mind, they are provided "for free" to the ASF and we
>>>>> can use
>>>>> them.
>>>>>
>>>>> Kind Regards,
>>>>> Stefan
>>>>>
>>>>> [1]
>>>>> https://lists.apache.org/thread.html/c67c09b0e53a15c2bc9fbaf5e664c6f3b1dc2c9c2358d646d4028292@%3Cdev.directory.apache.org%3E
>>>>>
>>>>>
>>>>> On 6/24/19 5:18 PM, Emmanuel Lécharny wrote:
>>>>>> Hi !
>>>>>>
>>>>>> I'm just wondering on which kind of machine are the pipelines
>>>>>> running?
>>>>>> The tests are taking almost forever. For instance, a test were we
>>>>>> add
>>>>>> 1000 entries and search using an index takes 9 seconds on my
>>>>>> computer
>>>>>> (ok, it's a fast one), and more 300 seconds on the pipeline..
>>>>>>
>>>>>> ...
>>>>>>
>>>>>> [INFO] Running
>>>>>> org.apache.directory.server.core.operations.search.SearchWithIndicesPersonIT
>>>>>>
>>>>>>
>>>>>> [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time
>>>>>> elapsed:
>>>>>> 314.49 s - in
>>>>>> org.apache.directory.server.core.operations.search.SearchWithIndicesPersonIT
>>>>>>
>>>>>>
>>>>>>
>>>>>> ...
>>>>>>
>>>>>>
>>>>>> That's more than 3O times slower !
>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>>
>>>>>> To unsubscribe, e-mail: dev-unsubscribe@directory.apache.org
>>>>>> For additional commands, e-mail: dev-help@directory.apache.org
>>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: dev-unsubscribe@directory.apache.org
>>>>> For additional commands, e-mail: dev-help@directory.apache.org
>>>>>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@directory.apache.org
For additional commands, e-mail: dev-help@directory.apache.org
Re: Question regarding pipeline performances
Posted by Emmanuel Lécharny <el...@gmail.com>.
Ok, commited a big change: reverts are now processed within a global
transaction, which saves a LOT of time. On my machine, building apacheds
takes 12 mins vs 22 mins before this simple change.
I'm waiting for the pipeline to be completed (probably after my lunch)
but I expect it to be way faster.
On 25/06/2019 10:13, Emmanuel Lécharny wrote:
> Trying something more atm : adding txn around the revert operation.
> That should speedup all the tests that uses the revert feature.
>
> On 25/06/2019 08:18, Emmanuel Lécharny wrote:
>> A bit better (3x faster):
>>
>>
>> [INFO] Running
>> org.apache.directory.server.core.operations.search.SearchWithIndicesPersonIT
>> [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time
>> elapsed: 96.514 s - in
>> org.apache.directory.server.core.operations.search.SearchWithIndicesPersonIT
>>
>>
>> On 25/06/2019 03:05, Emmanuel Lécharny wrote:
>>> Many thanks, Stefan.
>>>
>>> It makes sense now.
>>>
>>> I have modified the test that add 1000 entries before running so
>>> that those additions are made within a single transaction. That
>>> should speedup the addition considerably (on my laptop, the test now
>>> takes 6 seconds instead of 10, and most of the time is spent
>>> elsewhere - the entries addition takes less than 1 second compared
>>> to 4 seconds -). I expect to see the test running in way less time
>>> than 300 seconds now.
>>>
>>> I also have disabled the recording feature (we don't revert
>>> anything), which brings another huge speedup: the test now runs in
>>> 1.4 seconds on my computer
>>>
>>> Let's see what it gives in the pipeline result :-)
>>>
>>>
>>> Side note: this is exactly what I want to do for the Fortress tests:
>>> use external transaction to gather injections of data in one single
>>> database update.
>>>
>>>
>>> On 25/06/2019 00:04, Stefan Seelmann wrote:
>>>> Hi Emmanuel,
>>>>
>>>> yes, the Jenkins machines are rather slow, especially the H2x and H3x
>>>> nodes. I run lscpu and free -m on some, most show
>>>>
>>>> Intel(R) Xeon(R) CPU E5-2430 (from May 2012) or Intel(R) Xeon(R) CPU
>>>> E5-2620 v2 (from Sept 2013) and 96 GB RAM
>>>>
>>>> But I assume those are highly overprovisioned VMs with spinning disks
>>>> only. If you want more info you have to ask infra.
>>>>
>>>> See also the recent discussion with Shawn [1].
>>>>
>>>> But I don't mind, they are provided "for free" to the ASF and we
>>>> can use
>>>> them.
>>>>
>>>> Kind Regards,
>>>> Stefan
>>>>
>>>> [1]
>>>> https://lists.apache.org/thread.html/c67c09b0e53a15c2bc9fbaf5e664c6f3b1dc2c9c2358d646d4028292@%3Cdev.directory.apache.org%3E
>>>>
>>>>
>>>> On 6/24/19 5:18 PM, Emmanuel Lécharny wrote:
>>>>> Hi !
>>>>>
>>>>> I'm just wondering on which kind of machine are the pipelines
>>>>> running?
>>>>> The tests are taking almost forever. For instance, a test were we add
>>>>> 1000 entries and search using an index takes 9 seconds on my computer
>>>>> (ok, it's a fast one), and more 300 seconds on the pipeline..
>>>>>
>>>>> ...
>>>>>
>>>>> [INFO] Running
>>>>> org.apache.directory.server.core.operations.search.SearchWithIndicesPersonIT
>>>>>
>>>>>
>>>>> [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time
>>>>> elapsed:
>>>>> 314.49 s - in
>>>>> org.apache.directory.server.core.operations.search.SearchWithIndicesPersonIT
>>>>>
>>>>>
>>>>>
>>>>> ...
>>>>>
>>>>>
>>>>> That's more than 3O times slower !
>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: dev-unsubscribe@directory.apache.org
>>>>> For additional commands, e-mail: dev-help@directory.apache.org
>>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: dev-unsubscribe@directory.apache.org
>>>> For additional commands, e-mail: dev-help@directory.apache.org
>>>>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@directory.apache.org
For additional commands, e-mail: dev-help@directory.apache.org
Re: Question regarding pipeline performances
Posted by Emmanuel Lécharny <el...@gmail.com>.
Trying something more atm : adding txn around the revert operation. That
should speedup all the tests that uses the revert feature.
On 25/06/2019 08:18, Emmanuel Lécharny wrote:
> A bit better (3x faster):
>
>
> [INFO] Running
> org.apache.directory.server.core.operations.search.SearchWithIndicesPersonIT
> [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
> 96.514 s - in
> org.apache.directory.server.core.operations.search.SearchWithIndicesPersonIT
>
>
> On 25/06/2019 03:05, Emmanuel Lécharny wrote:
>> Many thanks, Stefan.
>>
>> It makes sense now.
>>
>> I have modified the test that add 1000 entries before running so that
>> those additions are made within a single transaction. That should
>> speedup the addition considerably (on my laptop, the test now takes 6
>> seconds instead of 10, and most of the time is spent elsewhere - the
>> entries addition takes less than 1 second compared to 4 seconds -). I
>> expect to see the test running in way less time than 300 seconds now.
>>
>> I also have disabled the recording feature (we don't revert
>> anything), which brings another huge speedup: the test now runs in
>> 1.4 seconds on my computer
>>
>> Let's see what it gives in the pipeline result :-)
>>
>>
>> Side note: this is exactly what I want to do for the Fortress tests:
>> use external transaction to gather injections of data in one single
>> database update.
>>
>>
>> On 25/06/2019 00:04, Stefan Seelmann wrote:
>>> Hi Emmanuel,
>>>
>>> yes, the Jenkins machines are rather slow, especially the H2x and H3x
>>> nodes. I run lscpu and free -m on some, most show
>>>
>>> Intel(R) Xeon(R) CPU E5-2430 (from May 2012) or Intel(R) Xeon(R) CPU
>>> E5-2620 v2 (from Sept 2013) and 96 GB RAM
>>>
>>> But I assume those are highly overprovisioned VMs with spinning disks
>>> only. If you want more info you have to ask infra.
>>>
>>> See also the recent discussion with Shawn [1].
>>>
>>> But I don't mind, they are provided "for free" to the ASF and we can
>>> use
>>> them.
>>>
>>> Kind Regards,
>>> Stefan
>>>
>>> [1]
>>> https://lists.apache.org/thread.html/c67c09b0e53a15c2bc9fbaf5e664c6f3b1dc2c9c2358d646d4028292@%3Cdev.directory.apache.org%3E
>>>
>>>
>>> On 6/24/19 5:18 PM, Emmanuel Lécharny wrote:
>>>> Hi !
>>>>
>>>> I'm just wondering on which kind of machine are the pipelines running?
>>>> The tests are taking almost forever. For instance, a test were we add
>>>> 1000 entries and search using an index takes 9 seconds on my computer
>>>> (ok, it's a fast one), and more 300 seconds on the pipeline..
>>>>
>>>> ...
>>>>
>>>> [INFO] Running
>>>> org.apache.directory.server.core.operations.search.SearchWithIndicesPersonIT
>>>>
>>>>
>>>> [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
>>>> 314.49 s - in
>>>> org.apache.directory.server.core.operations.search.SearchWithIndicesPersonIT
>>>>
>>>>
>>>>
>>>> ...
>>>>
>>>>
>>>> That's more than 3O times slower !
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: dev-unsubscribe@directory.apache.org
>>>> For additional commands, e-mail: dev-help@directory.apache.org
>>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: dev-unsubscribe@directory.apache.org
>>> For additional commands, e-mail: dev-help@directory.apache.org
>>>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@directory.apache.org
For additional commands, e-mail: dev-help@directory.apache.org
Re: Question regarding pipeline performances
Posted by Emmanuel Lécharny <el...@gmail.com>.
A bit better (3x faster):
[INFO] Running org.apache.directory.server.core.operations.search.SearchWithIndicesPersonIT
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 96.514 s - in org.apache.directory.server.core.operations.search.SearchWithIndicesPersonIT
On 25/06/2019 03:05, Emmanuel Lécharny wrote:
> Many thanks, Stefan.
>
> It makes sense now.
>
> I have modified the test that add 1000 entries before running so that
> those additions are made within a single transaction. That should
> speedup the addition considerably (on my laptop, the test now takes 6
> seconds instead of 10, and most of the time is spent elsewhere - the
> entries addition takes less than 1 second compared to 4 seconds -). I
> expect to see the test running in way less time than 300 seconds now.
>
> I also have disabled the recording feature (we don't revert anything),
> which brings another huge speedup: the test now runs in 1.4 seconds
> on my computer
>
> Let's see what it gives in the pipeline result :-)
>
>
> Side note: this is exactly what I want to do for the Fortress tests:
> use external transaction to gather injections of data in one single
> database update.
>
>
> On 25/06/2019 00:04, Stefan Seelmann wrote:
>> Hi Emmanuel,
>>
>> yes, the Jenkins machines are rather slow, especially the H2x and H3x
>> nodes. I run lscpu and free -m on some, most show
>>
>> Intel(R) Xeon(R) CPU E5-2430 (from May 2012) or Intel(R) Xeon(R) CPU
>> E5-2620 v2 (from Sept 2013) and 96 GB RAM
>>
>> But I assume those are highly overprovisioned VMs with spinning disks
>> only. If you want more info you have to ask infra.
>>
>> See also the recent discussion with Shawn [1].
>>
>> But I don't mind, they are provided "for free" to the ASF and we can use
>> them.
>>
>> Kind Regards,
>> Stefan
>>
>> [1]
>> https://lists.apache.org/thread.html/c67c09b0e53a15c2bc9fbaf5e664c6f3b1dc2c9c2358d646d4028292@%3Cdev.directory.apache.org%3E
>>
>>
>> On 6/24/19 5:18 PM, Emmanuel Lécharny wrote:
>>> Hi !
>>>
>>> I'm just wondering on which kind of machine are the pipelines running?
>>> The tests are taking almost forever. For instance, a test were we add
>>> 1000 entries and search using an index takes 9 seconds on my computer
>>> (ok, it's a fast one), and more 300 seconds on the pipeline..
>>>
>>> ...
>>>
>>> [INFO] Running
>>> org.apache.directory.server.core.operations.search.SearchWithIndicesPersonIT
>>>
>>>
>>> [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
>>> 314.49 s - in
>>> org.apache.directory.server.core.operations.search.SearchWithIndicesPersonIT
>>>
>>>
>>>
>>> ...
>>>
>>>
>>> That's more than 3O times slower !
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: dev-unsubscribe@directory.apache.org
>>> For additional commands, e-mail: dev-help@directory.apache.org
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@directory.apache.org
>> For additional commands, e-mail: dev-help@directory.apache.org
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@directory.apache.org
For additional commands, e-mail: dev-help@directory.apache.org
Re: Question regarding pipeline performances
Posted by Emmanuel Lécharny <el...@gmail.com>.
Many thanks, Stefan.
It makes sense now.
I have modified the test that add 1000 entries before running so that
those additions are made within a single transaction. That should
speedup the addition considerably (on my laptop, the test now takes 6
seconds instead of 10, and most of the time is spent elsewhere - the
entries addition takes less than 1 second compared to 4 seconds -). I
expect to see the test running in way less time than 300 seconds now.
I also have disabled the recording feature (we don't revert anything),
which brings another huge speedup: the test now runs in 1.4 seconds on
my computer
Let's see what it gives in the pipeline result :-)
Side note: this is exactly what I want to do for the Fortress tests: use
external transaction to gather injections of data in one single
database update.
On 25/06/2019 00:04, Stefan Seelmann wrote:
> Hi Emmanuel,
>
> yes, the Jenkins machines are rather slow, especially the H2x and H3x
> nodes. I run lscpu and free -m on some, most show
>
> Intel(R) Xeon(R) CPU E5-2430 (from May 2012) or Intel(R) Xeon(R) CPU
> E5-2620 v2 (from Sept 2013) and 96 GB RAM
>
> But I assume those are highly overprovisioned VMs with spinning disks
> only. If you want more info you have to ask infra.
>
> See also the recent discussion with Shawn [1].
>
> But I don't mind, they are provided "for free" to the ASF and we can use
> them.
>
> Kind Regards,
> Stefan
>
> [1]
> https://lists.apache.org/thread.html/c67c09b0e53a15c2bc9fbaf5e664c6f3b1dc2c9c2358d646d4028292@%3Cdev.directory.apache.org%3E
>
> On 6/24/19 5:18 PM, Emmanuel Lécharny wrote:
>> Hi !
>>
>> I'm just wondering on which kind of machine are the pipelines running?
>> The tests are taking almost forever. For instance, a test were we add
>> 1000 entries and search using an index takes 9 seconds on my computer
>> (ok, it's a fast one), and more 300 seconds on the pipeline..
>>
>> ...
>>
>> [INFO] Running
>> org.apache.directory.server.core.operations.search.SearchWithIndicesPersonIT
>>
>> [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
>> 314.49 s - in
>> org.apache.directory.server.core.operations.search.SearchWithIndicesPersonIT
>>
>>
>> ...
>>
>>
>> That's more than 3O times slower !
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@directory.apache.org
>> For additional commands, e-mail: dev-help@directory.apache.org
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@directory.apache.org
> For additional commands, e-mail: dev-help@directory.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@directory.apache.org
For additional commands, e-mail: dev-help@directory.apache.org
Re: Question regarding pipeline performances
Posted by Stefan Seelmann <ma...@stefan-seelmann.de>.
Hi Emmanuel,
yes, the Jenkins machines are rather slow, especially the H2x and H3x
nodes. I run lscpu and free -m on some, most show
Intel(R) Xeon(R) CPU E5-2430 (from May 2012) or Intel(R) Xeon(R) CPU
E5-2620 v2 (from Sept 2013) and 96 GB RAM
But I assume those are highly overprovisioned VMs with spinning disks
only. If you want more info you have to ask infra.
See also the recent discussion with Shawn [1].
But I don't mind, they are provided "for free" to the ASF and we can use
them.
Kind Regards,
Stefan
[1]
https://lists.apache.org/thread.html/c67c09b0e53a15c2bc9fbaf5e664c6f3b1dc2c9c2358d646d4028292@%3Cdev.directory.apache.org%3E
On 6/24/19 5:18 PM, Emmanuel Lécharny wrote:
> Hi !
>
> I'm just wondering on which kind of machine are the pipelines running?
> The tests are taking almost forever. For instance, a test were we add
> 1000 entries and search using an index takes 9 seconds on my computer
> (ok, it's a fast one), and more 300 seconds on the pipeline..
>
> ...
>
> [INFO] Running
> org.apache.directory.server.core.operations.search.SearchWithIndicesPersonIT
>
> [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
> 314.49 s - in
> org.apache.directory.server.core.operations.search.SearchWithIndicesPersonIT
>
>
> ...
>
>
> That's more than 3O times slower !
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@directory.apache.org
> For additional commands, e-mail: dev-help@directory.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@directory.apache.org
For additional commands, e-mail: dev-help@directory.apache.org