You are viewing a plain text version of this content. The canonical link for it is here.
Posted to oak-issues@jackrabbit.apache.org by "Alejandro Moratinos (Jira)" <ji...@apache.org> on 2024/10/23 06:26:00 UTC
[jira] [Comment Edited] (OAK-11171) Create benchmark
[ https://issues.apache.org/jira/browse/OAK-11171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17890839#comment-17890839 ]
Alejandro Moratinos edited comment on OAK-11171 at 10/23/24 6:25 AM:
---------------------------------------------------------------------
Created a new benchmark test to test external user login added to local and external groups using cached membership provider.
Executed different set of benchmarks with different set of parameters. All test were executed with 30s of runtime and 15s of warmup with different concurrency threads setup. Cache expiration is set with a large value to ensure values are kept through benchmark execution.
300 users, 10 external groups, 10 local groups
{code:java}
## No cache
java -Xmx2048m -Dprofile=false -Druntime=30 -Dwarmup=15 -jar target/oak-benchmarks-1.71-SNAPSHOT.jar benchmark \
--dynamicMembership true --numberOfUsers 300 --concurrency 1,5,10,20,50 \
--numberOfGroups 10 --numberOfLocalGroups 10 --cacheExpiration 0 \
--report false CachedMembershipLoginTest Oak-Segment-Tar
Apache Jackrabbit Oak 1.71-SNAPSHOT
# CachedMembershipLoginTest C min 10% 50% 90% max N mean
Oak-Segment-Tar 1 4 4 4 5 33 6770 4
Oak-Segment-Tar 5 10 16 18 20 61 8431 18
Oak-Segment-Tar 10 25 36 38 41 81 7825 38
Oak-Segment-Tar 20 44 72 75 79 183 7747 78
Oak-Segment-Tar 50 46 172 184 330 808 7311 206
Unique users 300 out of total 300. Groups 10. Err 0. Seed 1729165820373
## Cache
java -Xmx2048m -Dprofile=false -Druntime=30 -Dwarmup=15 -jar target/oak-benchmarks-1.71-SNAPSHOT.jar benchmark \
--dynamicMembership true --numberOfUsers 300 --concurrency 1,5,10,20,50 \
--numberOfGroups 10 --numberOfLocalGroups 10 --cacheExpiration 10000000 \
--report false CachedMembershipLoginTest Oak-Segment-Tar
Apache Jackrabbit Oak 1.71-SNAPSHOT
# CachedMembershipLoginTest C min 10% 50% 90% max N mean
Oak-Segment-Tar 1 1 2 2 2 27 14615 2
Oak-Segment-Tar 5 3 7 8 10 86 18042 8
Oak-Segment-Tar 10 8 13 15 18 49 19051 16
Oak-Segment-Tar 20 17 29 32 37 194 17872 34
Oak-Segment-Tar 50 13 77 81 156 457 16045 94
Unique users 300 out of total 300. Groups 10. Err 0. Seed 1729234661753{code}
300 users, 100 groups and 10 local groups
{code:java}
## No Cache
java -Xmx2048m -Dprofile=false -Druntime=30 -Dwarmup=15 -jar target/oak-benchmarks-1.71-SNAPSHOT.jar benchmark \
--dynamicMembership true --numberOfUsers 300 --concurrency 1,5,10,20,50 \
--numberOfGroups 100 --numberOfLocalGroups 10 --cacheExpiration 0 \
--report false CachedMembershipLoginTest Oak-Segment-Tar
Apache Jackrabbit Oak 1.71-SNAPSHOT
# CachedMembershipLoginTest C min 10% 50% 90% max N mean
Oak-Segment-Tar 1 41 43 44 47 102 671 45
Oak-Segment-Tar 5 125 184 192 201 267 779 193
Oak-Segment-Tar 10 233 412 446 480 625 676 446
Oak-Segment-Tar 20 428 872 960 1037 1839 633 957
Oak-Segment-Tar 50 798 2122 2450 2718 5039 629 2473
Unique users 300 out of total 300. Groups 100. Err 0. Seed 1729167541161
## Cache
java -Xmx2048m -Dprofile=false -Druntime=30 -Dwarmup=15 -jar target/oak-benchmarks-1.71-SNAPSHOT.jar benchmark \
--dynamicMembership true --numberOfUsers 300 --concurrency 1,5,10,20,50 \
--numberOfGroups 100 --numberOfLocalGroups 10 --cacheExpiration 10000000 \
--report false CachedMembershipLoginTest Oak-Segment-Tar
Apache Jackrabbit Oak 1.71-SNAPSHOT
# CachedMembershipLoginTest C min 10% 50% 90% max N mean
Oak-Segment-Tar 1 12 13 13 15 46 2103 14
Oak-Segment-Tar 5 38 56 64 72 148 2325 65
Oak-Segment-Tar 10 60 118 129 143 373 2268 133
Oak-Segment-Tar 20 109 246 272 302 773 2120 284
Oak-Segment-Tar 50 326 648 699 1334 2716 1899 797
Unique users 300 out of total 300. Groups 100. Err 0. Seed 1729170222022{code}
100 users, 200 groups and 100 local groups
{code:java}
## No Cache
java -Xmx2048m -Dprofile=false -Druntime=30 -Dwarmup=15 -jar target/oak-benchmarks-1.71-SNAPSHOT.jar benchmark \
--dynamicMembership true --numberOfUsers 100 --concurrency 1,5,10,20,50 \
--numberOfGroups 200 --numberOfLocalGroups 100 --cacheExpiration 0 \
--report false CachedMembershipLoginTest Oak-Segment-Tar
Apache Jackrabbit Oak 1.71-SNAPSHOT
# CachedMembershipLoginTest C min 10% 50% 90% max N mean
Oak-Segment-Tar 1 358 367 375 385 409 80 375
Oak-Segment-Tar 5 2115 2370 2420 2463 2491 65 2412
Oak-Segment-Tar 10 5028 5395 5640 5805 5984 60 5611
Oak-Segment-Tar 20 10090 10304 11285 12178 12757 60 11262
Oak-Segment-Tar 50 14954 15543 28794 33096 34039 74 25654
Unique users 97 out of total 100. Groups 200. Err 0. Seed 1729235215305
## Cache
java -Xmx2048m -Dprofile=false -Druntime=30 -Dwarmup=15 -jar target/oak-benchmarks-1.71-SNAPSHOT.jar benchmark \
--dynamicMembership true --numberOfUsers 100 --concurrency 1,5,10,20,50 \
--numberOfGroups 200 --numberOfLocalGroups 100 --cacheExpiration 10000000 \
--report false CachedMembershipLoginTest Oak-Segment-Tar
Apache Jackrabbit Oak 1.71-SNAPSHOT
# CachedMembershipLoginTest C min 10% 50% 90% max N mean
Oak-Segment-Tar 1 25 26 28 461 573 158 190
Oak-Segment-Tar 5 63 117 128 147 2393 942 160
Oak-Segment-Tar 10 137 234 255 289 790 1107 272
Oak-Segment-Tar 20 233 468 536 976 2444 998 606
Oak-Segment-Tar 50 643 1317 1451 3534 7357 759 2012
Unique users 100 out of total 100. Groups 200. Err 0. Seed 1729235457138{code}
100 users, 1200 groups and 100 local groups
{code:java}
## No Cache
java -Xmx2048m -Dprofile=false -Druntime=30 -Dwarmup=15 -jar target/oak-benchmarks-1.71-SNAPSHOT.jar benchmark \
--dynamicMembership true --numberOfUsers 100 --concurrency 1,5,10,20,50 \
--numberOfGroups 1200 --numberOfLocalGroups 100 --cacheExpiration 0 \
--report false CachedMembershipLoginTest Oak-Segment-Tar
Apache Jackrabbit Oak 1.71-SNAPSHOT
# CachedMembershipLoginTest C min 10% 50% 90% max N mean
Oak-Segment-Tar 1 3123 3123 3134 3173 3185 13 3139
Oak-Segment-Tar 5 14616 14675 15289 15698 15744 15 15246
Oak-Segment-Tar 10 27534 32217 34787 37053 38323 20 34477
Oak-Segment-Tar 20 41631 45477 66119 69066 69123 20 62596
Oak-Segment-Tar 50 107535 114668 154058 175837 176512 50 150076
Unique users 69 out of total 100. Groups 1200. Err 0. Seed 1729171093374
## Cache
java -Xmx2048m -Dprofile=false -Druntime=30 -Dwarmup=15 -jar target/oak-benchmarks-1.71-SNAPSHOT.jar benchmark \
--dynamicMembership true --numberOfUsers 100 --concurrency 1,5,10,20,50 \
--numberOfGroups 1200 --numberOfLocalGroups 100 --cacheExpiration 10000000 \
--report false CachedMembershipLoginTest Oak-Segment-Tar
Apache Jackrabbit Oak 1.71-SNAPSHOT
# CachedMembershipLoginTest C min 10% 50% 90% max N mean
Oak-Segment-Tar 1 170 1067 2477 2753 2785 13 2354
Oak-Segment-Tar 5 736 778 14244 15009 15176 16 9877
Oak-Segment-Tar 10 1687 1764 34183 35392 35404 13 26713
Oak-Segment-Tar 20 3116 3353 59325 67689 71273 33 40481
Oak-Segment-Tar 50 8123 9187 39456 108810 117207 73 54874
Unique users 78 out of total 100. Groups 1200. Err 0. Seed 1729236152455{code}
To test a configuration with short cache time which is closest to reality executed a test with 30s cache expiration time
{code:java}
java -Xmx2048m -Dprofile=false -Druntime=30 -Dwarmup=15 -Druntime=40 -jar target/oak-benchmarks-1.71-SNAPSHOT.jar benchmark \
--dynamicMembership true --numberOfUsers 100 --concurrency 1,5,10,20,50 \
--numberOfGroups 1200 --numberOfLocalGroups 100 --cacheExpiration 30000 \
--report false CachedMembershipLoginTest Oak-Segment-Tar
Apache Jackrabbit Oak 1.71-SNAPSHOT
# CachedMembershipLoginTest C min 10% 50% 90% max N mean
Oak-Segment-Tar 1 182 2050 2528 2555 2589 17 2395
Oak-Segment-Tar 5 685 694 14276 14692 14834 17 12682
Oak-Segment-Tar 10 2009 2186 32604 34728 36387 25 26588
Oak-Segment-Tar 20 3008 3619 62627 67037 67603 24 52872
Oak-Segment-Tar 50 7188 10529 138560 167500 168827 56 124069
Unique users 72 out of total 100. Groups 1200. Err 0. Seed 1729171927509 {code}
In general the results obtained with the cache version outperforms the non-cached version seeing huge reduction in execution times.
was (Author: JIRAUSER307002):
Created a new benchmark test to test external user login added to local and external groups using cached membership provider.
Executed different set of benchmarks with different set of parameters. All test were executed with 30s of runtime and 15s of warmup with different concurrency threads setup. Cache expiration is set with a large value to ensure values are kept through benchmark execution.
## 300 users, 10 external groups, 10 local groups
{code:java}
## No cache
java -Xmx2048m -Dprofile=false -Druntime=30 -Dwarmup=15 -jar target/oak-benchmarks-1.71-SNAPSHOT.jar benchmark \
--dynamicMembership true --numberOfUsers 300 --concurrency 1,5,10,20,50 \
--numberOfGroups 10 --numberOfLocalGroups 10 --cacheExpiration 0 \
--report false CachedMembershipLoginTest Oak-Segment-Tar
Apache Jackrabbit Oak 1.71-SNAPSHOT
# CachedMembershipLoginTest C min 10% 50% 90% max N mean
Oak-Segment-Tar 1 4 4 4 5 33 6770 4
Oak-Segment-Tar 5 10 16 18 20 61 8431 18
Oak-Segment-Tar 10 25 36 38 41 81 7825 38
Oak-Segment-Tar 20 44 72 75 79 183 7747 78
Oak-Segment-Tar 50 46 172 184 330 808 7311 206
Unique users 300 out of total 300. Groups 10. Err 0. Seed 1729165820373
## Cache
java -Xmx2048m -Dprofile=false -Druntime=30 -Dwarmup=15 -jar target/oak-benchmarks-1.71-SNAPSHOT.jar benchmark \
--dynamicMembership true --numberOfUsers 300 --concurrency 1,5,10,20,50 \
--numberOfGroups 10 --numberOfLocalGroups 10 --cacheExpiration 10000000 \
--report false CachedMembershipLoginTest Oak-Segment-Tar
Apache Jackrabbit Oak 1.71-SNAPSHOT
# CachedMembershipLoginTest C min 10% 50% 90% max N mean
Oak-Segment-Tar 1 1 2 2 2 27 14615 2
Oak-Segment-Tar 5 3 7 8 10 86 18042 8
Oak-Segment-Tar 10 8 13 15 18 49 19051 16
Oak-Segment-Tar 20 17 29 32 37 194 17872 34
Oak-Segment-Tar 50 13 77 81 156 457 16045 94
Unique users 300 out of total 300. Groups 10. Err 0. Seed 1729234661753{code}
### 300 users, 100 groups and 10 local groups
{code:java}
## No Cache
java -Xmx2048m -Dprofile=false -Druntime=30 -Dwarmup=15 -jar target/oak-benchmarks-1.71-SNAPSHOT.jar benchmark \
--dynamicMembership true --numberOfUsers 300 --concurrency 1,5,10,20,50 \
--numberOfGroups 100 --numberOfLocalGroups 10 --cacheExpiration 0 \
--report false CachedMembershipLoginTest Oak-Segment-Tar
Apache Jackrabbit Oak 1.71-SNAPSHOT
# CachedMembershipLoginTest C min 10% 50% 90% max N mean
Oak-Segment-Tar 1 41 43 44 47 102 671 45
Oak-Segment-Tar 5 125 184 192 201 267 779 193
Oak-Segment-Tar 10 233 412 446 480 625 676 446
Oak-Segment-Tar 20 428 872 960 1037 1839 633 957
Oak-Segment-Tar 50 798 2122 2450 2718 5039 629 2473
Unique users 300 out of total 300. Groups 100. Err 0. Seed 1729167541161
## Cache
java -Xmx2048m -Dprofile=false -Druntime=30 -Dwarmup=15 -jar target/oak-benchmarks-1.71-SNAPSHOT.jar benchmark \
--dynamicMembership true --numberOfUsers 300 --concurrency 1,5,10,20,50 \
--numberOfGroups 100 --numberOfLocalGroups 10 --cacheExpiration 10000000 \
--report false CachedMembershipLoginTest Oak-Segment-Tar
Apache Jackrabbit Oak 1.71-SNAPSHOT
# CachedMembershipLoginTest C min 10% 50% 90% max N mean
Oak-Segment-Tar 1 12 13 13 15 46 2103 14
Oak-Segment-Tar 5 38 56 64 72 148 2325 65
Oak-Segment-Tar 10 60 118 129 143 373 2268 133
Oak-Segment-Tar 20 109 246 272 302 773 2120 284
Oak-Segment-Tar 50 326 648 699 1334 2716 1899 797
Unique users 300 out of total 300. Groups 100. Err 0. Seed 1729170222022{code}
## 100 users, 200 groups and 100 local groups
{code:java}
## No Cache
java -Xmx2048m -Dprofile=false -Druntime=30 -Dwarmup=15 -jar target/oak-benchmarks-1.71-SNAPSHOT.jar benchmark \
--dynamicMembership true --numberOfUsers 100 --concurrency 1,5,10,20,50 \
--numberOfGroups 200 --numberOfLocalGroups 100 --cacheExpiration 0 \
--report false CachedMembershipLoginTest Oak-Segment-Tar
Apache Jackrabbit Oak 1.71-SNAPSHOT
# CachedMembershipLoginTest C min 10% 50% 90% max N mean
Oak-Segment-Tar 1 358 367 375 385 409 80 375
Oak-Segment-Tar 5 2115 2370 2420 2463 2491 65 2412
Oak-Segment-Tar 10 5028 5395 5640 5805 5984 60 5611
Oak-Segment-Tar 20 10090 10304 11285 12178 12757 60 11262
Oak-Segment-Tar 50 14954 15543 28794 33096 34039 74 25654
Unique users 97 out of total 100. Groups 200. Err 0. Seed 1729235215305
## Cache
java -Xmx2048m -Dprofile=false -Druntime=30 -Dwarmup=15 -jar target/oak-benchmarks-1.71-SNAPSHOT.jar benchmark \
--dynamicMembership true --numberOfUsers 100 --concurrency 1,5,10,20,50 \
--numberOfGroups 200 --numberOfLocalGroups 100 --cacheExpiration 10000000 \
--report false CachedMembershipLoginTest Oak-Segment-Tar
Apache Jackrabbit Oak 1.71-SNAPSHOT
# CachedMembershipLoginTest C min 10% 50% 90% max N mean
Oak-Segment-Tar 1 25 26 28 461 573 158 190
Oak-Segment-Tar 5 63 117 128 147 2393 942 160
Oak-Segment-Tar 10 137 234 255 289 790 1107 272
Oak-Segment-Tar 20 233 468 536 976 2444 998 606
Oak-Segment-Tar 50 643 1317 1451 3534 7357 759 2012
Unique users 100 out of total 100. Groups 200. Err 0. Seed 1729235457138{code}
## 100 users, 1200 groups and 100 local groups
{code:java}
## No Cache
java -Xmx2048m -Dprofile=false -Druntime=30 -Dwarmup=15 -jar target/oak-benchmarks-1.71-SNAPSHOT.jar benchmark \
--dynamicMembership true --numberOfUsers 100 --concurrency 1,5,10,20,50 \
--numberOfGroups 1200 --numberOfLocalGroups 100 --cacheExpiration 0 \
--report false CachedMembershipLoginTest Oak-Segment-Tar
Apache Jackrabbit Oak 1.71-SNAPSHOT
# CachedMembershipLoginTest C min 10% 50% 90% max N mean
Oak-Segment-Tar 1 3123 3123 3134 3173 3185 13 3139
Oak-Segment-Tar 5 14616 14675 15289 15698 15744 15 15246
Oak-Segment-Tar 10 27534 32217 34787 37053 38323 20 34477
Oak-Segment-Tar 20 41631 45477 66119 69066 69123 20 62596
Oak-Segment-Tar 50 107535 114668 154058 175837 176512 50 150076
Unique users 69 out of total 100. Groups 1200. Err 0. Seed 1729171093374
## Cache
java -Xmx2048m -Dprofile=false -Druntime=30 -Dwarmup=15 -jar target/oak-benchmarks-1.71-SNAPSHOT.jar benchmark \
--dynamicMembership true --numberOfUsers 100 --concurrency 1,5,10,20,50 \
--numberOfGroups 1200 --numberOfLocalGroups 100 --cacheExpiration 10000000 \
--report false CachedMembershipLoginTest Oak-Segment-Tar
Apache Jackrabbit Oak 1.71-SNAPSHOT
# CachedMembershipLoginTest C min 10% 50% 90% max N mean
Oak-Segment-Tar 1 170 1067 2477 2753 2785 13 2354
Oak-Segment-Tar 5 736 778 14244 15009 15176 16 9877
Oak-Segment-Tar 10 1687 1764 34183 35392 35404 13 26713
Oak-Segment-Tar 20 3116 3353 59325 67689 71273 33 40481
Oak-Segment-Tar 50 8123 9187 39456 108810 117207 73 54874
Unique users 78 out of total 100. Groups 1200. Err 0. Seed 1729236152455{code}
To test a configuration with short cache time which is closest to reality executed a test with 30s cache expiration time
{code:java}
java -Xmx2048m -Dprofile=false -Druntime=30 -Dwarmup=15 -Druntime=40 -jar target/oak-benchmarks-1.71-SNAPSHOT.jar benchmark \
--dynamicMembership true --numberOfUsers 100 --concurrency 1,5,10,20,50 \
--numberOfGroups 1200 --numberOfLocalGroups 100 --cacheExpiration 30000 \
--report false CachedMembershipLoginTest Oak-Segment-Tar
Apache Jackrabbit Oak 1.71-SNAPSHOT
# CachedMembershipLoginTest C min 10% 50% 90% max N mean
Oak-Segment-Tar 1 182 2050 2528 2555 2589 17 2395
Oak-Segment-Tar 5 685 694 14276 14692 14834 17 12682
Oak-Segment-Tar 10 2009 2186 32604 34728 36387 25 26588
Oak-Segment-Tar 20 3008 3619 62627 67037 67603 24 52872
Oak-Segment-Tar 50 7188 10529 138560 167500 168827 56 124069
Unique users 72 out of total 100. Groups 1200. Err 0. Seed 1729171927509 {code}
In general the results obtained with the cache version outperforms the non-cached version seeing huge reduction in execution times.
> Create benchmark
> ----------------
>
> Key: OAK-11171
> URL: https://issues.apache.org/jira/browse/OAK-11171
> Project: Jackrabbit Oak
> Issue Type: Technical task
> Components: security
> Reporter: Alejandro Moratinos
> Assignee: Alejandro Moratinos
> Priority: Major
>
> Create or modify actual benchmarks to test external group caching feature
--
This message was sent by Atlassian Jira
(v8.20.10#820010)