You are viewing a plain text version of this content. The canonical link for it is here.
Posted to oak-issues@jackrabbit.apache.org by "Alejandro Moratinos (Jira)" <ji...@apache.org> on 2024/10/23 06:26:00 UTC

[jira] [Comment Edited] (OAK-11171) Create benchmark

    [ https://issues.apache.org/jira/browse/OAK-11171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17890839#comment-17890839 ] 

Alejandro Moratinos edited comment on OAK-11171 at 10/23/24 6:25 AM:
---------------------------------------------------------------------

Created a new benchmark test to test external user login added to local and external groups using cached membership provider. 

Executed different set of benchmarks with different set of parameters. All test were executed with 30s of runtime and 15s of warmup with different concurrency threads setup. Cache expiration is set with a large value to ensure values are kept through benchmark execution.

 

300 users, 10 external groups, 10 local groups

 
{code:java}
## No cache
java -Xmx2048m -Dprofile=false -Druntime=30 -Dwarmup=15 -jar target/oak-benchmarks-1.71-SNAPSHOT.jar benchmark \
--dynamicMembership true --numberOfUsers 300 --concurrency 1,5,10,20,50  \
--numberOfGroups 10 --numberOfLocalGroups 10 --cacheExpiration 0 \
--report false CachedMembershipLoginTest Oak-Segment-Tar
Apache Jackrabbit Oak 1.71-SNAPSHOT
# CachedMembershipLoginTest        C     min     10%     50%     90%     max     N       mean
Oak-Segment-Tar                    1       4       4       4       5     33    6770       4
Oak-Segment-Tar                    5      10      16      18      20     61    8431      18
Oak-Segment-Tar                   10      25      36      38      41     81    7825      38
Oak-Segment-Tar                   20      44      72      75      79    183    7747      78
Oak-Segment-Tar                   50      46     172     184     330    808    7311     206
Unique users 300 out of total 300. Groups 10. Err 0. Seed 1729165820373

## Cache
java -Xmx2048m -Dprofile=false -Druntime=30 -Dwarmup=15 -jar target/oak-benchmarks-1.71-SNAPSHOT.jar benchmark \
--dynamicMembership true --numberOfUsers 300 --concurrency 1,5,10,20,50  \
--numberOfGroups 10 --numberOfLocalGroups 10 --cacheExpiration 10000000 \
--report false CachedMembershipLoginTest Oak-Segment-Tar
Apache Jackrabbit Oak 1.71-SNAPSHOT
# CachedMembershipLoginTest        C     min     10%     50%     90%     max     N       mean
Oak-Segment-Tar                    1       1       2       2       2     27   14615       2
Oak-Segment-Tar                    5       3       7       8      10     86   18042       8
Oak-Segment-Tar                   10       8      13      15      18     49   19051      16
Oak-Segment-Tar                   20      17      29      32      37    194   17872      34
Oak-Segment-Tar                   50      13      77      81     156    457   16045      94
Unique users 300 out of total 300. Groups 10. Err 0. Seed 1729234661753{code}
 

300 users, 100 groups and 10 local groups
{code:java}
## No Cache
java -Xmx2048m -Dprofile=false -Druntime=30 -Dwarmup=15 -jar target/oak-benchmarks-1.71-SNAPSHOT.jar benchmark \
--dynamicMembership true --numberOfUsers 300 --concurrency 1,5,10,20,50  \
--numberOfGroups 100 --numberOfLocalGroups 10 --cacheExpiration 0 \
--report false CachedMembershipLoginTest Oak-Segment-Tar
Apache Jackrabbit Oak 1.71-SNAPSHOT
# CachedMembershipLoginTest        C     min     10%     50%     90%     max     N       mean
Oak-Segment-Tar                    1      41      43      44      47    102     671      45
Oak-Segment-Tar                    5     125     184     192     201    267     779     193
Oak-Segment-Tar                   10     233     412     446     480    625     676     446
Oak-Segment-Tar                   20     428     872     960    1037   1839     633     957
Oak-Segment-Tar                   50     798    2122    2450    2718   5039     629    2473
Unique users 300 out of total 300. Groups 100. Err 0. Seed 1729167541161 

## Cache
java -Xmx2048m -Dprofile=false -Druntime=30 -Dwarmup=15 -jar target/oak-benchmarks-1.71-SNAPSHOT.jar benchmark \
--dynamicMembership true --numberOfUsers 300 --concurrency 1,5,10,20,50  \
--numberOfGroups 100 --numberOfLocalGroups 10 --cacheExpiration 10000000 \
--report false CachedMembershipLoginTest Oak-Segment-Tar 
Apache Jackrabbit Oak 1.71-SNAPSHOT
# CachedMembershipLoginTest        C     min     10%     50%     90%     max     N       mean
Oak-Segment-Tar                    1      12      13      13      15     46    2103      14
Oak-Segment-Tar                    5      38      56      64      72    148    2325      65
Oak-Segment-Tar                   10      60     118     129     143    373    2268     133
Oak-Segment-Tar                   20     109     246     272     302    773    2120     284
Oak-Segment-Tar                   50     326     648     699    1334   2716    1899     797
Unique users 300 out of total 300. Groups 100. Err 0. Seed 1729170222022{code}
 

100 users, 200 groups and 100 local groups
{code:java}
## No Cache
java -Xmx2048m -Dprofile=false -Druntime=30 -Dwarmup=15 -jar target/oak-benchmarks-1.71-SNAPSHOT.jar benchmark \
--dynamicMembership true --numberOfUsers 100 --concurrency 1,5,10,20,50  \
--numberOfGroups 200 --numberOfLocalGroups 100 --cacheExpiration 0 \
--report false CachedMembershipLoginTest Oak-Segment-Tar
Apache Jackrabbit Oak 1.71-SNAPSHOT
# CachedMembershipLoginTest        C     min     10%     50%     90%     max     N       mean
Oak-Segment-Tar                    1     358     367     375     385    409      80     375
Oak-Segment-Tar                    5    2115    2370    2420    2463   2491      65    2412
Oak-Segment-Tar                   10    5028    5395    5640    5805   5984      60    5611
Oak-Segment-Tar                   20   10090   10304   11285   12178  12757      60   11262
Oak-Segment-Tar                   50   14954   15543   28794   33096  34039      74   25654
Unique users 97 out of total 100. Groups 200. Err 0. Seed 1729235215305

## Cache
java -Xmx2048m -Dprofile=false -Druntime=30 -Dwarmup=15 -jar target/oak-benchmarks-1.71-SNAPSHOT.jar benchmark \
--dynamicMembership true --numberOfUsers 100 --concurrency 1,5,10,20,50  \
--numberOfGroups 200 --numberOfLocalGroups 100 --cacheExpiration 10000000 \
--report false CachedMembershipLoginTest Oak-Segment-Tar
Apache Jackrabbit Oak 1.71-SNAPSHOT 
# CachedMembershipLoginTest        C     min     10%     50%     90%     max     N       mean
Oak-Segment-Tar                    1      25      26      28     461    573     158     190
Oak-Segment-Tar                    5      63     117     128     147   2393     942     160
Oak-Segment-Tar                   10     137     234     255     289    790    1107     272
Oak-Segment-Tar                   20     233     468     536     976   2444     998     606
Oak-Segment-Tar                   50     643    1317    1451    3534   7357     759    2012
Unique users 100 out of total 100. Groups 200. Err 0. Seed 1729235457138{code}
 

100 users, 1200 groups and 100 local groups
{code:java}
## No Cache
java -Xmx2048m -Dprofile=false -Druntime=30 -Dwarmup=15  -jar target/oak-benchmarks-1.71-SNAPSHOT.jar benchmark \
--dynamicMembership true --numberOfUsers 100 --concurrency 1,5,10,20,50  \
--numberOfGroups 1200 --numberOfLocalGroups 100 --cacheExpiration 0 \
--report false CachedMembershipLoginTest Oak-Segment-Tar
Apache Jackrabbit Oak 1.71-SNAPSHOT
# CachedMembershipLoginTest        C     min     10%     50%     90%     max     N       mean
Oak-Segment-Tar                    1    3123    3123    3134    3173   3185      13    3139
Oak-Segment-Tar                    5   14616   14675   15289   15698  15744      15   15246
Oak-Segment-Tar                   10   27534   32217   34787   37053  38323      20   34477
Oak-Segment-Tar                   20   41631   45477   66119   69066  69123      20   62596
Oak-Segment-Tar                   50  107535  114668  154058  175837 176512      50  150076
Unique users 69 out of total 100. Groups 1200. Err 0. Seed 1729171093374 

## Cache
java -Xmx2048m -Dprofile=false -Druntime=30 -Dwarmup=15  -jar target/oak-benchmarks-1.71-SNAPSHOT.jar benchmark \
--dynamicMembership true --numberOfUsers 100 --concurrency 1,5,10,20,50  \
--numberOfGroups 1200 --numberOfLocalGroups 100 --cacheExpiration 10000000 \
--report false CachedMembershipLoginTest Oak-Segment-Tar
Apache Jackrabbit Oak 1.71-SNAPSHOT
# CachedMembershipLoginTest        C     min     10%     50%     90%     max     N       mean
Oak-Segment-Tar                    1     170    1067    2477    2753   2785      13    2354
Oak-Segment-Tar                    5     736     778   14244   15009  15176      16    9877
Oak-Segment-Tar                   10    1687    1764   34183   35392  35404      13   26713
Oak-Segment-Tar                   20    3116    3353   59325   67689  71273      33   40481
Oak-Segment-Tar                   50    8123    9187   39456  108810 117207      73   54874
Unique users 78 out of total 100. Groups 1200. Err 0. Seed 1729236152455{code}
 

To test a configuration with short cache time which is closest to reality executed a test with 30s cache expiration time
 
{code:java}
java -Xmx2048m -Dprofile=false -Druntime=30 -Dwarmup=15 -Druntime=40 -jar target/oak-benchmarks-1.71-SNAPSHOT.jar benchmark \
--dynamicMembership true --numberOfUsers 100 --concurrency 1,5,10,20,50  \
--numberOfGroups 1200 --numberOfLocalGroups 100 --cacheExpiration 30000 \
--report false CachedMembershipLoginTest Oak-Segment-Tar
Apache Jackrabbit Oak 1.71-SNAPSHOT
# CachedMembershipLoginTest        C     min     10%     50%     90%     max     N       mean
Oak-Segment-Tar                    1     182    2050    2528    2555   2589      17    2395
Oak-Segment-Tar                    5     685     694   14276   14692  14834      17   12682
Oak-Segment-Tar                   10    2009    2186   32604   34728  36387      25   26588
Oak-Segment-Tar                   20    3008    3619   62627   67037  67603      24   52872
Oak-Segment-Tar                   50    7188   10529  138560  167500 168827      56  124069
Unique users 72 out of total 100. Groups 1200. Err 0. Seed 1729171927509 {code}
 
In general the results obtained with the cache version outperforms the non-cached version seeing huge reduction in execution times. 


was (Author: JIRAUSER307002):
Created a new benchmark test to test external user login added to local and external groups using cached membership provider. 

Executed different set of benchmarks with different set of parameters. All test were executed with 30s of runtime and 15s of warmup with different concurrency threads setup. Cache expiration is set with a large value to ensure values are kept through benchmark execution.

 

## 300 users, 10 external groups, 10 local groups

 
{code:java}
## No cache
java -Xmx2048m -Dprofile=false -Druntime=30 -Dwarmup=15 -jar target/oak-benchmarks-1.71-SNAPSHOT.jar benchmark \
--dynamicMembership true --numberOfUsers 300 --concurrency 1,5,10,20,50  \
--numberOfGroups 10 --numberOfLocalGroups 10 --cacheExpiration 0 \
--report false CachedMembershipLoginTest Oak-Segment-Tar
Apache Jackrabbit Oak 1.71-SNAPSHOT
# CachedMembershipLoginTest        C     min     10%     50%     90%     max     N       mean
Oak-Segment-Tar                    1       4       4       4       5     33    6770       4
Oak-Segment-Tar                    5      10      16      18      20     61    8431      18
Oak-Segment-Tar                   10      25      36      38      41     81    7825      38
Oak-Segment-Tar                   20      44      72      75      79    183    7747      78
Oak-Segment-Tar                   50      46     172     184     330    808    7311     206
Unique users 300 out of total 300. Groups 10. Err 0. Seed 1729165820373

## Cache
java -Xmx2048m -Dprofile=false -Druntime=30 -Dwarmup=15 -jar target/oak-benchmarks-1.71-SNAPSHOT.jar benchmark \
--dynamicMembership true --numberOfUsers 300 --concurrency 1,5,10,20,50  \
--numberOfGroups 10 --numberOfLocalGroups 10 --cacheExpiration 10000000 \
--report false CachedMembershipLoginTest Oak-Segment-Tar
Apache Jackrabbit Oak 1.71-SNAPSHOT
# CachedMembershipLoginTest        C     min     10%     50%     90%     max     N       mean
Oak-Segment-Tar                    1       1       2       2       2     27   14615       2
Oak-Segment-Tar                    5       3       7       8      10     86   18042       8
Oak-Segment-Tar                   10       8      13      15      18     49   19051      16
Oak-Segment-Tar                   20      17      29      32      37    194   17872      34
Oak-Segment-Tar                   50      13      77      81     156    457   16045      94
Unique users 300 out of total 300. Groups 10. Err 0. Seed 1729234661753{code}
 

 
### 300 users, 100 groups and 10 local groups
{code:java}
## No Cache
java -Xmx2048m -Dprofile=false -Druntime=30 -Dwarmup=15 -jar target/oak-benchmarks-1.71-SNAPSHOT.jar benchmark \
--dynamicMembership true --numberOfUsers 300 --concurrency 1,5,10,20,50  \
--numberOfGroups 100 --numberOfLocalGroups 10 --cacheExpiration 0 \
--report false CachedMembershipLoginTest Oak-Segment-Tar
Apache Jackrabbit Oak 1.71-SNAPSHOT
# CachedMembershipLoginTest        C     min     10%     50%     90%     max     N       mean
Oak-Segment-Tar                    1      41      43      44      47    102     671      45
Oak-Segment-Tar                    5     125     184     192     201    267     779     193
Oak-Segment-Tar                   10     233     412     446     480    625     676     446
Oak-Segment-Tar                   20     428     872     960    1037   1839     633     957
Oak-Segment-Tar                   50     798    2122    2450    2718   5039     629    2473
Unique users 300 out of total 300. Groups 100. Err 0. Seed 1729167541161 

## Cache
java -Xmx2048m -Dprofile=false -Druntime=30 -Dwarmup=15 -jar target/oak-benchmarks-1.71-SNAPSHOT.jar benchmark \
--dynamicMembership true --numberOfUsers 300 --concurrency 1,5,10,20,50  \
--numberOfGroups 100 --numberOfLocalGroups 10 --cacheExpiration 10000000 \
--report false CachedMembershipLoginTest Oak-Segment-Tar 
Apache Jackrabbit Oak 1.71-SNAPSHOT
# CachedMembershipLoginTest        C     min     10%     50%     90%     max     N       mean
Oak-Segment-Tar                    1      12      13      13      15     46    2103      14
Oak-Segment-Tar                    5      38      56      64      72    148    2325      65
Oak-Segment-Tar                   10      60     118     129     143    373    2268     133
Oak-Segment-Tar                   20     109     246     272     302    773    2120     284
Oak-Segment-Tar                   50     326     648     699    1334   2716    1899     797
Unique users 300 out of total 300. Groups 100. Err 0. Seed 1729170222022{code}
## 100 users, 200 groups and 100 local groups
{code:java}
## No Cache
java -Xmx2048m -Dprofile=false -Druntime=30 -Dwarmup=15 -jar target/oak-benchmarks-1.71-SNAPSHOT.jar benchmark \
--dynamicMembership true --numberOfUsers 100 --concurrency 1,5,10,20,50  \
--numberOfGroups 200 --numberOfLocalGroups 100 --cacheExpiration 0 \
--report false CachedMembershipLoginTest Oak-Segment-Tar
Apache Jackrabbit Oak 1.71-SNAPSHOT
# CachedMembershipLoginTest        C     min     10%     50%     90%     max     N       mean
Oak-Segment-Tar                    1     358     367     375     385    409      80     375
Oak-Segment-Tar                    5    2115    2370    2420    2463   2491      65    2412
Oak-Segment-Tar                   10    5028    5395    5640    5805   5984      60    5611
Oak-Segment-Tar                   20   10090   10304   11285   12178  12757      60   11262
Oak-Segment-Tar                   50   14954   15543   28794   33096  34039      74   25654
Unique users 97 out of total 100. Groups 200. Err 0. Seed 1729235215305

## Cache
java -Xmx2048m -Dprofile=false -Druntime=30 -Dwarmup=15 -jar target/oak-benchmarks-1.71-SNAPSHOT.jar benchmark \
--dynamicMembership true --numberOfUsers 100 --concurrency 1,5,10,20,50  \
--numberOfGroups 200 --numberOfLocalGroups 100 --cacheExpiration 10000000 \
--report false CachedMembershipLoginTest Oak-Segment-Tar
Apache Jackrabbit Oak 1.71-SNAPSHOT 
# CachedMembershipLoginTest        C     min     10%     50%     90%     max     N       mean
Oak-Segment-Tar                    1      25      26      28     461    573     158     190
Oak-Segment-Tar                    5      63     117     128     147   2393     942     160
Oak-Segment-Tar                   10     137     234     255     289    790    1107     272
Oak-Segment-Tar                   20     233     468     536     976   2444     998     606
Oak-Segment-Tar                   50     643    1317    1451    3534   7357     759    2012
Unique users 100 out of total 100. Groups 200. Err 0. Seed 1729235457138{code}
## 100 users, 1200 groups and 100 local groups
{code:java}
## No Cache
java -Xmx2048m -Dprofile=false -Druntime=30 -Dwarmup=15  -jar target/oak-benchmarks-1.71-SNAPSHOT.jar benchmark \
--dynamicMembership true --numberOfUsers 100 --concurrency 1,5,10,20,50  \
--numberOfGroups 1200 --numberOfLocalGroups 100 --cacheExpiration 0 \
--report false CachedMembershipLoginTest Oak-Segment-Tar
Apache Jackrabbit Oak 1.71-SNAPSHOT
# CachedMembershipLoginTest        C     min     10%     50%     90%     max     N       mean
Oak-Segment-Tar                    1    3123    3123    3134    3173   3185      13    3139
Oak-Segment-Tar                    5   14616   14675   15289   15698  15744      15   15246
Oak-Segment-Tar                   10   27534   32217   34787   37053  38323      20   34477
Oak-Segment-Tar                   20   41631   45477   66119   69066  69123      20   62596
Oak-Segment-Tar                   50  107535  114668  154058  175837 176512      50  150076
Unique users 69 out of total 100. Groups 1200. Err 0. Seed 1729171093374 

## Cache
java -Xmx2048m -Dprofile=false -Druntime=30 -Dwarmup=15  -jar target/oak-benchmarks-1.71-SNAPSHOT.jar benchmark \
--dynamicMembership true --numberOfUsers 100 --concurrency 1,5,10,20,50  \
--numberOfGroups 1200 --numberOfLocalGroups 100 --cacheExpiration 10000000 \
--report false CachedMembershipLoginTest Oak-Segment-Tar
Apache Jackrabbit Oak 1.71-SNAPSHOT
# CachedMembershipLoginTest        C     min     10%     50%     90%     max     N       mean
Oak-Segment-Tar                    1     170    1067    2477    2753   2785      13    2354
Oak-Segment-Tar                    5     736     778   14244   15009  15176      16    9877
Oak-Segment-Tar                   10    1687    1764   34183   35392  35404      13   26713
Oak-Segment-Tar                   20    3116    3353   59325   67689  71273      33   40481
Oak-Segment-Tar                   50    8123    9187   39456  108810 117207      73   54874
Unique users 78 out of total 100. Groups 1200. Err 0. Seed 1729236152455{code}
To test a configuration with short cache time which is closest to reality executed a test with 30s cache expiration time
 
{code:java}
java -Xmx2048m -Dprofile=false -Druntime=30 -Dwarmup=15 -Druntime=40 -jar target/oak-benchmarks-1.71-SNAPSHOT.jar benchmark \
--dynamicMembership true --numberOfUsers 100 --concurrency 1,5,10,20,50  \
--numberOfGroups 1200 --numberOfLocalGroups 100 --cacheExpiration 30000 \
--report false CachedMembershipLoginTest Oak-Segment-Tar
Apache Jackrabbit Oak 1.71-SNAPSHOT
# CachedMembershipLoginTest        C     min     10%     50%     90%     max     N       mean
Oak-Segment-Tar                    1     182    2050    2528    2555   2589      17    2395
Oak-Segment-Tar                    5     685     694   14276   14692  14834      17   12682
Oak-Segment-Tar                   10    2009    2186   32604   34728  36387      25   26588
Oak-Segment-Tar                   20    3008    3619   62627   67037  67603      24   52872
Oak-Segment-Tar                   50    7188   10529  138560  167500 168827      56  124069
Unique users 72 out of total 100. Groups 1200. Err 0. Seed 1729171927509 {code}
 
In general the results obtained with the cache version outperforms the non-cached version seeing huge reduction in execution times. 

> Create benchmark
> ----------------
>
>                 Key: OAK-11171
>                 URL: https://issues.apache.org/jira/browse/OAK-11171
>             Project: Jackrabbit Oak
>          Issue Type: Technical task
>          Components: security
>            Reporter: Alejandro Moratinos
>            Assignee: Alejandro Moratinos
>            Priority: Major
>
> Create or modify actual benchmarks to test external group caching feature



--
This message was sent by Atlassian Jira
(v8.20.10#820010)