You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@solr.apache.org by "Andreas Gusenbauer [CELUM]" <An...@celum.com> on 2021/09/21 12:37:15 UTC
Slow response times when using query paramters
Hi all,
We encountered slow response times when querying solr with additional raw queries.
The response times decrease by a factor of 10 after deleting and rebuilding the index from scratch. After some minor modifications , the queries response times increase by a factor of 10 again.
Some information:
* solr index size is about 7GB
* 3,1 Mio documents
* Document/filter cache seem to be correctly used (hit count increases when we execute the same query twice)
* We can't use the result query cache for our query (see below)
* q: containers:80
* fq: (createdBy:106 assetStatus:0) ((allowPermissions:80 allowPermissions:22 allowPermissions:106) -assetStatus:0)
* rows: 100
* 20 Raw query parameters ( e.g qq_viewFieldPermission_customFields_202={!v='(((viewFieldPermission_customFields_202:80 viewFieldPermission_customFields_202:22 viewFieldPermission_customFields_202:106)))'})
* 20 additional fl values for raw query parameters ( e.g permission_customFields_date_202:exists(query($qq_viewFieldPermission_customFields_202)) )
* wt: json ( but xml does not make a difference)
Request:
<solr>/solr/assets/select?fl=id%2Cname%2Cpermission_customFields_date_200%3Aexists(query(%24qq_viewFieldPermission_customFields_200))%2Cpermission_customFields_date_201%3Aexists(query(%24qq_viewFieldPermission_customFields_201))%2Cpermission_customFields_date_202%3Aexists(query(%24qq_viewFieldPermission_customFields_202))%2Cpermission_customFields_date_207%3Aexists(query(%24qq_viewFieldPermission_customFields_207))%2Cpermission_customFields_date_208%3Aexists(query(%24qq_viewFieldPermission_customFields_208))%2Cpermission_customFields_date_209%3Aexists(query(%24qq_viewFieldPermission_customFields_209))%2Cpermission_customFields_date_216%3Aexists(query(%24qq_viewFieldPermission_customFields_216))%2Cpermission_customFields_date_217%3Aexists(query(%24qq_viewFieldPermission_customFields_217))%2Cpermission_customFields_date_218%3Aexists(query(%24qq_viewFieldPermission_customFields_218))%2Cpermission_customFields_date_219%3Aexists(query(%24qq_viewFieldPermission_customFields_219))%2Cpermission_customFields_date_222%3Aexists(query(%24qq_viewFieldPermission_customFields_222))%2Cpermission_customFields_date_223%3Aexists(query(%24qq_viewFieldPermission_customFields_223))%2Cpermission_customFields_date_160%3Aexists(query(%24qq_viewFieldPermission_customFields_160))%2Cpermission_customFields_date_224%3Aexists(query(%24qq_viewFieldPermission_customFields_224))%2Cpermission_customFields_date_225%3Aexists(query(%24qq_viewFieldPermission_customFields_225))%2Cpermission_customFields_date_226%3Aexists(query(%24qq_viewFieldPermission_customFields_226))%2Cpermission_customFields_date_100%3Aexists(query(%24qq_viewFieldPermission_customFields_100))%2Cpermission_customFields_date_231%3Aexists(query(%24qq_viewFieldPermission_customFields_231))%2Cpermission_customFields_date_170%3Aexists(query(%24qq_viewFieldPermission_customFields_170))%2Cpermission_customFields_date_235%3Aexists(query(%24qq_viewFieldPermission_customFields_235))&fq=(createdBy%3A106%20assetStatus%3A0)%20((allowPermissions%3A80%20allowPermissions%3A22%20allowPermissions%3A106)%20-assetStatus%3A0)&q=containers%3A80&qq_viewFieldPermission_customFields_100=%7B!v%3D%27(((viewFieldPermission_customFields_100%3A80%20viewFieldPermission_customFields_100%3A22%20viewFieldPermission_customFields_100%3A106)))%27%7D&qq_viewFieldPermission_customFields_110=%7B!v%3D%27(((viewFieldPermission_customFields_110%3A80%20viewFieldPermission_customFields_110%3A22%20viewFieldPermission_customFields_110%3A106)))%27%7D&qq_viewFieldPermission_customFields_120=%7B!v%3D%27(((viewFieldPermission_customFields_120%3A80%20viewFieldPermission_customFields_120%3A22%20viewFieldPermission_customFields_120%3A106)))%27%7D&qq_viewFieldPermission_customFields_130=%7B!v%3D%27(((viewFieldPermission_customFields_130%3A80%20viewFieldPermission_customFields_130%3A22%20viewFieldPermission_customFields_130%3A106)))%27%7D&qq_viewFieldPermission_customFields_160=%7B!v%3D%27(((viewFieldPermission_customFields_160%3A80%20viewFieldPermission_customFields_160%3A22%20viewFieldPermission_customFields_160%3A106)))%27%7D&qq_viewFieldPermission_customFields_170=%7B!v%3D%27(((viewFieldPermission_customFields_170%3A80%20viewFieldPermission_customFields_170%3A22%20viewFieldPermission_customFields_170%3A106)))%27%7D&qq_viewFieldPermission_customFields_180=%7B!v%3D%27(((viewFieldPermission_customFields_180%3A80%20viewFieldPermission_customFields_180%3A22%20viewFieldPermission_customFields_180%3A106)))%27%7D&qq_viewFieldPermission_customFields_190=%7B!v%3D%27(((viewFieldPermission_customFields_190%3A80%20viewFieldPermission_customFields_190%3A22%20viewFieldPermission_customFields_190%3A106)))%27%7D&qq_viewFieldPermission_customFields_200=%7B!v%3D%27(((viewFieldPermission_customFields_200%3A80%20viewFieldPermission_customFields_200%3A22%20viewFieldPermission_customFields_200%3A106)))%27%7D&qq_viewFieldPermission_customFields_201=%7B!v%3D%27(((viewFieldPermission_customFields_201%3A80%20viewFieldPermission_customFields_201%3A22%20viewFieldPermission_customFields_201%3A106)))%27%7D&qq_viewFieldPermission_customFields_202=%7B!v%3D%27(((viewFieldPermission_customFields_202%3A80%20viewFieldPermission_customFields_202%3A22%20viewFieldPermission_customFields_202%3A106)))%27%7D&qq_viewFieldPermission_customFields_207=%7B!v%3D%27(((viewFieldPermission_customFields_207%3A80%20viewFieldPermission_customFields_207%3A22%20viewFieldPermission_customFields_207%3A106)))%27%7D&qq_viewFieldPermission_customFields_208=%7B!v%3D%27(((viewFieldPermission_customFields_208%3A80%20viewFieldPermission_customFields_208%3A22%20viewFieldPermission_customFields_208%3A106)))%27%7D&qq_viewFieldPermission_customFields_209=%7B!v%3D%27(((viewFieldPermission_customFields_209%3A80%20viewFieldPermission_customFields_209%3A22%20viewFieldPermission_customFields_209%3A106)))%27%7D&qq_viewFieldPermission_customFields_216=%7B!v%3D%27(((viewFieldPermission_customFields_216%3A80%20viewFieldPermission_customFields_216%3A22%20viewFieldPermission_customFields_216%3A106)))%27%7D&qq_viewFieldPermission_customFields_217=%7B!v%3D%27(((viewFieldPermission_customFields_217%3A80%20viewFieldPermission_customFields_217%3A22%20viewFieldPermission_customFields_217%3A106)))%27%7D&qq_viewFieldPermission_customFields_218=%7B!v%3D%27(((viewFieldPermission_customFields_218%3A80%20viewFieldPermission_customFields_218%3A22%20viewFieldPermission_customFields_218%3A106)))%27%7D&qq_viewFieldPermission_customFields_219=%7B!v%3D%27(((viewFieldPermission_customFields_219%3A80%20viewFieldPermission_customFields_219%3A22%20viewFieldPermission_customFields_219%3A106)))%27%7D&qq_viewFieldPermission_customFields_222=%7B!v%3D%27(((viewFieldPermission_customFields_222%3A80%20viewFieldPermission_customFields_222%3A22%20viewFieldPermission_customFields_222%3A106)))%27%7D&qq_viewFieldPermission_customFields_223=%7B!v%3D%27(((viewFieldPermission_customFields_223%3A80%20viewFieldPermission_customFields_223%3A22%20viewFieldPermission_customFields_223%3A106)))%27%7D&qq_viewFieldPermission_customFields_224=%7B!v%3D%27(((viewFieldPermission_customFields_224%3A80%20viewFieldPermission_customFields_224%3A22%20viewFieldPermission_customFields_224%3A106)))%27%7D&qq_viewFieldPermission_customFields_225=%7B!v%3D%27(((viewFieldPermission_customFields_225%3A80%20viewFieldPermission_customFields_225%3A22%20viewFieldPermission_customFields_225%3A106)))%27%7D&qq_viewFieldPermission_customFields_226=%7B!v%3D%27(((viewFieldPermission_customFields_226%3A80%20viewFieldPermission_customFields_226%3A22%20viewFieldPermission_customFields_226%3A106)))%27%7D&qq_viewFieldPermission_customFields_231=%7B!v%3D%27(((viewFieldPermission_customFields_231%3A80%20viewFieldPermission_customFields_231%3A22%20viewFieldPermission_customFields_231%3A106)))%27%7D&qq_viewFieldPermission_customFields_235=%7B!v%3D%27(((viewFieldPermission_customFields_235%3A80%20viewFieldPermission_customFields_235%3A22%20viewFieldPermission_customFields_235%3A106)))%27%7D&qq_viewFieldPermission_customFields_236=%7B!v%3D%27(((viewFieldPermission_customFields_236%3A80%20viewFieldPermission_customFields_236%3A22%20viewFieldPermission_customFields_236%3A106)))%27%7D&qq_viewFieldPermission_customFields_237=%7B!v%3D%27(((viewFieldPermission_customFields_237%3A80%20viewFieldPermission_customFields_237%3A22%20viewFieldPermission_customFields_237%3A106)))%27%7D&qq_viewFieldPermission_customFields_238=%7B!v%3D%27(((viewFieldPermission_customFields_238%3A80%20viewFieldPermission_customFields_238%3A22%20viewFieldPermission_customFields_238%3A106)))%27%7D&qq_viewFieldPermission_customFields_241=%7B!v%3D%27(((viewFieldPermission_customFields_241%3A80%20viewFieldPermission_customFields_241%3A22%20viewFieldPermission_customFields_241%3A106)))%27%7D&qq_viewFieldPermission_customFields_242=%7B!v%3D%27(((viewFieldPermission_customFields_242%3A80%20viewFieldPermission_customFields_242%3A22%20viewFieldPermission_customFields_242%3A106)))%27%7D&rows=100&sort=id%20asc&start=300
When it is executed a second time the QTime is about 4 - 11 but the total response time is about 15 seconds !!
[cid:image001.png@01D7A8B1.E2775C70]
When it does NOT come from cache the QTime is about 1 second which seems realistic (see below): But the total response time is 15 seconds and approx. the same as above ( ~ 140 KB)!
{
"responseHeader":{
"status":0,
"QTime":914,
"params":{
"qq_viewFieldPermission_customFields_209":"{!v='(((viewFieldPermission_customFields_209:80 viewFieldPermission_customFields_209:22 viewFieldPermission_customFields_209:106)))'}",
"fl":"id,name,permission_customFields_date_200:exists(query($qq_viewFieldPermission_customFields_200)),permission_customFields_date_201:exists(query($qq_viewFieldPermission_customFields_201)),permission_customFields_date_202:exists(query($qq_viewFieldPermission_customFields_202)),permission_customFields_date_207:exists(query($qq_viewFieldPermission_customFields_207)),permission_customFields_date_208:exists(query($qq_viewFieldPermission_customFields_208)),permission_customFields_date_209:exists(query($qq_viewFieldPermission_customFields_209)),permission_customFields_date_216:exists(query($qq_viewFieldPermission_customFields_216)),permission_customFields_date_217:exists(query($qq_viewFieldPermission_customFields_217)),permission_customFields_date_218:exists(query($qq_viewFieldPermission_customFields_218)),permission_customFields_date_219:exists(query($qq_viewFieldPermission_customFields_219)),permission_customFields_date_222:exists(query($qq_viewFieldPermission_customFields_222)),permission_customFields_date_223:exists(query($qq_viewFieldPermission_customFields_223)),permission_customFields_date_160:exists(query($qq_viewFieldPermission_customFields_160)),permission_customFields_date_224:exists(query($qq_viewFieldPermission_customFields_224)),permission_customFields_date_225:exists(query($qq_viewFieldPermission_customFields_225)),permission_customFields_date_226:exists(query($qq_viewFieldPermission_customFields_226)),permission_customFields_date_100:exists(query($qq_viewFieldPermission_customFields_100)),permission_customFields_date_231:exists(query($qq_viewFieldPermission_customFields_231)),permission_customFields_date_170:exists(query($qq_viewFieldPermission_customFields_170)),permission_customFields_date_235:exists(query($qq_viewFieldPermission_customFields_235))",
"fq":"(createdBy:106 assetStatus:0) ((allowPermissions:80 allowPermissions:22 allowPermissions:106) -assetStatus:0)",
"qq_viewFieldPermission_customFields_170":"{!v='(((viewFieldPermission_customFields_170:80 viewFieldPermission_customFields_170:22 viewFieldPermission_customFields_170:106)))'}",
"qq_viewFieldPermission_customFields_190":"{!v='(((viewFieldPermission_customFields_190:80 viewFieldPermission_customFields_190:22 viewFieldPermission_customFields_190:106)))'}",
"qq_viewFieldPermission_customFields_130":"{!v='(((viewFieldPermission_customFields_130:80 viewFieldPermission_customFields_130:22 viewFieldPermission_customFields_130:106)))'}",
"qq_viewFieldPermission_customFields_110":"{!v='(((viewFieldPermission_customFields_110:80 viewFieldPermission_customFields_110:22 viewFieldPermission_customFields_110:106)))'}",
"qq_viewFieldPermission_customFields_231":"{!v='(((viewFieldPermission_customFields_231:80 viewFieldPermission_customFields_231:22 viewFieldPermission_customFields_231:106)))'}",
"qq_viewFieldPermission_customFields_236":"{!v='(((viewFieldPermission_customFields_236:80 viewFieldPermission_customFields_236:22 viewFieldPermission_customFields_236:106)))'}",
"qq_viewFieldPermission_customFields_237":"{!v='(((viewFieldPermission_customFields_237:80 viewFieldPermission_customFields_237:22 viewFieldPermission_customFields_237:106)))'}",
"qq_viewFieldPermission_customFields_235":"{!v='(((viewFieldPermission_customFields_235:80 viewFieldPermission_customFields_235:22 viewFieldPermission_customFields_235:106)))'}",
"qq_viewFieldPermission_customFields_218":"{!v='(((viewFieldPermission_customFields_218:80 viewFieldPermission_customFields_218:22 viewFieldPermission_customFields_218:106)))'}",
"qq_viewFieldPermission_customFields_219":"{!v='(((viewFieldPermission_customFields_219:80 viewFieldPermission_customFields_219:22 viewFieldPermission_customFields_219:106)))'}",
"qq_viewFieldPermission_customFields_216":"{!v='(((viewFieldPermission_customFields_216:80 viewFieldPermission_customFields_216:22 viewFieldPermission_customFields_216:106)))'}",
"qq_viewFieldPermission_customFields_238":"{!v='(((viewFieldPermission_customFields_238:80 viewFieldPermission_customFields_238:22 viewFieldPermission_customFields_238:106)))'}",
"qq_viewFieldPermission_customFields_217":"{!v='(((viewFieldPermission_customFields_217:80 viewFieldPermission_customFields_217:22 viewFieldPermission_customFields_217:106)))'}",
"debug":"true",
"start":"0",
"sort":"id asc",
"qq_viewFieldPermission_customFields_180":"{!v='(((viewFieldPermission_customFields_180:80 viewFieldPermission_customFields_180:22 viewFieldPermission_customFields_180:106)))'}",
"rows":"100",
"q":"containers:80",
"qq_viewFieldPermission_customFields_160":"{!v='(((viewFieldPermission_customFields_160:80 viewFieldPermission_customFields_160:22 viewFieldPermission_customFields_160:106)))'}",
"qq_viewFieldPermission_customFields_100":"{!v='(((viewFieldPermission_customFields_100:80 viewFieldPermission_customFields_100:22 viewFieldPermission_customFields_100:106)))'}",
"qq_viewFieldPermission_customFields_200":"{!v='(((viewFieldPermission_customFields_200:80 viewFieldPermission_customFields_200:22 viewFieldPermission_customFields_200:106)))'}",
"qq_viewFieldPermission_customFields_222":"{!v='(((viewFieldPermission_customFields_222:80 viewFieldPermission_customFields_222:22 viewFieldPermission_customFields_222:106)))'}",
"qq_viewFieldPermission_customFields_120":"{!v='(((viewFieldPermission_customFields_120:80 viewFieldPermission_customFields_120:22 viewFieldPermission_customFields_120:106)))'}",
"qq_viewFieldPermission_customFields_241":"{!v='(((viewFieldPermission_customFields_241:80 viewFieldPermission_customFields_241:22 viewFieldPermission_customFields_241:106)))'}",
"qq_viewFieldPermission_customFields_242":"{!v='(((viewFieldPermission_customFields_242:80 viewFieldPermission_customFields_242:22 viewFieldPermission_customFields_242:106)))'}",
"qq_viewFieldPermission_customFields_225":"{!v='(((viewFieldPermission_customFields_225:80 viewFieldPermission_customFields_225:22 viewFieldPermission_customFields_225:106)))'}",
"qq_viewFieldPermission_customFields_226":"{!v='(((viewFieldPermission_customFields_226:80 viewFieldPermission_customFields_226:22 viewFieldPermission_customFields_226:106)))'}",
"qq_viewFieldPermission_customFields_201":"{!v='(((viewFieldPermission_customFields_201:80 viewFieldPermission_customFields_201:22 viewFieldPermission_customFields_201:106)))'}",
"qq_viewFieldPermission_customFields_223":"{!v='(((viewFieldPermission_customFields_223:80 viewFieldPermission_customFields_223:22 viewFieldPermission_customFields_223:106)))'}",
"qq_viewFieldPermission_customFields_202":"{!v='(((viewFieldPermission_customFields_202:80 viewFieldPermission_customFields_202:22 viewFieldPermission_customFields_202:106)))'}",
"qq_viewFieldPermission_customFields_224":"{!v='(((viewFieldPermission_customFields_224:80 viewFieldPermission_customFields_224:22 viewFieldPermission_customFields_224:106)))'}",
"qq_viewFieldPermission_customFields_207":"{!v='(((viewFieldPermission_customFields_207:80 viewFieldPermission_customFields_207:22 viewFieldPermission_customFields_207:106)))'}",
"qq_viewFieldPermission_customFields_208":"{!v='(((viewFieldPermission_customFields_208:80 viewFieldPermission_customFields_208:22 viewFieldPermission_customFields_208:106)))'}"}},
"response":{"numFound":927,"start":0,"numFoundExact":true,"docs":[
{
"id":20,
"name":"b301112_27_u_32_u_40.tif",
"permission_customFields_date_200":false,
"permission_customFields_date_201":true,
"permission_customFields_date_202":true,
"permission_customFields_date_207":true,
"permission_customFields_date_208":true,
"permission_customFields_date_209":true,
"permission_customFields_date_216":false,
"permission_customFields_date_217":false,
"permission_customFields_date_218":false,
"permission_customFields_date_219":false,
"permission_customFields_date_222":false,
"permission_customFields_date_223":false,
"permission_customFields_date_160":false,
"permission_customFields_date_224":false,
"permission_customFields_date_225":false,
"permission_customFields_date_226":false,
"permission_customFields_date_100":false,
"permission_customFields_date_231":true,
"permission_customFields_date_170":false,
"permission_customFields_date_235":false},
[2 ... 99 other documents in response]
{
"id":4930,
"name":"b307682_03-10.tif",
"permission_customFields_date_200":false,
"permission_customFields_date_201":true,
"permission_customFields_date_202":true,
"permission_customFields_date_207":true,
"permission_customFields_date_208":true,
"permission_customFields_date_209":true,
"permission_customFields_date_216":false,
"permission_customFields_date_217":false,
"permission_customFields_date_218":false,
"permission_customFields_date_219":false,
"permission_customFields_date_222":false,
"permission_customFields_date_223":false,
"permission_customFields_date_160":false,
"permission_customFields_date_224":false,
"permission_customFields_date_225":false,
"permission_customFields_date_226":false,
"permission_customFields_date_100":false,
"permission_customFields_date_231":true,
"permission_customFields_date_170":false,
"permission_customFields_date_235":false}]
},
"debug":{
"rawquerystring":"containers:80",
"querystring":"containers:80",
"parsedquery":"IndexOrDocValuesQuery(containers:[80 TO 80])",
"parsedquery_toString":"containers:[80 TO 80]",
"explain":{
"20":"\n1.0 = containers:[80 TO 80]\n",
"76":"\n1.0 = containers:[80 TO 80]\n",
"1514":"\n1.0 = containers:[80 TO 80]\n",
"1565":"\n1.0 = containers:[80 TO 80]\n",
"1566":"\n1.0 = containers:[80 TO 80]\n",
"1607":"\n1.0 = containers:[80 TO 80]\n",
"1675":"\n1.0 = containers:[80 TO 80]\n",
"2489":"\n1.0 = containers:[80 TO 80]\n",
"2642":"\n1.0 = containers:[80 TO 80]\n",
"2643":"\n1.0 = containers:[80 TO 80]\n",
"2680":"\n1.0 = containers:[80 TO 80]\n",
"2681":"\n1.0 = containers:[80 TO 80]\n",
"2682":"\n1.0 = containers:[80 TO 80]\n",
"2683":"\n1.0 = containers:[80 TO 80]\n",
"2684":"\n1.0 = containers:[80 TO 80]\n",
"2685":"\n1.0 = containers:[80 TO 80]\n",
"2686":"\n1.0 = containers:[80 TO 80]\n",
"2696":"\n1.0 = containers:[80 TO 80]\n",
"2697":"\n1.0 = containers:[80 TO 80]\n",
"2698":"\n1.0 = containers:[80 TO 80]\n",
"2699":"\n1.0 = containers:[80 TO 80]\n",
"2700":"\n1.0 = containers:[80 TO 80]\n",
"2701":"\n1.0 = containers:[80 TO 80]\n",
"2702":"\n1.0 = containers:[80 TO 80]\n",
"2719":"\n1.0 = containers:[80 TO 80]\n",
"2720":"\n1.0 = containers:[80 TO 80]\n",
"2727":"\n1.0 = containers:[80 TO 80]\n",
"2739":"\n1.0 = containers:[80 TO 80]\n",
"2740":"\n1.0 = containers:[80 TO 80]\n",
"2741":"\n1.0 = containers:[80 TO 80]\n",
"2742":"\n1.0 = containers:[80 TO 80]\n",
"2743":"\n1.0 = containers:[80 TO 80]\n",
"2744":"\n1.0 = containers:[80 TO 80]\n",
"2745":"\n1.0 = containers:[80 TO 80]\n",
"2746":"\n1.0 = containers:[80 TO 80]\n",
"2747":"\n1.0 = containers:[80 TO 80]\n",
"2748":"\n1.0 = containers:[80 TO 80]\n",
"2749":"\n1.0 = containers:[80 TO 80]\n",
"2750":"\n1.0 = containers:[80 TO 80]\n",
"2755":"\n1.0 = containers:[80 TO 80]\n",
"2756":"\n1.0 = containers:[80 TO 80]\n",
"2757":"\n1.0 = containers:[80 TO 80]\n",
"2758":"\n1.0 = containers:[80 TO 80]\n",
"2759":"\n1.0 = containers:[80 TO 80]\n",
"2760":"\n1.0 = containers:[80 TO 80]\n",
"2761":"\n1.0 = containers:[80 TO 80]\n",
"2762":"\n1.0 = containers:[80 TO 80]\n",
"2763":"\n1.0 = containers:[80 TO 80]\n",
"2799":"\n1.0 = containers:[80 TO 80]\n",
"2800":"\n1.0 = containers:[80 TO 80]\n",
"2801":"\n1.0 = containers:[80 TO 80]\n",
"2802":"\n1.0 = containers:[80 TO 80]\n",
"2803":"\n1.0 = containers:[80 TO 80]\n",
"2804":"\n1.0 = containers:[80 TO 80]\n",
"2805":"\n1.0 = containers:[80 TO 80]\n",
"2829":"\n1.0 = containers:[80 TO 80]\n",
"2838":"\n1.0 = containers:[80 TO 80]\n",
"2839":"\n1.0 = containers:[80 TO 80]\n",
"2840":"\n1.0 = containers:[80 TO 80]\n",
"2841":"\n1.0 = containers:[80 TO 80]\n",
"2842":"\n1.0 = containers:[80 TO 80]\n",
"2843":"\n1.0 = containers:[80 TO 80]\n",
"2878":"\n1.0 = containers:[80 TO 80]\n",
"2889":"\n1.0 = containers:[80 TO 80]\n",
"2911":"\n1.0 = containers:[80 TO 80]\n",
"2945":"\n1.0 = containers:[80 TO 80]\n",
"2946":"\n1.0 = containers:[80 TO 80]\n",
"2947":"\n1.0 = containers:[80 TO 80]\n",
"2972":"\n1.0 = containers:[80 TO 80]\n",
"2973":"\n1.0 = containers:[80 TO 80]\n",
"2974":"\n1.0 = containers:[80 TO 80]\n",
"2975":"\n1.0 = containers:[80 TO 80]\n",
"2976":"\n1.0 = containers:[80 TO 80]\n",
"2999":"\n1.0 = containers:[80 TO 80]\n",
"3001":"\n1.0 = containers:[80 TO 80]\n",
"3062":"\n1.0 = containers:[80 TO 80]\n",
"3063":"\n1.0 = containers:[80 TO 80]\n",
"4335":"\n1.0 = containers:[80 TO 80]\n",
"4604":"\n1.0 = containers:[80 TO 80]\n",
"4606":"\n1.0 = containers:[80 TO 80]\n",
"4608":"\n1.0 = containers:[80 TO 80]\n",
"4705":"\n1.0 = containers:[80 TO 80]\n",
"4706":"\n1.0 = containers:[80 TO 80]\n",
"4707":"\n1.0 = containers:[80 TO 80]\n",
"4723":"\n1.0 = containers:[80 TO 80]\n",
"4724":"\n1.0 = containers:[80 TO 80]\n",
"4732":"\n1.0 = containers:[80 TO 80]\n",
"4736":"\n1.0 = containers:[80 TO 80]\n",
"4748":"\n1.0 = containers:[80 TO 80]\n",
"4749":"\n1.0 = containers:[80 TO 80]\n",
"4764":"\n1.0 = containers:[80 TO 80]\n",
"4766":"\n1.0 = containers:[80 TO 80]\n",
"4768":"\n1.0 = containers:[80 TO 80]\n",
"4769":"\n1.0 = containers:[80 TO 80]\n",
"4792":"\n1.0 = containers:[80 TO 80]\n",
"4793":"\n1.0 = containers:[80 TO 80]\n",
"4794":"\n1.0 = containers:[80 TO 80]\n",
"4815":"\n1.0 = containers:[80 TO 80]\n",
"4883":"\n1.0 = containers:[80 TO 80]\n",
"4930":"\n1.0 = containers:[80 TO 80]\n"},
"QParser":"LuceneQParser",
"filter_queries":["(createdBy:106 assetStatus:0) ((allowPermissions:80 allowPermissions:22 allowPermissions:106) -assetStatus:0)"],
"parsed_filter_queries":["(IndexOrDocValuesQuery(createdBy:[106 TO 106]) IndexOrDocValuesQuery(assetStatus:[0 TO 0])) ((IndexOrDocValuesQuery(allowPermissions:[80 TO 80]) IndexOrDocValuesQuery(allowPermissions:[22 TO 22]) IndexOrDocValuesQuery(allowPermissions:[106 TO 106])) -IndexOrDocValuesQuery(assetStatus:[0 TO 0]))"],
"timing":{
"time":912.0,
"prepare":{
"time":36.0,
"query":{
"time":35.0},
"facet":{
"time":0.0},
"facet_module":{
"time":0.0},
"mlt":{
"time":0.0},
"highlight":{
"time":0.0},
"stats":{
"time":0.0},
"expand":{
"time":0.0},
"terms":{
"time":0.0},
"debug":{
"time":0.0}},
"process":{
"time":870.0,
"query":{
"time":845.0},
"facet":{
"time":0.0},
"facet_module":{
"time":0.0},
"mlt":{
"time":0.0},
"highlight":{
"time":0.0},
"stats":{
"time":0.0},
"expand":{
"time":0.0},
"terms":{
"time":0.0},
"debug":{
"time":23.0}}}}}
I cannot explain the very long waiting times to download or deliver the http chunks on localhost. Even the TTFB is much higher than QTime.
Moreover I have no further idea what I can do to improve the response times.
Additionally I have measured solr with yourkit. it takes obviously about 11 seconds to process (BKDReader.intersect):
[cid:image002.png@01D7A8B3.718A2450]
When we remove more and more of the raw query evaluations (permission_customFields_date_202:exists(query($qq_viewFieldPermission_customFields_202)) etc.) from fl the following linear response times could be observed:
* keep only 10 instead of 15 -> total time of 7 seconds to response
* keep only 5 -> total time of 2,7 seconds to Response
* remove completely, then only a few milliseconds are measured (20 - 30 ms)
* [cid:image003.png@01D7A8B3.EF2264E0]
It is comprehensible that the query evaluation and fl parameters influence the processing, but it is not clear why the performance degrades that dramatic after some modifications in index.
As mentioned if I drop the whole index and rebuild everything again, the response times are about 1,5 seconds instead of 15 seconds. "Optimize index" didn't help.
Please let me know if someone has some ideas what causes the described problems or what we can change to improve.
Please also let me know if any additional details are required from my end.
Thanks in advance,
ako
Re: Slow response times when using query paramters
Posted by matthew sporleder <ms...@gmail.com>.
I'm sure this is well done report but this mailing list strips images, etc,
so it doesn't quite make it through. :)
You might want to consider a gist instead.
On Tue, Sep 21, 2021 at 8:37 AM Andreas Gusenbauer [CELUM] <
Andreas.Gusenbauer@celum.com> wrote:
> Hi all,
>
>
>
> We encountered slow response times when querying solr with additional raw
> queries.
>
> The response times decrease by a factor of 10 after deleting and
> rebuilding the index from scratch. After some minor modifications , the
> queries response times increase by a factor of 10 again.
>
>
>
> *Some information: *
>
> - solr index size is about 7GB
> - 3,1 Mio documents
> - Document/filter cache seem to be correctly used (hit count increases
> when we execute the same query twice)
> - We can’t use the result query cache for our query (see below)
> - *q*: containers:80
> - *fq*: (createdBy:106 assetStatus:0) ((allowPermissions:80
> allowPermissions:22 allowPermissions:106) -assetStatus:0)
> - *rows*: 100
> - 20 *Raw query parameters* ( e.g
> qq_viewFieldPermission_customFields_202={!v='(((viewFieldPermission_customFields_202:80
> viewFieldPermission_customFields_202:22
> viewFieldPermission_customFields_202:106)))'})
> - 20 additional *fl* values for raw query parameters ( e.g
> permission_customFields_date_202:exists(query($qq_viewFieldPermission_customFields_202))
> )
> - wt: json ( but xml does not make a difference)
>
>
>
>
>
> *Request:*
>
>
> <solr>/solr/assets/select?fl=id%2Cname%2Cpermission_customFields_date_200%3Aexists(query(%24qq_viewFieldPermission_customFields_200))%2Cpermission_customFields_date_201%3Aexists(query(%24qq_viewFieldPermission_customFields_201))%2Cpermission_customFields_date_202%3Aexists(query(%24qq_viewFieldPermission_customFields_202))%2Cpermission_customFields_date_207%3Aexists(query(%24qq_viewFieldPermission_customFields_207))%2Cpermission_customFields_date_208%3Aexists(query(%24qq_viewFieldPermission_customFields_208))%2Cpermission_customFields_date_209%3Aexists(query(%24qq_viewFieldPermission_customFields_209))%2Cpermission_customFields_date_216%3Aexists(query(%24qq_viewFieldPermission_customFields_216))%2Cpermission_customFields_date_217%3Aexists(query(%24qq_viewFieldPermission_customFields_217))%2Cpermission_customFields_date_218%3Aexists(query(%24qq_viewFieldPermission_customFields_218))%2Cpermission_customFields_date_219%3Aexists(query(%24qq_viewFieldPermission_customFields_219))%2Cpermission_customFields_date_222%3Aexists(query(%24qq_viewFieldPermission_customFields_222))%2Cpermission_customFields_date_223%3Aexists(query(%24qq_viewFieldPermission_customFields_223))%2Cpermission_customFields_date_160%3Aexists(query(%24qq_viewFieldPermission_customFields_160))%2Cpermission_customFields_date_224%3Aexists(query(%24qq_viewFieldPermission_customFields_224))%2Cpermission_customFields_date_225%3Aexists(query(%24qq_viewFieldPermission_customFields_225))%2Cpermission_customFields_date_226%3Aexists(query(%24qq_viewFieldPermission_customFields_226))%2Cpermission_customFields_date_100%3Aexists(query(%24qq_viewFieldPermission_customFields_100))%2Cpermission_customFields_date_231%3Aexists(query(%24qq_viewFieldPermission_customFields_231))%2Cpermission_customFields_date_170%3Aexists(query(%24qq_viewFieldPermission_customFields_170))%2Cpermission_customFields_date_235%3Aexists(query(%24qq_viewFieldPermission_customFields_235))&fq=(createdBy%3A106%20assetStatus%3A0)%20((allowPermissions%3A80%20allowPermissions%3A22%20allowPermissions%3A106)%20-assetStatus%3A0)&q=containers%3A80&qq_viewFieldPermission_customFields_100=%7B!v%3D%27(((viewFieldPermission_customFields_100%3A80%20viewFieldPermission_customFields_100%3A22%20viewFieldPermission_customFields_100%3A106)))%27%7D&qq_viewFieldPermission_customFields_110=%7B!v%3D%27(((viewFieldPermission_customFields_110%3A80%20viewFieldPermission_customFields_110%3A22%20viewFieldPermission_customFields_110%3A106)))%27%7D&qq_viewFieldPermission_customFields_120=%7B!v%3D%27(((viewFieldPermission_customFields_120%3A80%20viewFieldPermission_customFields_120%3A22%20viewFieldPermission_customFields_120%3A106)))%27%7D&qq_viewFieldPermission_customFields_130=%7B!v%3D%27(((viewFieldPermission_customFields_130%3A80%20viewFieldPermission_customFields_130%3A22%20viewFieldPermission_customFields_130%3A106)))%27%7D&qq_viewFieldPermission_customFields_160=%7B!v%3D%27(((viewFieldPermission_customFields_160%3A80%20viewFieldPermission_customFields_160%3A22%20viewFieldPermission_customFields_160%3A106)))%27%7D&qq_viewFieldPermission_customFields_170=%7B!v%3D%27(((viewFieldPermission_customFields_170%3A80%20viewFieldPermission_customFields_170%3A22%20viewFieldPermission_customFields_170%3A106)))%27%7D&qq_viewFieldPermission_customFields_180=%7B!v%3D%27(((viewFieldPermission_customFields_180%3A80%20viewFieldPermission_customFields_180%3A22%20viewFieldPermission_customFields_180%3A106)))%27%7D&qq_viewFieldPermission_customFields_190=%7B!v%3D%27(((viewFieldPermission_customFields_190%3A80%20viewFieldPermission_customFields_190%3A22%20viewFieldPermission_customFields_190%3A106)))%27%7D&qq_viewFieldPermission_customFields_200=%7B!v%3D%27(((viewFieldPermission_customFields_200%3A80%20viewFieldPermission_customFields_200%3A22%20viewFieldPermission_customFields_200%3A106)))%27%7D&qq_viewFieldPermission_customFields_201=%7B!v%3D%27(((viewFieldPermission_customFields_201%3A80%20viewFieldPermission_customFields_201%3A22%20viewFieldPermission_customFields_201%3A106)))%27%7D&qq_viewFieldPermission_customFields_202=%7B!v%3D%27(((viewFieldPermission_customFields_202%3A80%20viewFieldPermission_customFields_202%3A22%20viewFieldPermission_customFields_202%3A106)))%27%7D&qq_viewFieldPermission_customFields_207=%7B!v%3D%27(((viewFieldPermission_customFields_207%3A80%20viewFieldPermission_customFields_207%3A22%20viewFieldPermission_customFields_207%3A106)))%27%7D&qq_viewFieldPermission_customFields_208=%7B!v%3D%27(((viewFieldPermission_customFields_208%3A80%20viewFieldPermission_customFields_208%3A22%20viewFieldPermission_customFields_208%3A106)))%27%7D&qq_viewFieldPermission_customFields_209=%7B!v%3D%27(((viewFieldPermission_customFields_209%3A80%20viewFieldPermission_customFields_209%3A22%20viewFieldPermission_customFields_209%3A106)))%27%7D&qq_viewFieldPermission_customFields_216=%7B!v%3D%27(((viewFieldPermission_customFields_216%3A80%20viewFieldPermission_customFields_216%3A22%20viewFieldPermission_customFields_216%3A106)))%27%7D&qq_viewFieldPermission_customFields_217=%7B!v%3D%27(((viewFieldPermission_customFields_217%3A80%20viewFieldPermission_customFields_217%3A22%20viewFieldPermission_customFields_217%3A106)))%27%7D&qq_viewFieldPermission_customFields_218=%7B!v%3D%27(((viewFieldPermission_customFields_218%3A80%20viewFieldPermission_customFields_218%3A22%20viewFieldPermission_customFields_218%3A106)))%27%7D&qq_viewFieldPermission_customFields_219=%7B!v%3D%27(((viewFieldPermission_customFields_219%3A80%20viewFieldPermission_customFields_219%3A22%20viewFieldPermission_customFields_219%3A106)))%27%7D&qq_viewFieldPermission_customFields_222=%7B!v%3D%27(((viewFieldPermission_customFields_222%3A80%20viewFieldPermission_customFields_222%3A22%20viewFieldPermission_customFields_222%3A106)))%27%7D&qq_viewFieldPermission_customFields_223=%7B!v%3D%27(((viewFieldPermission_customFields_223%3A80%20viewFieldPermission_customFields_223%3A22%20viewFieldPermission_customFields_223%3A106)))%27%7D&qq_viewFieldPermission_customFields_224=%7B!v%3D%27(((viewFieldPermission_customFields_224%3A80%20viewFieldPermission_customFields_224%3A22%20viewFieldPermission_customFields_224%3A106)))%27%7D&qq_viewFieldPermission_customFields_225=%7B!v%3D%27(((viewFieldPermission_customFields_225%3A80%20viewFieldPermission_customFields_225%3A22%20viewFieldPermission_customFields_225%3A106)))%27%7D&qq_viewFieldPermission_customFields_226=%7B!v%3D%27(((viewFieldPermission_customFields_226%3A80%20viewFieldPermission_customFields_226%3A22%20viewFieldPermission_customFields_226%3A106)))%27%7D&qq_viewFieldPermission_customFields_231=%7B!v%3D%27(((viewFieldPermission_customFields_231%3A80%20viewFieldPermission_customFields_231%3A22%20viewFieldPermission_customFields_231%3A106)))%27%7D&qq_viewFieldPermission_customFields_235=%7B!v%3D%27(((viewFieldPermission_customFields_235%3A80%20viewFieldPermission_customFields_235%3A22%20viewFieldPermission_customFields_235%3A106)))%27%7D&qq_viewFieldPermission_customFields_236=%7B!v%3D%27(((viewFieldPermission_customFields_236%3A80%20viewFieldPermission_customFields_236%3A22%20viewFieldPermission_customFields_236%3A106)))%27%7D&qq_viewFieldPermission_customFields_237=%7B!v%3D%27(((viewFieldPermission_customFields_237%3A80%20viewFieldPermission_customFields_237%3A22%20viewFieldPermission_customFields_237%3A106)))%27%7D&qq_viewFieldPermission_customFields_238=%7B!v%3D%27(((viewFieldPermission_customFields_238%3A80%20viewFieldPermission_customFields_238%3A22%20viewFieldPermission_customFields_238%3A106)))%27%7D&qq_viewFieldPermission_customFields_241=%7B!v%3D%27(((viewFieldPermission_customFields_241%3A80%20viewFieldPermission_customFields_241%3A22%20viewFieldPermission_customFields_241%3A106)))%27%7D&qq_viewFieldPermission_customFields_242=%7B!v%3D%27(((viewFieldPermission_customFields_242%3A80%20viewFieldPermission_customFields_242%3A22%20viewFieldPermission_customFields_242%3A106)))%27%7D&rows=100&sort=id%20asc&start=300
>
>
>
> *When it is executed a second time the QTime is about 4 – 11 but the total
> response time is about 15 seconds !!*
>
>
>
> *When it does NOT come from cache the QTime is about 1 second which seems
> realistic (see below): But the total response time is 15 seconds and
> approx. the same as above ( ~ 140 KB)!*
>
> {
>
> "responseHeader":{
>
> "status":0,
>
> "QTime":914,
>
> "params":{
>
>
> "qq_viewFieldPermission_customFields_209":"{!v='(((viewFieldPermission_customFields_209:80
> viewFieldPermission_customFields_209:22
> viewFieldPermission_customFields_209:106)))'}",
>
>
> "fl":"id,name,permission_customFields_date_200:exists(query($qq_viewFieldPermission_customFields_200)),permission_customFields_date_201:exists(query($qq_viewFieldPermission_customFields_201)),permission_customFields_date_202:exists(query($qq_viewFieldPermission_customFields_202)),permission_customFields_date_207:exists(query($qq_viewFieldPermission_customFields_207)),permission_customFields_date_208:exists(query($qq_viewFieldPermission_customFields_208)),permission_customFields_date_209:exists(query($qq_viewFieldPermission_customFields_209)),permission_customFields_date_216:exists(query($qq_viewFieldPermission_customFields_216)),permission_customFields_date_217:exists(query($qq_viewFieldPermission_customFields_217)),permission_customFields_date_218:exists(query($qq_viewFieldPermission_customFields_218)),permission_customFields_date_219:exists(query($qq_viewFieldPermission_customFields_219)),permission_customFields_date_222:exists(query($qq_viewFieldPermission_customFields_222)),permission_customFields_date_223:exists(query($qq_viewFieldPermission_customFields_223)),permission_customFields_date_160:exists(query($qq_viewFieldPermission_customFields_160)),permission_customFields_date_224:exists(query($qq_viewFieldPermission_customFields_224)),permission_customFields_date_225:exists(query($qq_viewFieldPermission_customFields_225)),permission_customFields_date_226:exists(query($qq_viewFieldPermission_customFields_226)),permission_customFields_date_100:exists(query($qq_viewFieldPermission_customFields_100)),permission_customFields_date_231:exists(query($qq_viewFieldPermission_customFields_231)),permission_customFields_date_170:exists(query($qq_viewFieldPermission_customFields_170)),permission_customFields_date_235:exists(query($qq_viewFieldPermission_customFields_235))",
>
> "fq":"(createdBy:106 assetStatus:0) ((allowPermissions:80
> allowPermissions:22 allowPermissions:106) -assetStatus:0)",
>
>
> "qq_viewFieldPermission_customFields_170":"{!v='(((viewFieldPermission_customFields_170:80
> viewFieldPermission_customFields_170:22
> viewFieldPermission_customFields_170:106)))'}",
>
>
> "qq_viewFieldPermission_customFields_190":"{!v='(((viewFieldPermission_customFields_190:80
> viewFieldPermission_customFields_190:22
> viewFieldPermission_customFields_190:106)))'}",
>
>
> "qq_viewFieldPermission_customFields_130":"{!v='(((viewFieldPermission_customFields_130:80
> viewFieldPermission_customFields_130:22
> viewFieldPermission_customFields_130:106)))'}",
>
>
> "qq_viewFieldPermission_customFields_110":"{!v='(((viewFieldPermission_customFields_110:80
> viewFieldPermission_customFields_110:22
> viewFieldPermission_customFields_110:106)))'}",
>
>
> "qq_viewFieldPermission_customFields_231":"{!v='(((viewFieldPermission_customFields_231:80
> viewFieldPermission_customFields_231:22
> viewFieldPermission_customFields_231:106)))'}",
>
>
> "qq_viewFieldPermission_customFields_236":"{!v='(((viewFieldPermission_customFields_236:80
> viewFieldPermission_customFields_236:22
> viewFieldPermission_customFields_236:106)))'}",
>
>
> "qq_viewFieldPermission_customFields_237":"{!v='(((viewFieldPermission_customFields_237:80
> viewFieldPermission_customFields_237:22
> viewFieldPermission_customFields_237:106)))'}",
>
>
> "qq_viewFieldPermission_customFields_235":"{!v='(((viewFieldPermission_customFields_235:80
> viewFieldPermission_customFields_235:22
> viewFieldPermission_customFields_235:106)))'}",
>
>
> "qq_viewFieldPermission_customFields_218":"{!v='(((viewFieldPermission_customFields_218:80
> viewFieldPermission_customFields_218:22
> viewFieldPermission_customFields_218:106)))'}",
>
>
> "qq_viewFieldPermission_customFields_219":"{!v='(((viewFieldPermission_customFields_219:80
> viewFieldPermission_customFields_219:22
> viewFieldPermission_customFields_219:106)))'}",
>
>
> "qq_viewFieldPermission_customFields_216":"{!v='(((viewFieldPermission_customFields_216:80
> viewFieldPermission_customFields_216:22
> viewFieldPermission_customFields_216:106)))'}",
>
>
> "qq_viewFieldPermission_customFields_238":"{!v='(((viewFieldPermission_customFields_238:80
> viewFieldPermission_customFields_238:22
> viewFieldPermission_customFields_238:106)))'}",
>
>
> "qq_viewFieldPermission_customFields_217":"{!v='(((viewFieldPermission_customFields_217:80
> viewFieldPermission_customFields_217:22
> viewFieldPermission_customFields_217:106)))'}",
>
> "debug":"true",
>
> "start":"0",
>
> "sort":"id asc",
>
>
> "qq_viewFieldPermission_customFields_180":"{!v='(((viewFieldPermission_customFields_180:80
> viewFieldPermission_customFields_180:22
> viewFieldPermission_customFields_180:106)))'}",
>
> "rows":"100",
>
> "q":"containers:80",
>
>
> "qq_viewFieldPermission_customFields_160":"{!v='(((viewFieldPermission_customFields_160:80
> viewFieldPermission_customFields_160:22
> viewFieldPermission_customFields_160:106)))'}",
>
>
> "qq_viewFieldPermission_customFields_100":"{!v='(((viewFieldPermission_customFields_100:80
> viewFieldPermission_customFields_100:22
> viewFieldPermission_customFields_100:106)))'}",
>
>
> "qq_viewFieldPermission_customFields_200":"{!v='(((viewFieldPermission_customFields_200:80
> viewFieldPermission_customFields_200:22
> viewFieldPermission_customFields_200:106)))'}",
>
>
> "qq_viewFieldPermission_customFields_222":"{!v='(((viewFieldPermission_customFields_222:80
> viewFieldPermission_customFields_222:22
> viewFieldPermission_customFields_222:106)))'}",
>
>
> "qq_viewFieldPermission_customFields_120":"{!v='(((viewFieldPermission_customFields_120:80
> viewFieldPermission_customFields_120:22
> viewFieldPermission_customFields_120:106)))'}",
>
>
> "qq_viewFieldPermission_customFields_241":"{!v='(((viewFieldPermission_customFields_241:80
> viewFieldPermission_customFields_241:22
> viewFieldPermission_customFields_241:106)))'}",
>
>
> "qq_viewFieldPermission_customFields_242":"{!v='(((viewFieldPermission_customFields_242:80
> viewFieldPermission_customFields_242:22
> viewFieldPermission_customFields_242:106)))'}",
>
>
> "qq_viewFieldPermission_customFields_225":"{!v='(((viewFieldPermission_customFields_225:80
> viewFieldPermission_customFields_225:22
> viewFieldPermission_customFields_225:106)))'}",
>
>
> "qq_viewFieldPermission_customFields_226":"{!v='(((viewFieldPermission_customFields_226:80
> viewFieldPermission_customFields_226:22
> viewFieldPermission_customFields_226:106)))'}",
>
>
> "qq_viewFieldPermission_customFields_201":"{!v='(((viewFieldPermission_customFields_201:80
> viewFieldPermission_customFields_201:22
> viewFieldPermission_customFields_201:106)))'}",
>
>
> "qq_viewFieldPermission_customFields_223":"{!v='(((viewFieldPermission_customFields_223:80
> viewFieldPermission_customFields_223:22
> viewFieldPermission_customFields_223:106)))'}",
>
>
> "qq_viewFieldPermission_customFields_202":"{!v='(((viewFieldPermission_customFields_202:80
> viewFieldPermission_customFields_202:22
> viewFieldPermission_customFields_202:106)))'}",
>
>
> "qq_viewFieldPermission_customFields_224":"{!v='(((viewFieldPermission_customFields_224:80
> viewFieldPermission_customFields_224:22
> viewFieldPermission_customFields_224:106)))'}",
>
>
> "qq_viewFieldPermission_customFields_207":"{!v='(((viewFieldPermission_customFields_207:80
> viewFieldPermission_customFields_207:22
> viewFieldPermission_customFields_207:106)))'}",
>
>
> "qq_viewFieldPermission_customFields_208":"{!v='(((viewFieldPermission_customFields_208:80
> viewFieldPermission_customFields_208:22
> viewFieldPermission_customFields_208:106)))'}"}},
>
> "response":{"numFound":927,"start":0,"numFoundExact":true,"docs":[
>
> {
>
> "id":20,
>
> "name":"b301112_27_u_32_u_40.tif",
>
> "permission_customFields_date_200":false,
>
> "permission_customFields_date_201":true,
>
> "permission_customFields_date_202":true,
>
> "permission_customFields_date_207":true,
>
> "permission_customFields_date_208":true,
>
> "permission_customFields_date_209":true,
>
> "permission_customFields_date_216":false,
>
> "permission_customFields_date_217":false,
>
> "permission_customFields_date_218":false,
>
> "permission_customFields_date_219":false,
>
> "permission_customFields_date_222":false,
>
> "permission_customFields_date_223":false,
>
> "permission_customFields_date_160":false,
>
> "permission_customFields_date_224":false,
>
> "permission_customFields_date_225":false,
>
> "permission_customFields_date_226":false,
>
> "permission_customFields_date_100":false,
>
> "permission_customFields_date_231":true,
>
> "permission_customFields_date_170":false,
>
> "permission_customFields_date_235":false},
>
> [2 … 99 other documents in response]
>
> {
>
> "id":4930,
>
> "name":"b307682_03-10.tif",
>
> "permission_customFields_date_200":false,
>
> "permission_customFields_date_201":true,
>
> "permission_customFields_date_202":true,
>
> "permission_customFields_date_207":true,
>
> "permission_customFields_date_208":true,
>
> "permission_customFields_date_209":true,
>
> "permission_customFields_date_216":false,
>
> "permission_customFields_date_217":false,
>
> "permission_customFields_date_218":false,
>
> "permission_customFields_date_219":false,
>
> "permission_customFields_date_222":false,
>
> "permission_customFields_date_223":false,
>
> "permission_customFields_date_160":false,
>
> "permission_customFields_date_224":false,
>
> "permission_customFields_date_225":false,
>
> "permission_customFields_date_226":false,
>
> "permission_customFields_date_100":false,
>
> "permission_customFields_date_231":true,
>
> "permission_customFields_date_170":false,
>
> "permission_customFields_date_235":false}]
>
> },
>
> "debug":{
>
> "rawquerystring":"containers:80",
>
> "querystring":"containers:80",
>
> "parsedquery":"IndexOrDocValuesQuery(containers:[80 TO 80])",
>
> "parsedquery_toString":"containers:[80 TO 80]",
>
> "explain":{
>
> "20":"\n1.0 = containers:[80 TO 80]\n",
>
> "76":"\n1.0 = containers:[80 TO 80]\n",
>
> "1514":"\n1.0 = containers:[80 TO 80]\n",
>
> "1565":"\n1.0 = containers:[80 TO 80]\n",
>
> "1566":"\n1.0 = containers:[80 TO 80]\n",
>
> "1607":"\n1.0 = containers:[80 TO 80]\n",
>
> "1675":"\n1.0 = containers:[80 TO 80]\n",
>
> "2489":"\n1.0 = containers:[80 TO 80]\n",
>
> "2642":"\n1.0 = containers:[80 TO 80]\n",
>
> "2643":"\n1.0 = containers:[80 TO 80]\n",
>
> "2680":"\n1.0 = containers:[80 TO 80]\n",
>
> "2681":"\n1.0 = containers:[80 TO 80]\n",
>
> "2682":"\n1.0 = containers:[80 TO 80]\n",
>
> "2683":"\n1.0 = containers:[80 TO 80]\n",
>
> "2684":"\n1.0 = containers:[80 TO 80]\n",
>
> "2685":"\n1.0 = containers:[80 TO 80]\n",
>
> "2686":"\n1.0 = containers:[80 TO 80]\n",
>
> "2696":"\n1.0 = containers:[80 TO 80]\n",
>
> "2697":"\n1.0 = containers:[80 TO 80]\n",
>
> "2698":"\n1.0 = containers:[80 TO 80]\n",
>
> "2699":"\n1.0 = containers:[80 TO 80]\n",
>
> "2700":"\n1.0 = containers:[80 TO 80]\n",
>
> "2701":"\n1.0 = containers:[80 TO 80]\n",
>
> "2702":"\n1.0 = containers:[80 TO 80]\n",
>
> "2719":"\n1.0 = containers:[80 TO 80]\n",
>
> "2720":"\n1.0 = containers:[80 TO 80]\n",
>
> "2727":"\n1.0 = containers:[80 TO 80]\n",
>
> "2739":"\n1.0 = containers:[80 TO 80]\n",
>
> "2740":"\n1.0 = containers:[80 TO 80]\n",
>
> "2741":"\n1.0 = containers:[80 TO 80]\n",
>
> "2742":"\n1.0 = containers:[80 TO 80]\n",
>
> "2743":"\n1.0 = containers:[80 TO 80]\n",
>
> "2744":"\n1.0 = containers:[80 TO 80]\n",
>
> "2745":"\n1.0 = containers:[80 TO 80]\n",
>
> "2746":"\n1.0 = containers:[80 TO 80]\n",
>
> "2747":"\n1.0 = containers:[80 TO 80]\n",
>
> "2748":"\n1.0 = containers:[80 TO 80]\n",
>
> "2749":"\n1.0 = containers:[80 TO 80]\n",
>
> "2750":"\n1.0 = containers:[80 TO 80]\n",
>
> "2755":"\n1.0 = containers:[80 TO 80]\n",
>
> "2756":"\n1.0 = containers:[80 TO 80]\n",
>
> "2757":"\n1.0 = containers:[80 TO 80]\n",
>
> "2758":"\n1.0 = containers:[80 TO 80]\n",
>
> "2759":"\n1.0 = containers:[80 TO 80]\n",
>
> "2760":"\n1.0 = containers:[80 TO 80]\n",
>
> "2761":"\n1.0 = containers:[80 TO 80]\n",
>
> "2762":"\n1.0 = containers:[80 TO 80]\n",
>
> "2763":"\n1.0 = containers:[80 TO 80]\n",
>
> "2799":"\n1.0 = containers:[80 TO 80]\n",
>
> "2800":"\n1.0 = containers:[80 TO 80]\n",
>
> "2801":"\n1.0 = containers:[80 TO 80]\n",
>
> "2802":"\n1.0 = containers:[80 TO 80]\n",
>
> "2803":"\n1.0 = containers:[80 TO 80]\n",
>
> "2804":"\n1.0 = containers:[80 TO 80]\n",
>
> "2805":"\n1.0 = containers:[80 TO 80]\n",
>
> "2829":"\n1.0 = containers:[80 TO 80]\n",
>
> "2838":"\n1.0 = containers:[80 TO 80]\n",
>
> "2839":"\n1.0 = containers:[80 TO 80]\n",
>
> "2840":"\n1.0 = containers:[80 TO 80]\n",
>
> "2841":"\n1.0 = containers:[80 TO 80]\n",
>
> "2842":"\n1.0 = containers:[80 TO 80]\n",
>
> "2843":"\n1.0 = containers:[80 TO 80]\n",
>
> "2878":"\n1.0 = containers:[80 TO 80]\n",
>
> "2889":"\n1.0 = containers:[80 TO 80]\n",
>
> "2911":"\n1.0 = containers:[80 TO 80]\n",
>
> "2945":"\n1.0 = containers:[80 TO 80]\n",
>
> "2946":"\n1.0 = containers:[80 TO 80]\n",
>
> "2947":"\n1.0 = containers:[80 TO 80]\n",
>
> "2972":"\n1.0 = containers:[80 TO 80]\n",
>
> "2973":"\n1.0 = containers:[80 TO 80]\n",
>
> "2974":"\n1.0 = containers:[80 TO 80]\n",
>
> "2975":"\n1.0 = containers:[80 TO 80]\n",
>
> "2976":"\n1.0 = containers:[80 TO 80]\n",
>
> "2999":"\n1.0 = containers:[80 TO 80]\n",
>
> "3001":"\n1.0 = containers:[80 TO 80]\n",
>
> "3062":"\n1.0 = containers:[80 TO 80]\n",
>
> "3063":"\n1.0 = containers:[80 TO 80]\n",
>
> "4335":"\n1.0 = containers:[80 TO 80]\n",
>
> "4604":"\n1.0 = containers:[80 TO 80]\n",
>
> "4606":"\n1.0 = containers:[80 TO 80]\n",
>
> "4608":"\n1.0 = containers:[80 TO 80]\n",
>
> "4705":"\n1.0 = containers:[80 TO 80]\n",
>
> "4706":"\n1.0 = containers:[80 TO 80]\n",
>
> "4707":"\n1.0 = containers:[80 TO 80]\n",
>
> "4723":"\n1.0 = containers:[80 TO 80]\n",
>
> "4724":"\n1.0 = containers:[80 TO 80]\n",
>
> "4732":"\n1.0 = containers:[80 TO 80]\n",
>
> "4736":"\n1.0 = containers:[80 TO 80]\n",
>
> "4748":"\n1.0 = containers:[80 TO 80]\n",
>
> "4749":"\n1.0 = containers:[80 TO 80]\n",
>
> "4764":"\n1.0 = containers:[80 TO 80]\n",
>
> "4766":"\n1.0 = containers:[80 TO 80]\n",
>
> "4768":"\n1.0 = containers:[80 TO 80]\n",
>
> "4769":"\n1.0 = containers:[80 TO 80]\n",
>
> "4792":"\n1.0 = containers:[80 TO 80]\n",
>
> "4793":"\n1.0 = containers:[80 TO 80]\n",
>
> "4794":"\n1.0 = containers:[80 TO 80]\n",
>
> "4815":"\n1.0 = containers:[80 TO 80]\n",
>
> "4883":"\n1.0 = containers:[80 TO 80]\n",
>
> "4930":"\n1.0 = containers:[80 TO 80]\n"},
>
> "QParser":"LuceneQParser",
>
> "filter_queries":["(createdBy:106 assetStatus:0) ((allowPermissions:80
> allowPermissions:22 allowPermissions:106) -assetStatus:0)"],
>
> "parsed_filter_queries":["(IndexOrDocValuesQuery(createdBy:[106 TO
> 106]) IndexOrDocValuesQuery(assetStatus:[0 TO 0]))
> ((IndexOrDocValuesQuery(allowPermissions:[80 TO 80])
> IndexOrDocValuesQuery(allowPermissions:[22 TO 22])
> IndexOrDocValuesQuery(allowPermissions:[106 TO 106]))
> -IndexOrDocValuesQuery(assetStatus:[0 TO 0]))"],
>
> "timing":{
>
> "time":912.0,
>
> "prepare":{
>
> "time":36.0,
>
> "query":{
>
> "time":35.0},
>
> "facet":{
>
> "time":0.0},
>
> "facet_module":{
>
> "time":0.0},
>
> "mlt":{
>
> "time":0.0},
>
> "highlight":{
>
> "time":0.0},
>
> "stats":{
>
> "time":0.0},
>
> "expand":{
>
> "time":0.0},
>
> "terms":{
>
> "time":0.0},
>
> "debug":{
>
> "time":0.0}},
>
> "process":{
>
> "time":870.0,
>
> "query":{
>
> "time":845.0},
>
> "facet":{
>
> "time":0.0},
>
> "facet_module":{
>
> "time":0.0},
>
> "mlt":{
>
> "time":0.0},
>
> "highlight":{
>
> "time":0.0},
>
> "stats":{
>
> "time":0.0},
>
> "expand":{
>
> "time":0.0},
>
> "terms":{
>
> "time":0.0},
>
> "debug":{
>
> "time":23.0}}}}}
>
>
>
> I *cannot explain the very long waiting times to download or deliver* the
> http chunks on localhost. Even the TTFB is much higher than QTime.
>
> *Moreover I have no further idea what I can do to improve the response
> times.*
>
>
>
> *Additionally I have measured solr with yourkit. it takes obviously about
> 11 seconds to process (BKDReader.intersect): *
>
>
>
> When we *remove* more and more of the raw query evaluations
> (permission_customFields_date_202:exists(query($qq_viewFieldPermission_customFields_202))
> etc.) from *fl* the following linear response times could be observed:
>
>
>
> - keep only 10 instead of 15 -> total time of 7 seconds to response
> - keep only 5 -> total time of 2,7 seconds to Response
> - remove completely, then only a few milliseconds are measured (20 –
> 30 ms)
> -
>
> It is comprehensible that the query evaluation and fl parameters influence
> the processing, but it is not clear why the performance degrades that
> dramatic after some modifications in index.
>
> *As mentioned if I drop the whole index and rebuild everything again*,
> the response times are about 1,5 seconds instead of 15 seconds. *“Optimize
> index” didn’t help*.
>
>
>
> Please let me know if someone has some ideas what causes the described
> problems or what we can change to improve.
>
> Please also let me know if any additional details are required from my end.
>
>
>
> Thanks in advance,
>
> ako
>
>
>
>
>
Re: Slow response times when using query paramters
Posted by Jan Høydahl <ja...@cominvent.com>.
I'm sure you have a usecase for the huge "fl", but here you force Solr to evaluate tons of semi-complex queries on each returned document, and I'm sure it is the result enrichment that adds time to your query. The fact that you ask for 100 results only makes it worse since all these needs to be evaluated for 100 docs.
I recently had a similar challence with a customer, where we used "fl" to calculate a normalized score to be returned on every match. It was super slow, so we instead moved that processing to the client side.
So perhaps if you instead return the 20 raw fields needed for the permission calculations, you can move that part to the client side?
Another option is of course to dive into the "slow" code used by "fl" and find ways to optimize it.
Could also be that a custom Document Transformer (https://solr.apache.org/guide/8_9/transforming-result-documents.html) would perform better at this task than the built-in generic functions?
Jan
> 21. sep. 2021 kl. 14:37 skrev Andreas Gusenbauer [CELUM] <An...@celum.com>:
>
> Hi all,
>
> We encountered slow response times when querying solr with additional raw queries.
> The response times decrease by a factor of 10 after deleting and rebuilding the index from scratch. After some minor modifications , the queries response times increase by a factor of 10 again.
>
> Some information:
> solr index size is about 7GB
> 3,1 Mio documents
> Document/filter cache seem to be correctly used (hit count increases when we execute the same query twice)
> We can’t use the result query cache for our query (see below)
> q: containers:80
> fq: (createdBy:106 assetStatus:0) ((allowPermissions:80 allowPermissions:22 allowPermissions:106) -assetStatus:0)
> rows: 100
> 20 Raw query parameters ( e.g qq_viewFieldPermission_customFields_202={!v='(((viewFieldPermission_customFields_202:80 viewFieldPermission_customFields_202:22 viewFieldPermission_customFields_202:106)))'})
> 20 additional fl values for raw query parameters ( e.g permission_customFields_date_202:exists(query($qq_viewFieldPermission_customFields_202)) )
> wt: json ( but xml does not make a difference)
>
>
> Request:
> <solr>/solr/assets/select?fl=id%2Cname%2Cpermission_customFields_date_200%3Aexists(query(%24qq_viewFieldPermission_customFields_200))%2Cpermission_customFields_date_201%3Aexists(query(%24qq_viewFieldPermission_customFields_201))%2Cpermission_customFields_date_202%3Aexists(query(%24qq_viewFieldPermission_customFields_202))%2Cpermission_customFields_date_207%3Aexists(query(%24qq_viewFieldPermission_customFields_207))%2Cpermission_customFields_date_208%3Aexists(query(%24qq_viewFieldPermission_customFields_208))%2Cpermission_customFields_date_209%3Aexists(query(%24qq_viewFieldPermission_customFields_209))%2Cpermission_customFields_date_216%3Aexists(query(%24qq_viewFieldPermission_customFields_216))%2Cpermission_customFields_date_217%3Aexists(query(%24qq_viewFieldPermission_customFields_217))%2Cpermission_customFields_date_218%3Aexists(query(%24qq_viewFieldPermission_customFields_218))%2Cpermission_customFields_date_219%3Aexists(query(%24qq_viewFieldPermission_customFields_219))%2Cpermission_customFields_date_222%3Aexists(query(%24qq_viewFieldPermission_customFields_222))%2Cpermission_customFields_date_223%3Aexists(query(%24qq_viewFieldPermission_customFields_223))%2Cpermission_customFields_date_160%3Aexists(query(%24qq_viewFieldPermission_customFields_160))%2Cpermission_customFields_date_224%3Aexists(query(%24qq_viewFieldPermission_customFields_224))%2Cpermission_customFields_date_225%3Aexists(query(%24qq_viewFieldPermission_customFields_225))%2Cpermission_customFields_date_226%3Aexists(query(%24qq_viewFieldPermission_customFields_226))%2Cpermission_customFields_date_100%3Aexists(query(%24qq_viewFieldPermission_customFields_100))%2Cpermission_customFields_date_231%3Aexists(query(%24qq_viewFieldPermission_customFields_231))%2Cpermission_customFields_date_170%3Aexists(query(%24qq_viewFieldPermission_customFields_170))%2Cpermission_customFields_date_235%3Aexists(query(%24qq_viewFieldPermission_customFields_235))&fq=(createdBy%3A106%20assetStatus%3A0)%20((allowPermissions%3A80%20allowPermissions%3A22%20allowPermissions%3A106)%20-assetStatus%3A0)&q=containers%3A80&qq_viewFieldPermission_customFields_100=%7B!v%3D%27(((viewFieldPermission_customFields_100%3A80%20viewFieldPermission_customFields_100%3A22%20viewFieldPermission_customFields_100%3A106)))%27%7D&qq_viewFieldPermission_customFields_110=%7B!v%3D%27(((viewFieldPermission_customFields_110%3A80%20viewFieldPermission_customFields_110%3A22%20viewFieldPermission_customFields_110%3A106)))%27%7D&qq_viewFieldPermission_customFields_120=%7B!v%3D%27(((viewFieldPermission_customFields_120%3A80%20viewFieldPermission_customFields_120%3A22%20viewFieldPermission_customFields_120%3A106)))%27%7D&qq_viewFieldPermission_customFields_130=%7B!v%3D%27(((viewFieldPermission_customFields_130%3A80%20viewFieldPermission_customFields_130%3A22%20viewFieldPermission_customFields_130%3A106)))%27%7D&qq_viewFieldPermission_customFields_160=%7B!v%3D%27(((viewFieldPermission_customFields_160%3A80%20viewFieldPermission_customFields_160%3A22%20viewFieldPermission_customFields_160%3A106)))%27%7D&qq_viewFieldPermission_customFields_170=%7B!v%3D%27(((viewFieldPermission_customFields_170%3A80%20viewFieldPermission_customFields_170%3A22%20viewFieldPermission_customFields_170%3A106)))%27%7D&qq_viewFieldPermission_customFields_180=%7B!v%3D%27(((viewFieldPermission_customFields_180%3A80%20viewFieldPermission_customFields_180%3A22%20viewFieldPermission_customFields_180%3A106)))%27%7D&qq_viewFieldPermission_customFields_190=%7B!v%3D%27(((viewFieldPermission_customFields_190%3A80%20viewFieldPermission_customFields_190%3A22%20viewFieldPermission_customFields_190%3A106)))%27%7D&qq_viewFieldPermission_customFields_200=%7B!v%3D%27(((viewFieldPermission_customFields_200%3A80%20viewFieldPermission_customFields_200%3A22%20viewFieldPermission_customFields_200%3A106)))%27%7D&qq_viewFieldPermission_customFields_201=%7B!v%3D%27(((viewFieldPermission_customFields_201%3A80%20viewFieldPermission_customFields_201%3A22%20viewFieldPermission_customFields_201%3A106)))%27%7D&qq_viewFieldPermission_customFields_202=%7B!v%3D%27(((viewFieldPermission_customFields_202%3A80%20viewFieldPermission_customFields_202%3A22%20viewFieldPermission_customFields_202%3A106)))%27%7D&qq_viewFieldPermission_customFields_207=%7B!v%3D%27(((viewFieldPermission_customFields_207%3A80%20viewFieldPermission_customFields_207%3A22%20viewFieldPermission_customFields_207%3A106)))%27%7D&qq_viewFieldPermission_customFields_208=%7B!v%3D%27(((viewFieldPermission_customFields_208%3A80%20viewFieldPermission_customFields_208%3A22%20viewFieldPermission_customFields_208%3A106)))%27%7D&qq_viewFieldPermission_customFields_209=%7B!v%3D%27(((viewFieldPermission_customFields_209%3A80%20viewFieldPermission_customFields_209%3A22%20viewFieldPermission_customFields_209%3A106)))%27%7D&qq_viewFieldPermission_customFields_216=%7B!v%3D%27(((viewFieldPermission_customFields_216%3A80%20viewFieldPermission_customFields_216%3A22%20viewFieldPermission_customFields_216%3A106)))%27%7D&qq_viewFieldPermission_customFields_217=%7B!v%3D%27(((viewFieldPermission_customFields_217%3A80%20viewFieldPermission_customFields_217%3A22%20viewFieldPermission_customFields_217%3A106)))%27%7D&qq_viewFieldPermission_customFields_218=%7B!v%3D%27(((viewFieldPermission_customFields_218%3A80%20viewFieldPermission_customFields_218%3A22%20viewFieldPermission_customFields_218%3A106)))%27%7D&qq_viewFieldPermission_customFields_219=%7B!v%3D%27(((viewFieldPermission_customFields_219%3A80%20viewFieldPermission_customFields_219%3A22%20viewFieldPermission_customFields_219%3A106)))%27%7D&qq_viewFieldPermission_customFields_222=%7B!v%3D%27(((viewFieldPermission_customFields_222%3A80%20viewFieldPermission_customFields_222%3A22%20viewFieldPermission_customFields_222%3A106)))%27%7D&qq_viewFieldPermission_customFields_223=%7B!v%3D%27(((viewFieldPermission_customFields_223%3A80%20viewFieldPermission_customFields_223%3A22%20viewFieldPermission_customFields_223%3A106)))%27%7D&qq_viewFieldPermission_customFields_224=%7B!v%3D%27(((viewFieldPermission_customFields_224%3A80%20viewFieldPermission_customFields_224%3A22%20viewFieldPermission_customFields_224%3A106)))%27%7D&qq_viewFieldPermission_customFields_225=%7B!v%3D%27(((viewFieldPermission_customFields_225%3A80%20viewFieldPermission_customFields_225%3A22%20viewFieldPermission_customFields_225%3A106)))%27%7D&qq_viewFieldPermission_customFields_226=%7B!v%3D%27(((viewFieldPermission_customFields_226%3A80%20viewFieldPermission_customFields_226%3A22%20viewFieldPermission_customFields_226%3A106)))%27%7D&qq_viewFieldPermission_customFields_231=%7B!v%3D%27(((viewFieldPermission_customFields_231%3A80%20viewFieldPermission_customFields_231%3A22%20viewFieldPermission_customFields_231%3A106)))%27%7D&qq_viewFieldPermission_customFields_235=%7B!v%3D%27(((viewFieldPermission_customFields_235%3A80%20viewFieldPermission_customFields_235%3A22%20viewFieldPermission_customFields_235%3A106)))%27%7D&qq_viewFieldPermission_customFields_236=%7B!v%3D%27(((viewFieldPermission_customFields_236%3A80%20viewFieldPermission_customFields_236%3A22%20viewFieldPermission_customFields_236%3A106)))%27%7D&qq_viewFieldPermission_customFields_237=%7B!v%3D%27(((viewFieldPermission_customFields_237%3A80%20viewFieldPermission_customFields_237%3A22%20viewFieldPermission_customFields_237%3A106)))%27%7D&qq_viewFieldPermission_customFields_238=%7B!v%3D%27(((viewFieldPermission_customFields_238%3A80%20viewFieldPermission_customFields_238%3A22%20viewFieldPermission_customFields_238%3A106)))%27%7D&qq_viewFieldPermission_customFields_241=%7B!v%3D%27(((viewFieldPermission_customFields_241%3A80%20viewFieldPermission_customFields_241%3A22%20viewFieldPermission_customFields_241%3A106)))%27%7D&qq_viewFieldPermission_customFields_242=%7B!v%3D%27(((viewFieldPermission_customFields_242%3A80%20viewFieldPermission_customFields_242%3A22%20viewFieldPermission_customFields_242%3A106)))%27%7D&rows=100&sort=id%20asc&start=300
>
> When it is executed a second time the QTime is about 4 – 11 but the total response time is about 15 seconds !!
>
>
> When it does NOT come from cache the QTime is about 1 second which seems realistic (see below): But the total response time is 15 seconds and approx. the same as above ( ~ 140 KB)!
> {
> "responseHeader":{
> "status":0,
> "QTime":914,
> "params":{
> "qq_viewFieldPermission_customFields_209":"{!v='(((viewFieldPermission_customFields_209:80 viewFieldPermission_customFields_209:22 viewFieldPermission_customFields_209:106)))'}",
> "fl":"id,name,permission_customFields_date_200:exists(query($qq_viewFieldPermission_customFields_200)),permission_customFields_date_201:exists(query($qq_viewFieldPermission_customFields_201)),permission_customFields_date_202:exists(query($qq_viewFieldPermission_customFields_202)),permission_customFields_date_207:exists(query($qq_viewFieldPermission_customFields_207)),permission_customFields_date_208:exists(query($qq_viewFieldPermission_customFields_208)),permission_customFields_date_209:exists(query($qq_viewFieldPermission_customFields_209)),permission_customFields_date_216:exists(query($qq_viewFieldPermission_customFields_216)),permission_customFields_date_217:exists(query($qq_viewFieldPermission_customFields_217)),permission_customFields_date_218:exists(query($qq_viewFieldPermission_customFields_218)),permission_customFields_date_219:exists(query($qq_viewFieldPermission_customFields_219)),permission_customFields_date_222:exists(query($qq_viewFieldPermission_customFields_222)),permission_customFields_date_223:exists(query($qq_viewFieldPermission_customFields_223)),permission_customFields_date_160:exists(query($qq_viewFieldPermission_customFields_160)),permission_customFields_date_224:exists(query($qq_viewFieldPermission_customFields_224)),permission_customFields_date_225:exists(query($qq_viewFieldPermission_customFields_225)),permission_customFields_date_226:exists(query($qq_viewFieldPermission_customFields_226)),permission_customFields_date_100:exists(query($qq_viewFieldPermission_customFields_100)),permission_customFields_date_231:exists(query($qq_viewFieldPermission_customFields_231)),permission_customFields_date_170:exists(query($qq_viewFieldPermission_customFields_170)),permission_customFields_date_235:exists(query($qq_viewFieldPermission_customFields_235))",
> "fq":"(createdBy:106 assetStatus:0) ((allowPermissions:80 allowPermissions:22 allowPermissions:106) -assetStatus:0)",
> "qq_viewFieldPermission_customFields_170":"{!v='(((viewFieldPermission_customFields_170:80 viewFieldPermission_customFields_170:22 viewFieldPermission_customFields_170:106)))'}",
> "qq_viewFieldPermission_customFields_190":"{!v='(((viewFieldPermission_customFields_190:80 viewFieldPermission_customFields_190:22 viewFieldPermission_customFields_190:106)))'}",
> "qq_viewFieldPermission_customFields_130":"{!v='(((viewFieldPermission_customFields_130:80 viewFieldPermission_customFields_130:22 viewFieldPermission_customFields_130:106)))'}",
> "qq_viewFieldPermission_customFields_110":"{!v='(((viewFieldPermission_customFields_110:80 viewFieldPermission_customFields_110:22 viewFieldPermission_customFields_110:106)))'}",
> "qq_viewFieldPermission_customFields_231":"{!v='(((viewFieldPermission_customFields_231:80 viewFieldPermission_customFields_231:22 viewFieldPermission_customFields_231:106)))'}",
> "qq_viewFieldPermission_customFields_236":"{!v='(((viewFieldPermission_customFields_236:80 viewFieldPermission_customFields_236:22 viewFieldPermission_customFields_236:106)))'}",
> "qq_viewFieldPermission_customFields_237":"{!v='(((viewFieldPermission_customFields_237:80 viewFieldPermission_customFields_237:22 viewFieldPermission_customFields_237:106)))'}",
> "qq_viewFieldPermission_customFields_235":"{!v='(((viewFieldPermission_customFields_235:80 viewFieldPermission_customFields_235:22 viewFieldPermission_customFields_235:106)))'}",
> "qq_viewFieldPermission_customFields_218":"{!v='(((viewFieldPermission_customFields_218:80 viewFieldPermission_customFields_218:22 viewFieldPermission_customFields_218:106)))'}",
> "qq_viewFieldPermission_customFields_219":"{!v='(((viewFieldPermission_customFields_219:80 viewFieldPermission_customFields_219:22 viewFieldPermission_customFields_219:106)))'}",
> "qq_viewFieldPermission_customFields_216":"{!v='(((viewFieldPermission_customFields_216:80 viewFieldPermission_customFields_216:22 viewFieldPermission_customFields_216:106)))'}",
> "qq_viewFieldPermission_customFields_238":"{!v='(((viewFieldPermission_customFields_238:80 viewFieldPermission_customFields_238:22 viewFieldPermission_customFields_238:106)))'}",
> "qq_viewFieldPermission_customFields_217":"{!v='(((viewFieldPermission_customFields_217:80 viewFieldPermission_customFields_217:22 viewFieldPermission_customFields_217:106)))'}",
> "debug":"true",
> "start":"0",
> "sort":"id asc",
> "qq_viewFieldPermission_customFields_180":"{!v='(((viewFieldPermission_customFields_180:80 viewFieldPermission_customFields_180:22 viewFieldPermission_customFields_180:106)))'}",
> "rows":"100",
> "q":"containers:80",
> "qq_viewFieldPermission_customFields_160":"{!v='(((viewFieldPermission_customFields_160:80 viewFieldPermission_customFields_160:22 viewFieldPermission_customFields_160:106)))'}",
> "qq_viewFieldPermission_customFields_100":"{!v='(((viewFieldPermission_customFields_100:80 viewFieldPermission_customFields_100:22 viewFieldPermission_customFields_100:106)))'}",
> "qq_viewFieldPermission_customFields_200":"{!v='(((viewFieldPermission_customFields_200:80 viewFieldPermission_customFields_200:22 viewFieldPermission_customFields_200:106)))'}",
> "qq_viewFieldPermission_customFields_222":"{!v='(((viewFieldPermission_customFields_222:80 viewFieldPermission_customFields_222:22 viewFieldPermission_customFields_222:106)))'}",
> "qq_viewFieldPermission_customFields_120":"{!v='(((viewFieldPermission_customFields_120:80 viewFieldPermission_customFields_120:22 viewFieldPermission_customFields_120:106)))'}",
> "qq_viewFieldPermission_customFields_241":"{!v='(((viewFieldPermission_customFields_241:80 viewFieldPermission_customFields_241:22 viewFieldPermission_customFields_241:106)))'}",
> "qq_viewFieldPermission_customFields_242":"{!v='(((viewFieldPermission_customFields_242:80 viewFieldPermission_customFields_242:22 viewFieldPermission_customFields_242:106)))'}",
> "qq_viewFieldPermission_customFields_225":"{!v='(((viewFieldPermission_customFields_225:80 viewFieldPermission_customFields_225:22 viewFieldPermission_customFields_225:106)))'}",
> "qq_viewFieldPermission_customFields_226":"{!v='(((viewFieldPermission_customFields_226:80 viewFieldPermission_customFields_226:22 viewFieldPermission_customFields_226:106)))'}",
> "qq_viewFieldPermission_customFields_201":"{!v='(((viewFieldPermission_customFields_201:80 viewFieldPermission_customFields_201:22 viewFieldPermission_customFields_201:106)))'}",
> "qq_viewFieldPermission_customFields_223":"{!v='(((viewFieldPermission_customFields_223:80 viewFieldPermission_customFields_223:22 viewFieldPermission_customFields_223:106)))'}",
> "qq_viewFieldPermission_customFields_202":"{!v='(((viewFieldPermission_customFields_202:80 viewFieldPermission_customFields_202:22 viewFieldPermission_customFields_202:106)))'}",
> "qq_viewFieldPermission_customFields_224":"{!v='(((viewFieldPermission_customFields_224:80 viewFieldPermission_customFields_224:22 viewFieldPermission_customFields_224:106)))'}",
> "qq_viewFieldPermission_customFields_207":"{!v='(((viewFieldPermission_customFields_207:80 viewFieldPermission_customFields_207:22 viewFieldPermission_customFields_207:106)))'}",
> "qq_viewFieldPermission_customFields_208":"{!v='(((viewFieldPermission_customFields_208:80 viewFieldPermission_customFields_208:22 viewFieldPermission_customFields_208:106)))'}"}},
> "response":{"numFound":927,"start":0,"numFoundExact":true,"docs":[
> {
> "id":20,
> "name":"b301112_27_u_32_u_40.tif",
> "permission_customFields_date_200":false,
> "permission_customFields_date_201":true,
> "permission_customFields_date_202":true,
> "permission_customFields_date_207":true,
> "permission_customFields_date_208":true,
> "permission_customFields_date_209":true,
> "permission_customFields_date_216":false,
> "permission_customFields_date_217":false,
> "permission_customFields_date_218":false,
> "permission_customFields_date_219":false,
> "permission_customFields_date_222":false,
> "permission_customFields_date_223":false,
> "permission_customFields_date_160":false,
> "permission_customFields_date_224":false,
> "permission_customFields_date_225":false,
> "permission_customFields_date_226":false,
> "permission_customFields_date_100":false,
> "permission_customFields_date_231":true,
> "permission_customFields_date_170":false,
> "permission_customFields_date_235":false},
> [2 … 99 other documents in response]
> {
> "id":4930,
> "name":"b307682_03-10.tif",
> "permission_customFields_date_200":false,
> "permission_customFields_date_201":true,
> "permission_customFields_date_202":true,
> "permission_customFields_date_207":true,
> "permission_customFields_date_208":true,
> "permission_customFields_date_209":true,
> "permission_customFields_date_216":false,
> "permission_customFields_date_217":false,
> "permission_customFields_date_218":false,
> "permission_customFields_date_219":false,
> "permission_customFields_date_222":false,
> "permission_customFields_date_223":false,
> "permission_customFields_date_160":false,
> "permission_customFields_date_224":false,
> "permission_customFields_date_225":false,
> "permission_customFields_date_226":false,
> "permission_customFields_date_100":false,
> "permission_customFields_date_231":true,
> "permission_customFields_date_170":false,
> "permission_customFields_date_235":false}]
> },
> "debug":{
> "rawquerystring":"containers:80",
> "querystring":"containers:80",
> "parsedquery":"IndexOrDocValuesQuery(containers:[80 TO 80])",
> "parsedquery_toString":"containers:[80 TO 80]",
> "explain":{
> "20":"\n1.0 = containers:[80 TO 80]\n",
> "76":"\n1.0 = containers:[80 TO 80]\n",
> "1514":"\n1.0 = containers:[80 TO 80]\n",
> "1565":"\n1.0 = containers:[80 TO 80]\n",
> "1566":"\n1.0 = containers:[80 TO 80]\n",
> "1607":"\n1.0 = containers:[80 TO 80]\n",
> "1675":"\n1.0 = containers:[80 TO 80]\n",
> "2489":"\n1.0 = containers:[80 TO 80]\n",
> "2642":"\n1.0 = containers:[80 TO 80]\n",
> "2643":"\n1.0 = containers:[80 TO 80]\n",
> "2680":"\n1.0 = containers:[80 TO 80]\n",
> "2681":"\n1.0 = containers:[80 TO 80]\n",
> "2682":"\n1.0 = containers:[80 TO 80]\n",
> "2683":"\n1.0 = containers:[80 TO 80]\n",
> "2684":"\n1.0 = containers:[80 TO 80]\n",
> "2685":"\n1.0 = containers:[80 TO 80]\n",
> "2686":"\n1.0 = containers:[80 TO 80]\n",
> "2696":"\n1.0 = containers:[80 TO 80]\n",
> "2697":"\n1.0 = containers:[80 TO 80]\n",
> "2698":"\n1.0 = containers:[80 TO 80]\n",
> "2699":"\n1.0 = containers:[80 TO 80]\n",
> "2700":"\n1.0 = containers:[80 TO 80]\n",
> "2701":"\n1.0 = containers:[80 TO 80]\n",
> "2702":"\n1.0 = containers:[80 TO 80]\n",
> "2719":"\n1.0 = containers:[80 TO 80]\n",
> "2720":"\n1.0 = containers:[80 TO 80]\n",
> "2727":"\n1.0 = containers:[80 TO 80]\n",
> "2739":"\n1.0 = containers:[80 TO 80]\n",
> "2740":"\n1.0 = containers:[80 TO 80]\n",
> "2741":"\n1.0 = containers:[80 TO 80]\n",
> "2742":"\n1.0 = containers:[80 TO 80]\n",
> "2743":"\n1.0 = containers:[80 TO 80]\n",
> "2744":"\n1.0 = containers:[80 TO 80]\n",
> "2745":"\n1.0 = containers:[80 TO 80]\n",
> "2746":"\n1.0 = containers:[80 TO 80]\n",
> "2747":"\n1.0 = containers:[80 TO 80]\n",
> "2748":"\n1.0 = containers:[80 TO 80]\n",
> "2749":"\n1.0 = containers:[80 TO 80]\n",
> "2750":"\n1.0 = containers:[80 TO 80]\n",
> "2755":"\n1.0 = containers:[80 TO 80]\n",
> "2756":"\n1.0 = containers:[80 TO 80]\n",
> "2757":"\n1.0 = containers:[80 TO 80]\n",
> "2758":"\n1.0 = containers:[80 TO 80]\n",
> "2759":"\n1.0 = containers:[80 TO 80]\n",
> "2760":"\n1.0 = containers:[80 TO 80]\n",
> "2761":"\n1.0 = containers:[80 TO 80]\n",
> "2762":"\n1.0 = containers:[80 TO 80]\n",
> "2763":"\n1.0 = containers:[80 TO 80]\n",
> "2799":"\n1.0 = containers:[80 TO 80]\n",
> "2800":"\n1.0 = containers:[80 TO 80]\n",
> "2801":"\n1.0 = containers:[80 TO 80]\n",
> "2802":"\n1.0 = containers:[80 TO 80]\n",
> "2803":"\n1.0 = containers:[80 TO 80]\n",
> "2804":"\n1.0 = containers:[80 TO 80]\n",
> "2805":"\n1.0 = containers:[80 TO 80]\n",
> "2829":"\n1.0 = containers:[80 TO 80]\n",
> "2838":"\n1.0 = containers:[80 TO 80]\n",
> "2839":"\n1.0 = containers:[80 TO 80]\n",
> "2840":"\n1.0 = containers:[80 TO 80]\n",
> "2841":"\n1.0 = containers:[80 TO 80]\n",
> "2842":"\n1.0 = containers:[80 TO 80]\n",
> "2843":"\n1.0 = containers:[80 TO 80]\n",
> "2878":"\n1.0 = containers:[80 TO 80]\n",
> "2889":"\n1.0 = containers:[80 TO 80]\n",
> "2911":"\n1.0 = containers:[80 TO 80]\n",
> "2945":"\n1.0 = containers:[80 TO 80]\n",
> "2946":"\n1.0 = containers:[80 TO 80]\n",
> "2947":"\n1.0 = containers:[80 TO 80]\n",
> "2972":"\n1.0 = containers:[80 TO 80]\n",
> "2973":"\n1.0 = containers:[80 TO 80]\n",
> "2974":"\n1.0 = containers:[80 TO 80]\n",
> "2975":"\n1.0 = containers:[80 TO 80]\n",
> "2976":"\n1.0 = containers:[80 TO 80]\n",
> "2999":"\n1.0 = containers:[80 TO 80]\n",
> "3001":"\n1.0 = containers:[80 TO 80]\n",
> "3062":"\n1.0 = containers:[80 TO 80]\n",
> "3063":"\n1.0 = containers:[80 TO 80]\n",
> "4335":"\n1.0 = containers:[80 TO 80]\n",
> "4604":"\n1.0 = containers:[80 TO 80]\n",
> "4606":"\n1.0 = containers:[80 TO 80]\n",
> "4608":"\n1.0 = containers:[80 TO 80]\n",
> "4705":"\n1.0 = containers:[80 TO 80]\n",
> "4706":"\n1.0 = containers:[80 TO 80]\n",
> "4707":"\n1.0 = containers:[80 TO 80]\n",
> "4723":"\n1.0 = containers:[80 TO 80]\n",
> "4724":"\n1.0 = containers:[80 TO 80]\n",
> "4732":"\n1.0 = containers:[80 TO 80]\n",
> "4736":"\n1.0 = containers:[80 TO 80]\n",
> "4748":"\n1.0 = containers:[80 TO 80]\n",
> "4749":"\n1.0 = containers:[80 TO 80]\n",
> "4764":"\n1.0 = containers:[80 TO 80]\n",
> "4766":"\n1.0 = containers:[80 TO 80]\n",
> "4768":"\n1.0 = containers:[80 TO 80]\n",
> "4769":"\n1.0 = containers:[80 TO 80]\n",
> "4792":"\n1.0 = containers:[80 TO 80]\n",
> "4793":"\n1.0 = containers:[80 TO 80]\n",
> "4794":"\n1.0 = containers:[80 TO 80]\n",
> "4815":"\n1.0 = containers:[80 TO 80]\n",
> "4883":"\n1.0 = containers:[80 TO 80]\n",
> "4930":"\n1.0 = containers:[80 TO 80]\n"},
> "QParser":"LuceneQParser",
> "filter_queries":["(createdBy:106 assetStatus:0) ((allowPermissions:80 allowPermissions:22 allowPermissions:106) -assetStatus:0)"],
> "parsed_filter_queries":["(IndexOrDocValuesQuery(createdBy:[106 TO 106]) IndexOrDocValuesQuery(assetStatus:[0 TO 0])) ((IndexOrDocValuesQuery(allowPermissions:[80 TO 80]) IndexOrDocValuesQuery(allowPermissions:[22 TO 22]) IndexOrDocValuesQuery(allowPermissions:[106 TO 106])) -IndexOrDocValuesQuery(assetStatus:[0 TO 0]))"],
> "timing":{
> "time":912.0,
> "prepare":{
> "time":36.0,
> "query":{
> "time":35.0},
> "facet":{
> "time":0.0},
> "facet_module":{
> "time":0.0},
> "mlt":{
> "time":0.0},
> "highlight":{
> "time":0.0},
> "stats":{
> "time":0.0},
> "expand":{
> "time":0.0},
> "terms":{
> "time":0.0},
> "debug":{
> "time":0.0}},
> "process":{
> "time":870.0,
> "query":{
> "time":845.0},
> "facet":{
> "time":0.0},
> "facet_module":{
> "time":0.0},
> "mlt":{
> "time":0.0},
> "highlight":{
> "time":0.0},
> "stats":{
> "time":0.0},
> "expand":{
> "time":0.0},
> "terms":{
> "time":0.0},
> "debug":{
> "time":23.0}}}}}
>
> I cannot explain the very long waiting times to download or deliver the http chunks on localhost. Even the TTFB is much higher than QTime.
> Moreover I have no further idea what I can do to improve the response times.
>
> Additionally I have measured solr with yourkit. it takes obviously about 11 seconds to process (BKDReader.intersect):
>
>
> When we remove more and more of the raw query evaluations (permission_customFields_date_202:exists(query($qq_viewFieldPermission_customFields_202)) etc.) from fl the following linear response times could be observed:
>
> keep only 10 instead of 15 -> total time of 7 seconds to response
> keep only 5 -> total time of 2,7 seconds to Response
> remove completely, then only a few milliseconds are measured (20 – 30 ms)
>
> It is comprehensible that the query evaluation and fl parameters influence the processing, but it is not clear why the performance degrades that dramatic after some modifications in index.
> As mentioned if I drop the whole index and rebuild everything again, the response times are about 1,5 seconds instead of 15 seconds. “Optimize index” didn’t help.
>
>
> Please let me know if someone has some ideas what causes the described problems or what we can change to improve.
> Please also let me know if any additional details are required from my end.
>
> Thanks in advance,
> ako