You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Harsh Verma <hv...@tableau.com> on 2018/06/28 17:41:38 UTC
Custom FieldType not getting populated in the response
Hi All,
I am trying to fetch values (docValues to be exact) from external redis during query time using a custom written FieldType which I am loading in my solr instance. I notice that my custom FieldType is being initialized based on the schema when loading the core. I also notice that during query time, my custom field type code is called but the results back from solr do not populate the custom field even if I explicitly request it. Also, I am asking the results to be sorted by the custom field but results are not sorted.
Solr version: 7.3.1
Here is my implementation: https://gist.github.com/vermaharsh/042e1cf07070c6d9b3b6cc7eaaf0b49c
Here is my solrconfig.xml: https://gist.github.com/vermaharsh/97d7310b242fd7ba3d8d3a3bda209ac3
Here is my managed-schema: https://gist.github.com/vermaharsh/8a89195377802a6bbcdde9215a2fdaf5
Query that I am making: /solr/redis/select?fl=hits&q=*:*&sort=hits%20asc
Response that I get back: https://gist.github.com/vermaharsh/2f63282b10320c4c35a9f85016fe30c0
Another query with debug flag: solr/redis/select?debugQuery=on&fl=*,hits&q=*:*&sort=hits%20desc
Response: https://gist.github.com/vermaharsh/f7f74a65a5403ecec9310ceb81cb674c
Sort order should be - 10, 9, 8, 6, 5, 4, 3, 2, 1, 7; because, corresponding hits value from redis are - 11111, 900, 800, 600, 500, 400, 300, 200, 100, 7.
Can someone help me identify what am I missing?
Thanks,
Harsh
Copy of discussion on #solr irc channel.
<harsh> I am trying to write a custom solr FieldType which fetches value from external redis instance. I got Solr to load my custom jars and have defined the schema with custom fieldType class. But at query time, I cannot get values in the response.
[08:38] <harsh> I do not see any errors in the logs as well, so cannot tell if something failed
[08:38] <harsh> here is the code for my custom FieldType - https://gist.github.com/vermaharsh/042e1cf07070c6d9b3b6cc7eaaf0b49c
[08:39] <harsh> this is my solrconfig.xml for my custom configset - https://gist.github.com/vermaharsh/97d7310b242fd7ba3d8d3a3bda209ac3
[08:42] <harsh> I basically placed the necessary jars under contrib/redis-field
[08:43] <harsh> and my managed-schema - https://gist.github.com/vermaharsh/8a89195377802a6bbcdde9215a2fdaf5
[08:43] <harsh> I am using solr version 7.3.1
[08:44] <harsh> any idea why I am not getting the value for my custom field type back?
[08:45] <@elyograg> harsh: I'm not familiar with the API to that level. FYI, this should go in #solr -- this channel is for development of Solr itself.
[08:46] <@elyograg> the field is not marked as stored. I wonder if that might tell Solr that it shouldn't be returned at all. (I don't know whether setting stored=true might require something more in your code, though)
[08:46] <harsh> I am using it as docValue, but I can try stored. Though, as you mentioned, not sure if that would need more to be implemented in the code
[08:47] <harsh> I will try #solr channel for the question as well
[08:47] <@elyograg> ah, I didn't scroll to the right enough to see that part. :)
[08:48] <@elyograg> I wonder if you might need useDocValuesAsStored="true".
[08:49] <harsh> I thought that is the default value
[08:51] <harsh> for completeness, this is the query that I am using - /solr/redis/select?fl=hits&q=*:*&sort=hits%20asc
[08:52] <harsh> and response that I got back - https://gist.github.com/vermaharsh/2f63282b10320c4c35a9f85016fe30c0
[08:53] == dataminion [~leahy@c-69-181-118-61.hsd1.ca.comcast.net] has joined #solr-dev
[08:53] <@elyograg> this one's probably going to need to go to the mailing list. with all the pastes you've mentioned here.
[08:57] == dataminion [~leahy@c-69-181-118-61.hsd1.ca.comcast.net] has quit [Ping timeout: 264 seconds]
[08:59] <@elyograg> you may be right about that being default. The code in FieldType.java seems to support that.
[09:00] <@elyograg> if (schemaVersion >= 1.6f) properties |= USE_DOCVALUES_AS_STORED;
[09:02] <harsh> alright, thanks elyograg for looking into it. I will send this to mailing list as well.
RE: Custom FieldType not getting populated in the response
Posted by Harsh Verma <hv...@tableau.com>.
To provide an update here.
I was able to get the plugin to work as expected, but I have to make couple of changes - one to solrconfig.xml and another to my query itself.
1. Disable resultQueryCache in Solr. This fixed the sorting issue for me. After result for a query are cached, updates made to redis for updating the hits count do not affect the final sorting of the results even though new hits value is fetched from redis on each query. I would love to know how can I work around it. If values are fetched on each request, I would like Solr to go ahead and sort the results again as well.
2. Instead of just using the field name to get it in the results, I have to use field() function. So new query has fl=*,hits:field(hits). Again, not ideal and would love to understand what is missing in the implementation that is forcing me to use this function format in query.
Here is the working response - https://gist.github.com/vermaharsh/d61f5e220f0858ccf9938791da3d3bac
~Harsh
From: Harsh Verma
Sent: Thursday, June 28, 2018 10:42 AM
To: solr-user@lucene.apache.org
Subject: Custom FieldType not getting populated in the response
Hi All,
I am trying to fetch values (docValues to be exact) from external redis during query time using a custom written FieldType which I am loading in my solr instance. I notice that my custom FieldType is being initialized based on the schema when loading the core. I also notice that during query time, my custom field type code is called but the results back from solr do not populate the custom field even if I explicitly request it. Also, I am asking the results to be sorted by the custom field but results are not sorted.
Solr version: 7.3.1
Here is my implementation: https://gist.github.com/vermaharsh/042e1cf07070c6d9b3b6cc7eaaf0b49c
Here is my solrconfig.xml: https://gist.github.com/vermaharsh/97d7310b242fd7ba3d8d3a3bda209ac3
Here is my managed-schema: https://gist.github.com/vermaharsh/8a89195377802a6bbcdde9215a2fdaf5
Query that I am making: /solr/redis/select?fl=hits&q=*:*&sort=hits%20asc
Response that I get back: https://gist.github.com/vermaharsh/2f63282b10320c4c35a9f85016fe30c0
Another query with debug flag: solr/redis/select?debugQuery=on&fl=*,hits&q=*:*&sort=hits%20desc
Response: https://gist.github.com/vermaharsh/f7f74a65a5403ecec9310ceb81cb674c
Sort order should be - 10, 9, 8, 6, 5, 4, 3, 2, 1, 7; because, corresponding hits value from redis are - 11111, 900, 800, 600, 500, 400, 300, 200, 100, 7.
Can someone help me identify what am I missing?
Thanks,
Harsh
Copy of discussion on #solr irc channel.
<harsh> I am trying to write a custom solr FieldType which fetches value from external redis instance. I got Solr to load my custom jars and have defined the schema with custom fieldType class. But at query time, I cannot get values in the response.
[08:38] <harsh> I do not see any errors in the logs as well, so cannot tell if something failed
[08:38] <harsh> here is the code for my custom FieldType - https://gist.github.com/vermaharsh/042e1cf07070c6d9b3b6cc7eaaf0b49c
[08:39] <harsh> this is my solrconfig.xml for my custom configset - https://gist.github.com/vermaharsh/97d7310b242fd7ba3d8d3a3bda209ac3
[08:42] <harsh> I basically placed the necessary jars under contrib/redis-field
[08:43] <harsh> and my managed-schema - https://gist.github.com/vermaharsh/8a89195377802a6bbcdde9215a2fdaf5
[08:43] <harsh> I am using solr version 7.3.1
[08:44] <harsh> any idea why I am not getting the value for my custom field type back?
[08:45] <@elyograg> harsh: I'm not familiar with the API to that level. FYI, this should go in #solr -- this channel is for development of Solr itself.
[08:46] <@elyograg> the field is not marked as stored. I wonder if that might tell Solr that it shouldn't be returned at all. (I don't know whether setting stored=true might require something more in your code, though)
[08:46] <harsh> I am using it as docValue, but I can try stored. Though, as you mentioned, not sure if that would need more to be implemented in the code
[08:47] <harsh> I will try #solr channel for the question as well
[08:47] <@elyograg> ah, I didn't scroll to the right enough to see that part. :)
[08:48] <@elyograg> I wonder if you might need useDocValuesAsStored="true".
[08:49] <harsh> I thought that is the default value
[08:51] <harsh> for completeness, this is the query that I am using - /solr/redis/select?fl=hits&q=*:*&sort=hits%20asc
[08:52] <harsh> and response that I got back - https://gist.github.com/vermaharsh/2f63282b10320c4c35a9f85016fe30c0
[08:53] == dataminion [~leahy@c-69-181-118-61.hsd1.ca.comcast.net] has joined #solr-dev
[08:53] <@elyograg> this one's probably going to need to go to the mailing list. with all the pastes you've mentioned here.
[08:57] == dataminion [~leahy@c-69-181-118-61.hsd1.ca.comcast.net] has quit [Ping timeout: 264 seconds]
[08:59] <@elyograg> you may be right about that being default. The code in FieldType.java seems to support that.
[09:00] <@elyograg> if (schemaVersion >= 1.6f) properties |= USE_DOCVALUES_AS_STORED;
[09:02] <harsh> alright, thanks elyograg for looking into it. I will send this to mailing list as well.