You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Kevin Stone <Ke...@jax.org> on 2013/07/19 14:06:27 UTC
custom field type plugin
I have a particular use case that I think might require a custom field type, however I am having trouble getting the plugin to work.
My use case has to do with genetics data, and we are running into several situations were we need to be able to query multiple regions of a chromosome (or gene, or other object types). All that really boils down to is being able to give a number, e.g. 10234, and return documents that have regions containing the number. So you'd have a document with a list like ["10000:16090","400:8000","40123:43564"], and it should come back because 10234 falls between "10000:16090". If there is a better or easier way to do this please speak up. I'd rather not have to use a "join" on another index, because 1) it's more complex to set up, and 2) we might need to join against something else and you can only do one join at a time.
Anyway… I tried creating a field type similar to a PointType just to see if I could get one working. I added the following jars to get it to compile: apache-solr-core-4.0.0,lucene-core-4.0.0,lucene-queries-4.0.0,apache-solr-solrj-4.0.0. I am running solr 4.0.0 on jetty, and put my jar file in a sharedLib folder, and specified it in my solr.xml (I have multiple cores).
After starting up solr, I got the line that it picked up the jar:
INFO: Adding 'file:/blah/blah/lib/CustomPlugins.jar' to classloader
But I get this error about it not being able to find the AbstractSubTypeFieldType class.
Here is the first bit of the trace:
SEVERE: null:java.lang.NoClassDefFoundError: org/apache/solr/schema/AbstractSubTypeFieldType
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:791)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
...etc…
Any hints as to what I did wrong? I can provide source code, or a fuller stack trace, config settings, etc.
Also, I did try to unpack the solr.war, stick my jar in WEB-INF/lib, then repack. However, when I did that, I get a NoClassDefFoundError for my plugin itself.
Thanks,
Kevin
The information in this email, including attachments, may be confidential and is intended solely for the addressee(s). If you believe you received this email by mistake, please notify the sender by return email as soon as possible.
Re: custom field type plugin
Posted by "Smiley, David W." <ds...@mitre.org>.
To eliminate the possibility of errors, you need to buffer the query as
indicated in the wiki. If you don't and you use a super-small maxDistErr
as you tell me you are doing, then you are merely making the probability
of hitting an error small (perhaps even very very small), but not
nonexistent. I wish there was a field type that wrapped all this up so
that users wouldn't have to concern themselves with these tricky details.
I created an issue to track it:
https://issues.apache.org/jira/browse/SOLR-5072
~ David
On 7/24/13 9:26 AM, "Kevin Stone" <Ke...@jax.org> wrote:
>I tried reducing the maxDistErr to "0.01", just to test making it smaller.
>I got maxLevels down to 45, and slightly better query times (Indexing time
>was about the same). However, my queries are not accurate anymore. I need
>to pad by 2 or 3 whole numbers to get a hit now, which won't work in real
>use. I can play with the number a bit more, but I didn't see anything
>wrong when I had it at "0.000000009". I do know about using a small
>decimal value to pad around my coordinates, and I'll probably do that for
>the real implementation, but for testing, whole numbers were working for
>all my edge cases.
>
>-Kevin
>
>On 7/23/13 10:45 PM, "Smiley, David W." <ds...@mitre.org> wrote:
>
>>Kevin,
>>
>>Those are some good query response times but they could be better.
>>You've
>>configured the field type sub-optimally. Look again at
>>http://wiki.apache.org/solr/SpatialForTimeDurations and note in
>>particular
>>maxDistErr. You've left it at the value that comes pre-configured with
>>Solr, 0.000000009, which is ~1 meter measured in degrees, and this value
>>makes no sense when your numeric range is in whole numbers. I suspect
>>you
>>inherited this value from Hoss's slides. **Instead use 1.** (as shown on
>>the wiki). This affects performance in a big way since you've configured
>>the prefixTree to hold 2.22e18 values (calculated via (max-min) /
>>maxDistErr) as opposed to "just" 2e10. Your log shows maxLevels is 50
>>for
>>quad tree. The comments in QuadPrefixTree (and I put them there once)
>>indicate maxLevels of 50 is about as much as is supported. But again,
>>I'm
>>not certain what the limit really is without validating. Hopefully you
>>can stay clear of 50. To do some tests, try querying just on the edge on
>>either side of an indexed value to make sure you match the point and then
>>don't match the indexed point as you would expect based on the
>>instructions. Also, be sure to read more of the details on "Search" on
>>this wiki page in which you are advised to buffer the query shape
>>slightly; you didn't do this in your examples below. This is all a bit
>>of
>>a hack when using a field that internally is using floating point instead
>>of fixed precision.
>>
>>~ David Smiley
>>
>>On 7/23/13 9:32 PM, "Kevin Stone" <Ke...@jax.org> wrote:
>>
>>>Sorry for the late response. I needed to find the time to load a lot of
>>>extra data (closer to what we're anticipating). I have an index with
>>>close
>>>to 220,000 documents, each with at least two coordinate regions anywhere
>>>between -10 billion to +10 billion, but could potentially have up to
>>>maybe
>>>half dozen regions in one document. The reason for the negatives, is
>>>because you can read a chromosome either backwards or forwards, so many
>>>coordinates can be minus.
>>>
>>>Here is the schema field definition:
>>>
>>> <fieldType name="geneticLocation"
>>> class="solr.SpatialRecursivePrefixTreeFieldType"
>>> multiValued="true"
>>> geo="false"
>>> worldBounds="-100000000000 -100000000000 100000000000
>>>100000000000"
>>> distErrPct="0"
>>> maxDistErr="0.000000009"
>>> units="degrees"
>>> />
>>>
>>>
>>>Here is the first query in the log:
>>>
>>>INFO:
>>>geneticLocation{class=org.apache.solr.schema.SpatialRecursivePrefixTreeF
>>>i
>>>e
>>>l
>>>dType,analyzer=org.apache.solr.schema.FieldType$DefaultAnalyzer,args={di
>>>s
>>>t
>>>E
>>>rrPct=0, geo=false, multiValued=true, worldBounds=-100000000000
>>>-100000000000 100000000000 100000000000, maxDistErr=0.000000009,
>>>units=degrees}} strat:
>>>RecursivePrefixTreeStrategy(prefixGridScanLevel:46,SPG:(QuadPrefixTree(m
>>>a
>>>x
>>>L
>>>evels:50,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc,
>>>worldBounds=Rect(minX=-1.0E11,maxX=1.0E11,minY=-1.0E11,maxY=1.0E11)})))
>>>maxLevels: 50
>>>Jul 23, 2013 9:11:45 PM org.apache.solr.core.SolrCore execute
>>>INFO: [testIndex] webapp=/solr path=/select
>>>params={wt=xml&q=humanCoordinate:"Intersects(0+60330+6033041244+10000000
>>>0
>>>0
>>>0
>>>)"&rows=100} hits=81112 status=0 QTime=122
>>>
>>>
>>>
>>>
>>>
>>>Here are some other queries to give different timings (the one above
>>>brings back quite a lot):
>>>
>>>INFO: [testIndex] webapp=/solr path=/select
>>>params={wt=xml&q=humanCoordinate:"Intersects(0+6000000000+6900000000+100
>>>0
>>>0
>>>0
>>>00000)"&rows=100} hits=6031 status=0 QTime=10
>>>Jul 23, 2013 9:13:43 PM org.apache.solr.core.SolrCore execute
>>>INFO: [testIndex] webapp=/solr path=/select
>>>params={wt=xml&q=humanCoordinate:"Intersects(0+0+10000000+10000000000)"&
>>>r
>>>o
>>>w
>>>s=100} hits=500 status=0 QTime=15
>>>Jul 23, 2013 9:14:14 PM org.apache.solr.core.SolrCore execute
>>>INFO: [testIndex] webapp=/solr path=/select
>>>params={wt=xml&q=humanCoordinate:"Intersects(0+7831329+7831329+100000000
>>>0
>>>0
>>>)
>>>"&rows=100} hits=4 status=0 QTime=17
>>>INFO: [testIndex] webapp=/solr path=/select
>>>params={wt=xml&q=humanCoordinate:"Intersects(-10000000000+-1051057963+-1
>>>0
>>>0
>>>1
>>>057963+0)"&rows=100} hits=661 status=0 QTime=8
>>>
>>>
>>>
>>>The query times look pretty fast to me. Certainly I'm pretty impressed.
>>>Our other backup solutions (involving SQL) likely wouldn't even touch
>>>this
>>>in terms of speed.
>>>
>>>
>>>
>>>We will be testing this more in depth in the coming month. I am sort of
>>>jumping ahead of our team to research possible solutions, since this is
>>>something that worried us. Looks like it might work!
>>>
>>>Thanks,
>>>-Kevin
>>>
>>>On 7/23/13 1:47 PM, "David Smiley (@MITRE.org)" <DS...@mitre.org>
>>>wrote:
>>>
>>>>Oh cool! I'm glad it at least seemed to work. Can you post your
>>>>configuration of the field type and report from Solr's logs what the
>>>>"maxLevels" is used for this field, which is logged the first time you
>>>>use
>>>>the field type?
>>>>
>>>>Maybe there isn't a limit under 10B after all. Some quick'n'dirty
>>>>calculations I just did indicate there shouldn't be a problem but
>>>>real-world
>>>>usage will be a better proof. Indexing probably won't be terribly
>>>>slow,
>>>>queries could get pretty slow if the amount of indexed data is really
>>>>high.
>>>>I'd love to hear how it works out for you. Your use-case would benefit
>>>>a
>>>>lot from an improved prefix tree implementation.
>>>>
>>>>I don't gather how a 3rd dimension would play into this. Support for
>>>>multi-dimensional spatial is on the drawing board.
>>>>
>>>>~ David
>>>>
>>>>
>>>>Kevin Stone wrote
>>>>> What are the dangers of trying to use a range of 10 billion? Simply a
>>>>> slower index time? Or will I get inaccurate results?
>>>>> I have tried it on a very small sample of documents, and it seemed to
>>>>> work. I could spend some time this week trying to get a more robust
>>>>>(and
>>>>> accurate) dataset loaded to play around with. The reason for the 10
>>>>> billion is to support being able to query for a region on a
>>>>>chromosome.
>>>>>
>>>>> A user might want to know what genes overlap a point on a specific
>>>>> chromosome. Unless I can use 3 dimensional coordinates (which gave an
>>>>> error when I tried it), I'll need to multiply the coordinates by some
>>>>> offset for each chromosome to be able to normalise the data (at both
>>>>>index
>>>>> and query time). The largest chromosome (chr 1) has almost
>>>>>250,000,000
>>>>> base pairs. I could probably squeeze the rest a bit smaller, but I'd
>>>>> rather use one size for all chromosomes, since we have more than just
>>>>> human data to deal with. It would get quite messy otherwise.
>>>>>
>>>>>
>>>>> On 7/22/13 11:50 AM, "David Smiley (@MITRE.org)" <
>>>>
>>>>> DSMILEY@
>>>>
>>>>> > wrote:
>>>>>
>>>>>>Like Hoss said, you're going to have to solve this using
>>>>>>http://wiki.apache.org/solr/SpatialForTimeDurations
>>>>>>Using PointType is *not* going to work because your durations are
>>>>>>multi-valued per document.
>>>>>>
>>>>>>It would be useful to create a custom field type that wraps the
>>>>>>capability
>>>>>>outlined on the wiki to make it easier to use without requiring the
>>>>>>user
>>>>>>to
>>>>>>think spatially.
>>>>>>
>>>>>>You mentioned that these numeric ranges extend upwards of 10 billion
>>>>>>or
>>>>>>so.
>>>>>>Unfortunately, the current "prefix tree" implementation under the
>>>>>>hood
>>>>>>for
>>>>>>non-geodetic spatial, the QuadTree, is unlikely to scale to numbers
>>>>>>that
>>>>>>big. I don't know where the boundary is, but I doubt 10B. You could
>>>>>>try
>>>>>>and see what happens. I'm working (very slowly on very little spare
>>>>>>time)
>>>>>>on improving the PrefixTree implementations to scale to such large
>>>>>>numbers;
>>>>>>I hope something will be available this fall.
>>>>>>
>>>>>>~ David Smiley
>>>>>>
>>>>>>
>>>>>>Kevin Stone wrote
>>>>>>> I have a particular use case that I think might require a custom
>>>>>>>field
>>>>>>> type, however I am having trouble getting the plugin to work.
>>>>>>> My use case has to do with genetics data, and we are running into
>>>>>>>several
>>>>>>> situations were we need to be able to query multiple regions of a
>>>>>>> chromosome (or gene, or other object types). All that really boils
>>>>>>>down
>>>>>>>to
>>>>>>> is being able to give a number, e.g. 10234, and return documents
>>>>>>>that
>>>>>>>have
>>>>>>> regions containing the number. So you'd have a document with a list
>>>>>>>like
>>>>>>> ["10000:16090","400:8000","40123:43564"], and it should come back
>>>>>>>because
>>>>>>> 10234 falls between "10000:16090". If there is a better or easier
>>>>>>>way
>>>>>>>to
>>>>>>> do this please speak up. I'd rather not have to use a "join" on
>>>>>>>another
>>>>>>> index, because 1) it's more complex to set up, and 2) we might need
>>>>>>>to
>>>>>>> join against something else and you can only do one join at a time.
>>>>>>>
>>>>>>> AnywayŠ I tried creating a field type similar to a PointType just
>>>>>>>to
>>>>>>>see
>>>>>>> if I could get one working. I added the following jars to get it to
>>>>>>> compile:
>>>>>>>
>>>>>>>apache-solr-core-4.0.0,lucene-core-4.0.0,lucene-queries-4.0.0,apache
>>>>>>>-
>>>>>>>s
>>>>>>>o
>>>>>>>lr
>>>>>>>-solrj-4.0.0.
>>>>>>> I am running solr 4.0.0 on jetty, and put my jar file in a
>>>>>>>sharedLib
>>>>>>> folder, and specified it in my solr.xml (I have multiple cores).
>>>>>>>
>>>>>>> After starting up solr, I got the line that it picked up the jar:
>>>>>>> INFO: Adding 'file:/blah/blah/lib/CustomPlugins.jar' to classloader
>>>>>>>
>>>>>>> But I get this error about it not being able to find the
>>>>>>> AbstractSubTypeFieldType class.
>>>>>>> Here is the first bit of the trace:
>>>>>>>
>>>>>>> SEVERE: null:java.lang.NoClassDefFoundError:
>>>>>>> org/apache/solr/schema/AbstractSubTypeFieldType
>>>>>>> at java.lang.ClassLoader.defineClass1(Native Method)
>>>>>>> at java.lang.ClassLoader.defineClass(ClassLoader.java:791)
>>>>>>> at
>>>>>>>java.security.SecureClassLoader.defineClass(SecureClassLoader.java:1
>>>>>>>4
>>>>>>>2
>>>>>>>)
>>>>>>> at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
>>>>>>> at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
>>>>>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
>>>>>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
>>>>>>> ...etcŠ
>>>>>>>
>>>>>>>
>>>>>>> Any hints as to what I did wrong? I can provide source code, or a
>>>>>>>fuller
>>>>>>> stack trace, config settings, etc.
>>>>>>>
>>>>>>> Also, I did try to unpack the solr.war, stick my jar in
>>>>>>>WEB-INF/lib,
>>>>>>>then
>>>>>>> repack. However, when I did that, I get a NoClassDefFoundError for
>>>>>>>my
>>>>>>> plugin itself.
>>>>>>>
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Kevin
>>>>>>>
>>>>>>> The information in this email, including attachments, may be
>>>>>>>confidential
>>>>>>> and is intended solely for the addressee(s). If you believe you
>>>>>>>received
>>>>>>> this email by mistake, please notify the sender by return email as
>>>>>>>soon
>>>>>>>as
>>>>>>> possible.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>-----
>>>>>> Author:
>>>>>>http://www.packtpub.com/apache-solr-3-enterprise-search-server/book
>>>>>>--
>>>>>>View this message in context:
>>>>>>http://lucene.472066.n3.nabble.com/custom-field-type-plugin-tp4079086
>>>>>>p
>>>>>>4
>>>>>>0
>>>>>>79
>>>>>>494.html
>>>>>>Sent from the Solr - User mailing list archive at Nabble.com.
>>>>>
>>>>>
>>>>> The information in this email, including attachments, may be
>>>>>confidential
>>>>> and is intended solely for the addressee(s). If you believe you
>>>>>received
>>>>> this email by mistake, please notify the sender by return email as
>>>>>soon
>>>>>as
>>>>> possible.
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>-----
>>>> Author:
>>>>http://www.packtpub.com/apache-solr-3-enterprise-search-server/book
>>>>--
>>>>View this message in context:
>>>>http://lucene.472066.n3.nabble.com/custom-field-type-plugin-tp4079086p4
>>>>0
>>>>7
>>>>9
>>>>822.html
>>>>Sent from the Solr - User mailing list archive at Nabble.com.
>>>
>>>
>>>The information in this email, including attachments, may be
>>>confidential
>>>and is intended solely for the addressee(s). If you believe you received
>>>this email by mistake, please notify the sender by return email as soon
>>>as possible.
>>
>
>
>The information in this email, including attachments, may be confidential
>and is intended solely for the addressee(s). If you believe you received
>this email by mistake, please notify the sender by return email as soon
>as possible.
Re: custom field type plugin
Posted by Kevin Stone <Ke...@jax.org>.
I tried reducing the maxDistErr to "0.01", just to test making it smaller.
I got maxLevels down to 45, and slightly better query times (Indexing time
was about the same). However, my queries are not accurate anymore. I need
to pad by 2 or 3 whole numbers to get a hit now, which won't work in real
use. I can play with the number a bit more, but I didn't see anything
wrong when I had it at "0.000000009". I do know about using a small
decimal value to pad around my coordinates, and I'll probably do that for
the real implementation, but for testing, whole numbers were working for
all my edge cases.
-Kevin
On 7/23/13 10:45 PM, "Smiley, David W." <ds...@mitre.org> wrote:
>Kevin,
>
>Those are some good query response times but they could be better. You've
>configured the field type sub-optimally. Look again at
>http://wiki.apache.org/solr/SpatialForTimeDurations and note in particular
>maxDistErr. You've left it at the value that comes pre-configured with
>Solr, 0.000000009, which is ~1 meter measured in degrees, and this value
>makes no sense when your numeric range is in whole numbers. I suspect you
>inherited this value from Hoss's slides. **Instead use 1.** (as shown on
>the wiki). This affects performance in a big way since you've configured
>the prefixTree to hold 2.22e18 values (calculated via (max-min) /
>maxDistErr) as opposed to "just" 2e10. Your log shows maxLevels is 50 for
>quad tree. The comments in QuadPrefixTree (and I put them there once)
>indicate maxLevels of 50 is about as much as is supported. But again, I'm
>not certain what the limit really is without validating. Hopefully you
>can stay clear of 50. To do some tests, try querying just on the edge on
>either side of an indexed value to make sure you match the point and then
>don't match the indexed point as you would expect based on the
>instructions. Also, be sure to read more of the details on "Search" on
>this wiki page in which you are advised to buffer the query shape
>slightly; you didn't do this in your examples below. This is all a bit of
>a hack when using a field that internally is using floating point instead
>of fixed precision.
>
>~ David Smiley
>
>On 7/23/13 9:32 PM, "Kevin Stone" <Ke...@jax.org> wrote:
>
>>Sorry for the late response. I needed to find the time to load a lot of
>>extra data (closer to what we're anticipating). I have an index with
>>close
>>to 220,000 documents, each with at least two coordinate regions anywhere
>>between -10 billion to +10 billion, but could potentially have up to
>>maybe
>>half dozen regions in one document. The reason for the negatives, is
>>because you can read a chromosome either backwards or forwards, so many
>>coordinates can be minus.
>>
>>Here is the schema field definition:
>>
>> <fieldType name="geneticLocation"
>> class="solr.SpatialRecursivePrefixTreeFieldType"
>> multiValued="true"
>> geo="false"
>> worldBounds="-100000000000 -100000000000 100000000000
>>100000000000"
>> distErrPct="0"
>> maxDistErr="0.000000009"
>> units="degrees"
>> />
>>
>>
>>Here is the first query in the log:
>>
>>INFO:
>>geneticLocation{class=org.apache.solr.schema.SpatialRecursivePrefixTreeFi
>>e
>>l
>>dType,analyzer=org.apache.solr.schema.FieldType$DefaultAnalyzer,args={dis
>>t
>>E
>>rrPct=0, geo=false, multiValued=true, worldBounds=-100000000000
>>-100000000000 100000000000 100000000000, maxDistErr=0.000000009,
>>units=degrees}} strat:
>>RecursivePrefixTreeStrategy(prefixGridScanLevel:46,SPG:(QuadPrefixTree(ma
>>x
>>L
>>evels:50,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc,
>>worldBounds=Rect(minX=-1.0E11,maxX=1.0E11,minY=-1.0E11,maxY=1.0E11)})))
>>maxLevels: 50
>>Jul 23, 2013 9:11:45 PM org.apache.solr.core.SolrCore execute
>>INFO: [testIndex] webapp=/solr path=/select
>>params={wt=xml&q=humanCoordinate:"Intersects(0+60330+6033041244+100000000
>>0
>>0
>>)"&rows=100} hits=81112 status=0 QTime=122
>>
>>
>>
>>
>>
>>Here are some other queries to give different timings (the one above
>>brings back quite a lot):
>>
>>INFO: [testIndex] webapp=/solr path=/select
>>params={wt=xml&q=humanCoordinate:"Intersects(0+6000000000+6900000000+1000
>>0
>>0
>>00000)"&rows=100} hits=6031 status=0 QTime=10
>>Jul 23, 2013 9:13:43 PM org.apache.solr.core.SolrCore execute
>>INFO: [testIndex] webapp=/solr path=/select
>>params={wt=xml&q=humanCoordinate:"Intersects(0+0+10000000+10000000000)"&r
>>o
>>w
>>s=100} hits=500 status=0 QTime=15
>>Jul 23, 2013 9:14:14 PM org.apache.solr.core.SolrCore execute
>>INFO: [testIndex] webapp=/solr path=/select
>>params={wt=xml&q=humanCoordinate:"Intersects(0+7831329+7831329+1000000000
>>0
>>)
>>"&rows=100} hits=4 status=0 QTime=17
>>INFO: [testIndex] webapp=/solr path=/select
>>params={wt=xml&q=humanCoordinate:"Intersects(-10000000000+-1051057963+-10
>>0
>>1
>>057963+0)"&rows=100} hits=661 status=0 QTime=8
>>
>>
>>
>>The query times look pretty fast to me. Certainly I'm pretty impressed.
>>Our other backup solutions (involving SQL) likely wouldn't even touch
>>this
>>in terms of speed.
>>
>>
>>
>>We will be testing this more in depth in the coming month. I am sort of
>>jumping ahead of our team to research possible solutions, since this is
>>something that worried us. Looks like it might work!
>>
>>Thanks,
>>-Kevin
>>
>>On 7/23/13 1:47 PM, "David Smiley (@MITRE.org)" <DS...@mitre.org>
>>wrote:
>>
>>>Oh cool! I'm glad it at least seemed to work. Can you post your
>>>configuration of the field type and report from Solr's logs what the
>>>"maxLevels" is used for this field, which is logged the first time you
>>>use
>>>the field type?
>>>
>>>Maybe there isn't a limit under 10B after all. Some quick'n'dirty
>>>calculations I just did indicate there shouldn't be a problem but
>>>real-world
>>>usage will be a better proof. Indexing probably won't be terribly slow,
>>>queries could get pretty slow if the amount of indexed data is really
>>>high.
>>>I'd love to hear how it works out for you. Your use-case would benefit
>>>a
>>>lot from an improved prefix tree implementation.
>>>
>>>I don't gather how a 3rd dimension would play into this. Support for
>>>multi-dimensional spatial is on the drawing board.
>>>
>>>~ David
>>>
>>>
>>>Kevin Stone wrote
>>>> What are the dangers of trying to use a range of 10 billion? Simply a
>>>> slower index time? Or will I get inaccurate results?
>>>> I have tried it on a very small sample of documents, and it seemed to
>>>> work. I could spend some time this week trying to get a more robust
>>>>(and
>>>> accurate) dataset loaded to play around with. The reason for the 10
>>>> billion is to support being able to query for a region on a
>>>>chromosome.
>>>>
>>>> A user might want to know what genes overlap a point on a specific
>>>> chromosome. Unless I can use 3 dimensional coordinates (which gave an
>>>> error when I tried it), I'll need to multiply the coordinates by some
>>>> offset for each chromosome to be able to normalise the data (at both
>>>>index
>>>> and query time). The largest chromosome (chr 1) has almost 250,000,000
>>>> base pairs. I could probably squeeze the rest a bit smaller, but I'd
>>>> rather use one size for all chromosomes, since we have more than just
>>>> human data to deal with. It would get quite messy otherwise.
>>>>
>>>>
>>>> On 7/22/13 11:50 AM, "David Smiley (@MITRE.org)" <
>>>
>>>> DSMILEY@
>>>
>>>> > wrote:
>>>>
>>>>>Like Hoss said, you're going to have to solve this using
>>>>>http://wiki.apache.org/solr/SpatialForTimeDurations
>>>>>Using PointType is *not* going to work because your durations are
>>>>>multi-valued per document.
>>>>>
>>>>>It would be useful to create a custom field type that wraps the
>>>>>capability
>>>>>outlined on the wiki to make it easier to use without requiring the
>>>>>user
>>>>>to
>>>>>think spatially.
>>>>>
>>>>>You mentioned that these numeric ranges extend upwards of 10 billion
>>>>>or
>>>>>so.
>>>>>Unfortunately, the current "prefix tree" implementation under the hood
>>>>>for
>>>>>non-geodetic spatial, the QuadTree, is unlikely to scale to numbers
>>>>>that
>>>>>big. I don't know where the boundary is, but I doubt 10B. You could
>>>>>try
>>>>>and see what happens. I'm working (very slowly on very little spare
>>>>>time)
>>>>>on improving the PrefixTree implementations to scale to such large
>>>>>numbers;
>>>>>I hope something will be available this fall.
>>>>>
>>>>>~ David Smiley
>>>>>
>>>>>
>>>>>Kevin Stone wrote
>>>>>> I have a particular use case that I think might require a custom
>>>>>>field
>>>>>> type, however I am having trouble getting the plugin to work.
>>>>>> My use case has to do with genetics data, and we are running into
>>>>>>several
>>>>>> situations were we need to be able to query multiple regions of a
>>>>>> chromosome (or gene, or other object types). All that really boils
>>>>>>down
>>>>>>to
>>>>>> is being able to give a number, e.g. 10234, and return documents
>>>>>>that
>>>>>>have
>>>>>> regions containing the number. So you'd have a document with a list
>>>>>>like
>>>>>> ["10000:16090","400:8000","40123:43564"], and it should come back
>>>>>>because
>>>>>> 10234 falls between "10000:16090". If there is a better or easier
>>>>>>way
>>>>>>to
>>>>>> do this please speak up. I'd rather not have to use a "join" on
>>>>>>another
>>>>>> index, because 1) it's more complex to set up, and 2) we might need
>>>>>>to
>>>>>> join against something else and you can only do one join at a time.
>>>>>>
>>>>>> AnywayŠ I tried creating a field type similar to a PointType just to
>>>>>>see
>>>>>> if I could get one working. I added the following jars to get it to
>>>>>> compile:
>>>>>>
>>>>>>apache-solr-core-4.0.0,lucene-core-4.0.0,lucene-queries-4.0.0,apache-
>>>>>>s
>>>>>>o
>>>>>>lr
>>>>>>-solrj-4.0.0.
>>>>>> I am running solr 4.0.0 on jetty, and put my jar file in a sharedLib
>>>>>> folder, and specified it in my solr.xml (I have multiple cores).
>>>>>>
>>>>>> After starting up solr, I got the line that it picked up the jar:
>>>>>> INFO: Adding 'file:/blah/blah/lib/CustomPlugins.jar' to classloader
>>>>>>
>>>>>> But I get this error about it not being able to find the
>>>>>> AbstractSubTypeFieldType class.
>>>>>> Here is the first bit of the trace:
>>>>>>
>>>>>> SEVERE: null:java.lang.NoClassDefFoundError:
>>>>>> org/apache/solr/schema/AbstractSubTypeFieldType
>>>>>> at java.lang.ClassLoader.defineClass1(Native Method)
>>>>>> at java.lang.ClassLoader.defineClass(ClassLoader.java:791)
>>>>>> at
>>>>>>java.security.SecureClassLoader.defineClass(SecureClassLoader.java:14
>>>>>>2
>>>>>>)
>>>>>> at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
>>>>>> at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
>>>>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
>>>>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
>>>>>> ...etcŠ
>>>>>>
>>>>>>
>>>>>> Any hints as to what I did wrong? I can provide source code, or a
>>>>>>fuller
>>>>>> stack trace, config settings, etc.
>>>>>>
>>>>>> Also, I did try to unpack the solr.war, stick my jar in WEB-INF/lib,
>>>>>>then
>>>>>> repack. However, when I did that, I get a NoClassDefFoundError for
>>>>>>my
>>>>>> plugin itself.
>>>>>>
>>>>>>
>>>>>> Thanks,
>>>>>> Kevin
>>>>>>
>>>>>> The information in this email, including attachments, may be
>>>>>>confidential
>>>>>> and is intended solely for the addressee(s). If you believe you
>>>>>>received
>>>>>> this email by mistake, please notify the sender by return email as
>>>>>>soon
>>>>>>as
>>>>>> possible.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>-----
>>>>> Author:
>>>>>http://www.packtpub.com/apache-solr-3-enterprise-search-server/book
>>>>>--
>>>>>View this message in context:
>>>>>http://lucene.472066.n3.nabble.com/custom-field-type-plugin-tp4079086p
>>>>>4
>>>>>0
>>>>>79
>>>>>494.html
>>>>>Sent from the Solr - User mailing list archive at Nabble.com.
>>>>
>>>>
>>>> The information in this email, including attachments, may be
>>>>confidential
>>>> and is intended solely for the addressee(s). If you believe you
>>>>received
>>>> this email by mistake, please notify the sender by return email as
>>>>soon
>>>>as
>>>> possible.
>>>
>>>
>>>
>>>
>>>
>>>-----
>>> Author:
>>>http://www.packtpub.com/apache-solr-3-enterprise-search-server/book
>>>--
>>>View this message in context:
>>>http://lucene.472066.n3.nabble.com/custom-field-type-plugin-tp4079086p40
>>>7
>>>9
>>>822.html
>>>Sent from the Solr - User mailing list archive at Nabble.com.
>>
>>
>>The information in this email, including attachments, may be confidential
>>and is intended solely for the addressee(s). If you believe you received
>>this email by mistake, please notify the sender by return email as soon
>>as possible.
>
The information in this email, including attachments, may be confidential and is intended solely for the addressee(s). If you believe you received this email by mistake, please notify the sender by return email as soon as possible.
Re: custom field type plugin
Posted by "Smiley, David W." <ds...@mitre.org>.
Kevin,
Those are some good query response times but they could be better. You've
configured the field type sub-optimally. Look again at
http://wiki.apache.org/solr/SpatialForTimeDurations and note in particular
maxDistErr. You've left it at the value that comes pre-configured with
Solr, 0.000000009, which is ~1 meter measured in degrees, and this value
makes no sense when your numeric range is in whole numbers. I suspect you
inherited this value from Hoss's slides. **Instead use 1.** (as shown on
the wiki). This affects performance in a big way since you've configured
the prefixTree to hold 2.22e18 values (calculated via (max-min) /
maxDistErr) as opposed to "just" 2e10. Your log shows maxLevels is 50 for
quad tree. The comments in QuadPrefixTree (and I put them there once)
indicate maxLevels of 50 is about as much as is supported. But again, I'm
not certain what the limit really is without validating. Hopefully you
can stay clear of 50. To do some tests, try querying just on the edge on
either side of an indexed value to make sure you match the point and then
don't match the indexed point as you would expect based on the
instructions. Also, be sure to read more of the details on "Search" on
this wiki page in which you are advised to buffer the query shape
slightly; you didn't do this in your examples below. This is all a bit of
a hack when using a field that internally is using floating point instead
of fixed precision.
~ David Smiley
On 7/23/13 9:32 PM, "Kevin Stone" <Ke...@jax.org> wrote:
>Sorry for the late response. I needed to find the time to load a lot of
>extra data (closer to what we're anticipating). I have an index with close
>to 220,000 documents, each with at least two coordinate regions anywhere
>between -10 billion to +10 billion, but could potentially have up to maybe
>half dozen regions in one document. The reason for the negatives, is
>because you can read a chromosome either backwards or forwards, so many
>coordinates can be minus.
>
>Here is the schema field definition:
>
> <fieldType name="geneticLocation"
> class="solr.SpatialRecursivePrefixTreeFieldType"
> multiValued="true"
> geo="false"
> worldBounds="-100000000000 -100000000000 100000000000
>100000000000"
> distErrPct="0"
> maxDistErr="0.000000009"
> units="degrees"
> />
>
>
>Here is the first query in the log:
>
>INFO:
>geneticLocation{class=org.apache.solr.schema.SpatialRecursivePrefixTreeFie
>l
>dType,analyzer=org.apache.solr.schema.FieldType$DefaultAnalyzer,args={dist
>E
>rrPct=0, geo=false, multiValued=true, worldBounds=-100000000000
>-100000000000 100000000000 100000000000, maxDistErr=0.000000009,
>units=degrees}} strat:
>RecursivePrefixTreeStrategy(prefixGridScanLevel:46,SPG:(QuadPrefixTree(max
>L
>evels:50,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc,
>worldBounds=Rect(minX=-1.0E11,maxX=1.0E11,minY=-1.0E11,maxY=1.0E11)})))
>maxLevels: 50
>Jul 23, 2013 9:11:45 PM org.apache.solr.core.SolrCore execute
>INFO: [testIndex] webapp=/solr path=/select
>params={wt=xml&q=humanCoordinate:"Intersects(0+60330+6033041244+1000000000
>0
>)"&rows=100} hits=81112 status=0 QTime=122
>
>
>
>
>
>Here are some other queries to give different timings (the one above
>brings back quite a lot):
>
>INFO: [testIndex] webapp=/solr path=/select
>params={wt=xml&q=humanCoordinate:"Intersects(0+6000000000+6900000000+10000
>0
>00000)"&rows=100} hits=6031 status=0 QTime=10
>Jul 23, 2013 9:13:43 PM org.apache.solr.core.SolrCore execute
>INFO: [testIndex] webapp=/solr path=/select
>params={wt=xml&q=humanCoordinate:"Intersects(0+0+10000000+10000000000)"&ro
>w
>s=100} hits=500 status=0 QTime=15
>Jul 23, 2013 9:14:14 PM org.apache.solr.core.SolrCore execute
>INFO: [testIndex] webapp=/solr path=/select
>params={wt=xml&q=humanCoordinate:"Intersects(0+7831329+7831329+10000000000
>)
>"&rows=100} hits=4 status=0 QTime=17
>INFO: [testIndex] webapp=/solr path=/select
>params={wt=xml&q=humanCoordinate:"Intersects(-10000000000+-1051057963+-100
>1
>057963+0)"&rows=100} hits=661 status=0 QTime=8
>
>
>
>The query times look pretty fast to me. Certainly I'm pretty impressed.
>Our other backup solutions (involving SQL) likely wouldn't even touch this
>in terms of speed.
>
>
>
>We will be testing this more in depth in the coming month. I am sort of
>jumping ahead of our team to research possible solutions, since this is
>something that worried us. Looks like it might work!
>
>Thanks,
>-Kevin
>
>On 7/23/13 1:47 PM, "David Smiley (@MITRE.org)" <DS...@mitre.org> wrote:
>
>>Oh cool! I'm glad it at least seemed to work. Can you post your
>>configuration of the field type and report from Solr's logs what the
>>"maxLevels" is used for this field, which is logged the first time you
>>use
>>the field type?
>>
>>Maybe there isn't a limit under 10B after all. Some quick'n'dirty
>>calculations I just did indicate there shouldn't be a problem but
>>real-world
>>usage will be a better proof. Indexing probably won't be terribly slow,
>>queries could get pretty slow if the amount of indexed data is really
>>high.
>>I'd love to hear how it works out for you. Your use-case would benefit a
>>lot from an improved prefix tree implementation.
>>
>>I don't gather how a 3rd dimension would play into this. Support for
>>multi-dimensional spatial is on the drawing board.
>>
>>~ David
>>
>>
>>Kevin Stone wrote
>>> What are the dangers of trying to use a range of 10 billion? Simply a
>>> slower index time? Or will I get inaccurate results?
>>> I have tried it on a very small sample of documents, and it seemed to
>>> work. I could spend some time this week trying to get a more robust
>>>(and
>>> accurate) dataset loaded to play around with. The reason for the 10
>>> billion is to support being able to query for a region on a chromosome.
>>>
>>> A user might want to know what genes overlap a point on a specific
>>> chromosome. Unless I can use 3 dimensional coordinates (which gave an
>>> error when I tried it), I'll need to multiply the coordinates by some
>>> offset for each chromosome to be able to normalise the data (at both
>>>index
>>> and query time). The largest chromosome (chr 1) has almost 250,000,000
>>> base pairs. I could probably squeeze the rest a bit smaller, but I'd
>>> rather use one size for all chromosomes, since we have more than just
>>> human data to deal with. It would get quite messy otherwise.
>>>
>>>
>>> On 7/22/13 11:50 AM, "David Smiley (@MITRE.org)" <
>>
>>> DSMILEY@
>>
>>> > wrote:
>>>
>>>>Like Hoss said, you're going to have to solve this using
>>>>http://wiki.apache.org/solr/SpatialForTimeDurations
>>>>Using PointType is *not* going to work because your durations are
>>>>multi-valued per document.
>>>>
>>>>It would be useful to create a custom field type that wraps the
>>>>capability
>>>>outlined on the wiki to make it easier to use without requiring the
>>>>user
>>>>to
>>>>think spatially.
>>>>
>>>>You mentioned that these numeric ranges extend upwards of 10 billion or
>>>>so.
>>>>Unfortunately, the current "prefix tree" implementation under the hood
>>>>for
>>>>non-geodetic spatial, the QuadTree, is unlikely to scale to numbers
>>>>that
>>>>big. I don't know where the boundary is, but I doubt 10B. You could
>>>>try
>>>>and see what happens. I'm working (very slowly on very little spare
>>>>time)
>>>>on improving the PrefixTree implementations to scale to such large
>>>>numbers;
>>>>I hope something will be available this fall.
>>>>
>>>>~ David Smiley
>>>>
>>>>
>>>>Kevin Stone wrote
>>>>> I have a particular use case that I think might require a custom
>>>>>field
>>>>> type, however I am having trouble getting the plugin to work.
>>>>> My use case has to do with genetics data, and we are running into
>>>>>several
>>>>> situations were we need to be able to query multiple regions of a
>>>>> chromosome (or gene, or other object types). All that really boils
>>>>>down
>>>>>to
>>>>> is being able to give a number, e.g. 10234, and return documents that
>>>>>have
>>>>> regions containing the number. So you'd have a document with a list
>>>>>like
>>>>> ["10000:16090","400:8000","40123:43564"], and it should come back
>>>>>because
>>>>> 10234 falls between "10000:16090". If there is a better or easier way
>>>>>to
>>>>> do this please speak up. I'd rather not have to use a "join" on
>>>>>another
>>>>> index, because 1) it's more complex to set up, and 2) we might need
>>>>>to
>>>>> join against something else and you can only do one join at a time.
>>>>>
>>>>> AnywayŠ I tried creating a field type similar to a PointType just to
>>>>>see
>>>>> if I could get one working. I added the following jars to get it to
>>>>> compile:
>>>>>
>>>>>apache-solr-core-4.0.0,lucene-core-4.0.0,lucene-queries-4.0.0,apache-s
>>>>>o
>>>>>lr
>>>>>-solrj-4.0.0.
>>>>> I am running solr 4.0.0 on jetty, and put my jar file in a sharedLib
>>>>> folder, and specified it in my solr.xml (I have multiple cores).
>>>>>
>>>>> After starting up solr, I got the line that it picked up the jar:
>>>>> INFO: Adding 'file:/blah/blah/lib/CustomPlugins.jar' to classloader
>>>>>
>>>>> But I get this error about it not being able to find the
>>>>> AbstractSubTypeFieldType class.
>>>>> Here is the first bit of the trace:
>>>>>
>>>>> SEVERE: null:java.lang.NoClassDefFoundError:
>>>>> org/apache/solr/schema/AbstractSubTypeFieldType
>>>>> at java.lang.ClassLoader.defineClass1(Native Method)
>>>>> at java.lang.ClassLoader.defineClass(ClassLoader.java:791)
>>>>> at
>>>>>java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142
>>>>>)
>>>>> at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
>>>>> at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
>>>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
>>>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
>>>>> ...etcŠ
>>>>>
>>>>>
>>>>> Any hints as to what I did wrong? I can provide source code, or a
>>>>>fuller
>>>>> stack trace, config settings, etc.
>>>>>
>>>>> Also, I did try to unpack the solr.war, stick my jar in WEB-INF/lib,
>>>>>then
>>>>> repack. However, when I did that, I get a NoClassDefFoundError for my
>>>>> plugin itself.
>>>>>
>>>>>
>>>>> Thanks,
>>>>> Kevin
>>>>>
>>>>> The information in this email, including attachments, may be
>>>>>confidential
>>>>> and is intended solely for the addressee(s). If you believe you
>>>>>received
>>>>> this email by mistake, please notify the sender by return email as
>>>>>soon
>>>>>as
>>>>> possible.
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>-----
>>>> Author:
>>>>http://www.packtpub.com/apache-solr-3-enterprise-search-server/book
>>>>--
>>>>View this message in context:
>>>>http://lucene.472066.n3.nabble.com/custom-field-type-plugin-tp4079086p4
>>>>0
>>>>79
>>>>494.html
>>>>Sent from the Solr - User mailing list archive at Nabble.com.
>>>
>>>
>>> The information in this email, including attachments, may be
>>>confidential
>>> and is intended solely for the addressee(s). If you believe you
>>>received
>>> this email by mistake, please notify the sender by return email as soon
>>>as
>>> possible.
>>
>>
>>
>>
>>
>>-----
>> Author:
>>http://www.packtpub.com/apache-solr-3-enterprise-search-server/book
>>--
>>View this message in context:
>>http://lucene.472066.n3.nabble.com/custom-field-type-plugin-tp4079086p407
>>9
>>822.html
>>Sent from the Solr - User mailing list archive at Nabble.com.
>
>
>The information in this email, including attachments, may be confidential
>and is intended solely for the addressee(s). If you believe you received
>this email by mistake, please notify the sender by return email as soon
>as possible.
Re: custom field type plugin
Posted by Kevin Stone <Ke...@jax.org>.
Sorry for the late response. I needed to find the time to load a lot of
extra data (closer to what we're anticipating). I have an index with close
to 220,000 documents, each with at least two coordinate regions anywhere
between -10 billion to +10 billion, but could potentially have up to maybe
half dozen regions in one document. The reason for the negatives, is
because you can read a chromosome either backwards or forwards, so many
coordinates can be minus.
Here is the schema field definition:
<fieldType name="geneticLocation"
class="solr.SpatialRecursivePrefixTreeFieldType"
multiValued="true"
geo="false"
worldBounds="-100000000000 -100000000000 100000000000
100000000000"
distErrPct="0"
maxDistErr="0.000000009"
units="degrees"
/>
Here is the first query in the log:
INFO:
geneticLocation{class=org.apache.solr.schema.SpatialRecursivePrefixTreeFiel
dType,analyzer=org.apache.solr.schema.FieldType$DefaultAnalyzer,args={distE
rrPct=0, geo=false, multiValued=true, worldBounds=-100000000000
-100000000000 100000000000 100000000000, maxDistErr=0.000000009,
units=degrees}} strat:
RecursivePrefixTreeStrategy(prefixGridScanLevel:46,SPG:(QuadPrefixTree(maxL
evels:50,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc,
worldBounds=Rect(minX=-1.0E11,maxX=1.0E11,minY=-1.0E11,maxY=1.0E11)})))
maxLevels: 50
Jul 23, 2013 9:11:45 PM org.apache.solr.core.SolrCore execute
INFO: [testIndex] webapp=/solr path=/select
params={wt=xml&q=humanCoordinate:"Intersects(0+60330+6033041244+10000000000
)"&rows=100} hits=81112 status=0 QTime=122
Here are some other queries to give different timings (the one above
brings back quite a lot):
INFO: [testIndex] webapp=/solr path=/select
params={wt=xml&q=humanCoordinate:"Intersects(0+6000000000+6900000000+100000
00000)"&rows=100} hits=6031 status=0 QTime=10
Jul 23, 2013 9:13:43 PM org.apache.solr.core.SolrCore execute
INFO: [testIndex] webapp=/solr path=/select
params={wt=xml&q=humanCoordinate:"Intersects(0+0+10000000+10000000000)"&row
s=100} hits=500 status=0 QTime=15
Jul 23, 2013 9:14:14 PM org.apache.solr.core.SolrCore execute
INFO: [testIndex] webapp=/solr path=/select
params={wt=xml&q=humanCoordinate:"Intersects(0+7831329+7831329+10000000000)
"&rows=100} hits=4 status=0 QTime=17
INFO: [testIndex] webapp=/solr path=/select
params={wt=xml&q=humanCoordinate:"Intersects(-10000000000+-1051057963+-1001
057963+0)"&rows=100} hits=661 status=0 QTime=8
The query times look pretty fast to me. Certainly I'm pretty impressed.
Our other backup solutions (involving SQL) likely wouldn't even touch this
in terms of speed.
We will be testing this more in depth in the coming month. I am sort of
jumping ahead of our team to research possible solutions, since this is
something that worried us. Looks like it might work!
Thanks,
-Kevin
On 7/23/13 1:47 PM, "David Smiley (@MITRE.org)" <DS...@mitre.org> wrote:
>Oh cool! I'm glad it at least seemed to work. Can you post your
>configuration of the field type and report from Solr's logs what the
>"maxLevels" is used for this field, which is logged the first time you use
>the field type?
>
>Maybe there isn't a limit under 10B after all. Some quick'n'dirty
>calculations I just did indicate there shouldn't be a problem but
>real-world
>usage will be a better proof. Indexing probably won't be terribly slow,
>queries could get pretty slow if the amount of indexed data is really
>high.
>I'd love to hear how it works out for you. Your use-case would benefit a
>lot from an improved prefix tree implementation.
>
>I don't gather how a 3rd dimension would play into this. Support for
>multi-dimensional spatial is on the drawing board.
>
>~ David
>
>
>Kevin Stone wrote
>> What are the dangers of trying to use a range of 10 billion? Simply a
>> slower index time? Or will I get inaccurate results?
>> I have tried it on a very small sample of documents, and it seemed to
>> work. I could spend some time this week trying to get a more robust (and
>> accurate) dataset loaded to play around with. The reason for the 10
>> billion is to support being able to query for a region on a chromosome.
>>
>> A user might want to know what genes overlap a point on a specific
>> chromosome. Unless I can use 3 dimensional coordinates (which gave an
>> error when I tried it), I'll need to multiply the coordinates by some
>> offset for each chromosome to be able to normalise the data (at both
>>index
>> and query time). The largest chromosome (chr 1) has almost 250,000,000
>> base pairs. I could probably squeeze the rest a bit smaller, but I'd
>> rather use one size for all chromosomes, since we have more than just
>> human data to deal with. It would get quite messy otherwise.
>>
>>
>> On 7/22/13 11:50 AM, "David Smiley (@MITRE.org)" <
>
>> DSMILEY@
>
>> > wrote:
>>
>>>Like Hoss said, you're going to have to solve this using
>>>http://wiki.apache.org/solr/SpatialForTimeDurations
>>>Using PointType is *not* going to work because your durations are
>>>multi-valued per document.
>>>
>>>It would be useful to create a custom field type that wraps the
>>>capability
>>>outlined on the wiki to make it easier to use without requiring the user
>>>to
>>>think spatially.
>>>
>>>You mentioned that these numeric ranges extend upwards of 10 billion or
>>>so.
>>>Unfortunately, the current "prefix tree" implementation under the hood
>>>for
>>>non-geodetic spatial, the QuadTree, is unlikely to scale to numbers that
>>>big. I don't know where the boundary is, but I doubt 10B. You could
>>>try
>>>and see what happens. I'm working (very slowly on very little spare
>>>time)
>>>on improving the PrefixTree implementations to scale to such large
>>>numbers;
>>>I hope something will be available this fall.
>>>
>>>~ David Smiley
>>>
>>>
>>>Kevin Stone wrote
>>>> I have a particular use case that I think might require a custom field
>>>> type, however I am having trouble getting the plugin to work.
>>>> My use case has to do with genetics data, and we are running into
>>>>several
>>>> situations were we need to be able to query multiple regions of a
>>>> chromosome (or gene, or other object types). All that really boils
>>>>down
>>>>to
>>>> is being able to give a number, e.g. 10234, and return documents that
>>>>have
>>>> regions containing the number. So you'd have a document with a list
>>>>like
>>>> ["10000:16090","400:8000","40123:43564"], and it should come back
>>>>because
>>>> 10234 falls between "10000:16090". If there is a better or easier way
>>>>to
>>>> do this please speak up. I'd rather not have to use a "join" on
>>>>another
>>>> index, because 1) it's more complex to set up, and 2) we might need to
>>>> join against something else and you can only do one join at a time.
>>>>
>>>> AnywayŠ I tried creating a field type similar to a PointType just to
>>>>see
>>>> if I could get one working. I added the following jars to get it to
>>>> compile:
>>>>
>>>>apache-solr-core-4.0.0,lucene-core-4.0.0,lucene-queries-4.0.0,apache-so
>>>>lr
>>>>-solrj-4.0.0.
>>>> I am running solr 4.0.0 on jetty, and put my jar file in a sharedLib
>>>> folder, and specified it in my solr.xml (I have multiple cores).
>>>>
>>>> After starting up solr, I got the line that it picked up the jar:
>>>> INFO: Adding 'file:/blah/blah/lib/CustomPlugins.jar' to classloader
>>>>
>>>> But I get this error about it not being able to find the
>>>> AbstractSubTypeFieldType class.
>>>> Here is the first bit of the trace:
>>>>
>>>> SEVERE: null:java.lang.NoClassDefFoundError:
>>>> org/apache/solr/schema/AbstractSubTypeFieldType
>>>> at java.lang.ClassLoader.defineClass1(Native Method)
>>>> at java.lang.ClassLoader.defineClass(ClassLoader.java:791)
>>>> at
>>>>java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
>>>> at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
>>>> at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
>>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
>>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
>>>> ...etcŠ
>>>>
>>>>
>>>> Any hints as to what I did wrong? I can provide source code, or a
>>>>fuller
>>>> stack trace, config settings, etc.
>>>>
>>>> Also, I did try to unpack the solr.war, stick my jar in WEB-INF/lib,
>>>>then
>>>> repack. However, when I did that, I get a NoClassDefFoundError for my
>>>> plugin itself.
>>>>
>>>>
>>>> Thanks,
>>>> Kevin
>>>>
>>>> The information in this email, including attachments, may be
>>>>confidential
>>>> and is intended solely for the addressee(s). If you believe you
>>>>received
>>>> this email by mistake, please notify the sender by return email as
>>>>soon
>>>>as
>>>> possible.
>>>
>>>
>>>
>>>
>>>
>>>-----
>>> Author:
>>>http://www.packtpub.com/apache-solr-3-enterprise-search-server/book
>>>--
>>>View this message in context:
>>>http://lucene.472066.n3.nabble.com/custom-field-type-plugin-tp4079086p40
>>>79
>>>494.html
>>>Sent from the Solr - User mailing list archive at Nabble.com.
>>
>>
>> The information in this email, including attachments, may be
>>confidential
>> and is intended solely for the addressee(s). If you believe you received
>> this email by mistake, please notify the sender by return email as soon
>>as
>> possible.
>
>
>
>
>
>-----
> Author:
>http://www.packtpub.com/apache-solr-3-enterprise-search-server/book
>--
>View this message in context:
>http://lucene.472066.n3.nabble.com/custom-field-type-plugin-tp4079086p4079
>822.html
>Sent from the Solr - User mailing list archive at Nabble.com.
The information in this email, including attachments, may be confidential and is intended solely for the addressee(s). If you believe you received this email by mistake, please notify the sender by return email as soon as possible.
Re: custom field type plugin
Posted by "David Smiley (@MITRE.org)" <DS...@mitre.org>.
Oh cool! I'm glad it at least seemed to work. Can you post your
configuration of the field type and report from Solr's logs what the
"maxLevels" is used for this field, which is logged the first time you use
the field type?
Maybe there isn't a limit under 10B after all. Some quick'n'dirty
calculations I just did indicate there shouldn't be a problem but real-world
usage will be a better proof. Indexing probably won't be terribly slow,
queries could get pretty slow if the amount of indexed data is really high.
I'd love to hear how it works out for you. Your use-case would benefit a
lot from an improved prefix tree implementation.
I don't gather how a 3rd dimension would play into this. Support for
multi-dimensional spatial is on the drawing board.
~ David
Kevin Stone wrote
> What are the dangers of trying to use a range of 10 billion? Simply a
> slower index time? Or will I get inaccurate results?
> I have tried it on a very small sample of documents, and it seemed to
> work. I could spend some time this week trying to get a more robust (and
> accurate) dataset loaded to play around with. The reason for the 10
> billion is to support being able to query for a region on a chromosome.
>
> A user might want to know what genes overlap a point on a specific
> chromosome. Unless I can use 3 dimensional coordinates (which gave an
> error when I tried it), I'll need to multiply the coordinates by some
> offset for each chromosome to be able to normalise the data (at both index
> and query time). The largest chromosome (chr 1) has almost 250,000,000
> base pairs. I could probably squeeze the rest a bit smaller, but I'd
> rather use one size for all chromosomes, since we have more than just
> human data to deal with. It would get quite messy otherwise.
>
>
> On 7/22/13 11:50 AM, "David Smiley (@MITRE.org)" <
> DSMILEY@
> > wrote:
>
>>Like Hoss said, you're going to have to solve this using
>>http://wiki.apache.org/solr/SpatialForTimeDurations
>>Using PointType is *not* going to work because your durations are
>>multi-valued per document.
>>
>>It would be useful to create a custom field type that wraps the capability
>>outlined on the wiki to make it easier to use without requiring the user
>>to
>>think spatially.
>>
>>You mentioned that these numeric ranges extend upwards of 10 billion or
>>so.
>>Unfortunately, the current "prefix tree" implementation under the hood for
>>non-geodetic spatial, the QuadTree, is unlikely to scale to numbers that
>>big. I don't know where the boundary is, but I doubt 10B. You could try
>>and see what happens. I'm working (very slowly on very little spare time)
>>on improving the PrefixTree implementations to scale to such large
>>numbers;
>>I hope something will be available this fall.
>>
>>~ David Smiley
>>
>>
>>Kevin Stone wrote
>>> I have a particular use case that I think might require a custom field
>>> type, however I am having trouble getting the plugin to work.
>>> My use case has to do with genetics data, and we are running into
>>>several
>>> situations were we need to be able to query multiple regions of a
>>> chromosome (or gene, or other object types). All that really boils down
>>>to
>>> is being able to give a number, e.g. 10234, and return documents that
>>>have
>>> regions containing the number. So you'd have a document with a list like
>>> ["10000:16090","400:8000","40123:43564"], and it should come back
>>>because
>>> 10234 falls between "10000:16090". If there is a better or easier way to
>>> do this please speak up. I'd rather not have to use a "join" on another
>>> index, because 1) it's more complex to set up, and 2) we might need to
>>> join against something else and you can only do one join at a time.
>>>
>>> AnywayŠ I tried creating a field type similar to a PointType just to see
>>> if I could get one working. I added the following jars to get it to
>>> compile:
>>>
>>>apache-solr-core-4.0.0,lucene-core-4.0.0,lucene-queries-4.0.0,apache-solr
>>>-solrj-4.0.0.
>>> I am running solr 4.0.0 on jetty, and put my jar file in a sharedLib
>>> folder, and specified it in my solr.xml (I have multiple cores).
>>>
>>> After starting up solr, I got the line that it picked up the jar:
>>> INFO: Adding 'file:/blah/blah/lib/CustomPlugins.jar' to classloader
>>>
>>> But I get this error about it not being able to find the
>>> AbstractSubTypeFieldType class.
>>> Here is the first bit of the trace:
>>>
>>> SEVERE: null:java.lang.NoClassDefFoundError:
>>> org/apache/solr/schema/AbstractSubTypeFieldType
>>> at java.lang.ClassLoader.defineClass1(Native Method)
>>> at java.lang.ClassLoader.defineClass(ClassLoader.java:791)
>>> at
>>>java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
>>> at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
>>> at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
>>> ...etcŠ
>>>
>>>
>>> Any hints as to what I did wrong? I can provide source code, or a fuller
>>> stack trace, config settings, etc.
>>>
>>> Also, I did try to unpack the solr.war, stick my jar in WEB-INF/lib,
>>>then
>>> repack. However, when I did that, I get a NoClassDefFoundError for my
>>> plugin itself.
>>>
>>>
>>> Thanks,
>>> Kevin
>>>
>>> The information in this email, including attachments, may be
>>>confidential
>>> and is intended solely for the addressee(s). If you believe you received
>>> this email by mistake, please notify the sender by return email as soon
>>>as
>>> possible.
>>
>>
>>
>>
>>
>>-----
>> Author:
>>http://www.packtpub.com/apache-solr-3-enterprise-search-server/book
>>--
>>View this message in context:
>>http://lucene.472066.n3.nabble.com/custom-field-type-plugin-tp4079086p4079
>>494.html
>>Sent from the Solr - User mailing list archive at Nabble.com.
>
>
> The information in this email, including attachments, may be confidential
> and is intended solely for the addressee(s). If you believe you received
> this email by mistake, please notify the sender by return email as soon as
> possible.
-----
Author: http://www.packtpub.com/apache-solr-3-enterprise-search-server/book
--
View this message in context: http://lucene.472066.n3.nabble.com/custom-field-type-plugin-tp4079086p4079822.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: custom field type plugin
Posted by Kevin Stone <Ke...@jax.org>.
What are the dangers of trying to use a range of 10 billion? Simply a
slower index time? Or will I get inaccurate results?
I have tried it on a very small sample of documents, and it seemed to
work. I could spend some time this week trying to get a more robust (and
accurate) dataset loaded to play around with. The reason for the 10
billion is to support being able to query for a region on a chromosome.
A user might want to know what genes overlap a point on a specific
chromosome. Unless I can use 3 dimensional coordinates (which gave an
error when I tried it), I'll need to multiply the coordinates by some
offset for each chromosome to be able to normalise the data (at both index
and query time). The largest chromosome (chr 1) has almost 250,000,000
base pairs. I could probably squeeze the rest a bit smaller, but I'd
rather use one size for all chromosomes, since we have more than just
human data to deal with. It would get quite messy otherwise.
On 7/22/13 11:50 AM, "David Smiley (@MITRE.org)" <DS...@mitre.org> wrote:
>Like Hoss said, you're going to have to solve this using
>http://wiki.apache.org/solr/SpatialForTimeDurations
>Using PointType is *not* going to work because your durations are
>multi-valued per document.
>
>It would be useful to create a custom field type that wraps the capability
>outlined on the wiki to make it easier to use without requiring the user
>to
>think spatially.
>
>You mentioned that these numeric ranges extend upwards of 10 billion or
>so.
>Unfortunately, the current "prefix tree" implementation under the hood for
>non-geodetic spatial, the QuadTree, is unlikely to scale to numbers that
>big. I don't know where the boundary is, but I doubt 10B. You could try
>and see what happens. I'm working (very slowly on very little spare time)
>on improving the PrefixTree implementations to scale to such large
>numbers;
>I hope something will be available this fall.
>
>~ David Smiley
>
>
>Kevin Stone wrote
>> I have a particular use case that I think might require a custom field
>> type, however I am having trouble getting the plugin to work.
>> My use case has to do with genetics data, and we are running into
>>several
>> situations were we need to be able to query multiple regions of a
>> chromosome (or gene, or other object types). All that really boils down
>>to
>> is being able to give a number, e.g. 10234, and return documents that
>>have
>> regions containing the number. So you'd have a document with a list like
>> ["10000:16090","400:8000","40123:43564"], and it should come back
>>because
>> 10234 falls between "10000:16090". If there is a better or easier way to
>> do this please speak up. I'd rather not have to use a "join" on another
>> index, because 1) it's more complex to set up, and 2) we might need to
>> join against something else and you can only do one join at a time.
>>
>> AnywayŠ I tried creating a field type similar to a PointType just to see
>> if I could get one working. I added the following jars to get it to
>> compile:
>>
>>apache-solr-core-4.0.0,lucene-core-4.0.0,lucene-queries-4.0.0,apache-solr
>>-solrj-4.0.0.
>> I am running solr 4.0.0 on jetty, and put my jar file in a sharedLib
>> folder, and specified it in my solr.xml (I have multiple cores).
>>
>> After starting up solr, I got the line that it picked up the jar:
>> INFO: Adding 'file:/blah/blah/lib/CustomPlugins.jar' to classloader
>>
>> But I get this error about it not being able to find the
>> AbstractSubTypeFieldType class.
>> Here is the first bit of the trace:
>>
>> SEVERE: null:java.lang.NoClassDefFoundError:
>> org/apache/solr/schema/AbstractSubTypeFieldType
>> at java.lang.ClassLoader.defineClass1(Native Method)
>> at java.lang.ClassLoader.defineClass(ClassLoader.java:791)
>> at
>>java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
>> at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
>> at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
>> at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
>> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
>> ...etcŠ
>>
>>
>> Any hints as to what I did wrong? I can provide source code, or a fuller
>> stack trace, config settings, etc.
>>
>> Also, I did try to unpack the solr.war, stick my jar in WEB-INF/lib,
>>then
>> repack. However, when I did that, I get a NoClassDefFoundError for my
>> plugin itself.
>>
>>
>> Thanks,
>> Kevin
>>
>> The information in this email, including attachments, may be
>>confidential
>> and is intended solely for the addressee(s). If you believe you received
>> this email by mistake, please notify the sender by return email as soon
>>as
>> possible.
>
>
>
>
>
>-----
> Author:
>http://www.packtpub.com/apache-solr-3-enterprise-search-server/book
>--
>View this message in context:
>http://lucene.472066.n3.nabble.com/custom-field-type-plugin-tp4079086p4079
>494.html
>Sent from the Solr - User mailing list archive at Nabble.com.
The information in this email, including attachments, may be confidential and is intended solely for the addressee(s). If you believe you received this email by mistake, please notify the sender by return email as soon as possible.
Re: custom field type plugin
Posted by "David Smiley (@MITRE.org)" <DS...@mitre.org>.
Like Hoss said, you're going to have to solve this using
http://wiki.apache.org/solr/SpatialForTimeDurations
Using PointType is *not* going to work because your durations are
multi-valued per document.
It would be useful to create a custom field type that wraps the capability
outlined on the wiki to make it easier to use without requiring the user to
think spatially.
You mentioned that these numeric ranges extend upwards of 10 billion or so.
Unfortunately, the current "prefix tree" implementation under the hood for
non-geodetic spatial, the QuadTree, is unlikely to scale to numbers that
big. I don't know where the boundary is, but I doubt 10B. You could try
and see what happens. I'm working (very slowly on very little spare time)
on improving the PrefixTree implementations to scale to such large numbers;
I hope something will be available this fall.
~ David Smiley
Kevin Stone wrote
> I have a particular use case that I think might require a custom field
> type, however I am having trouble getting the plugin to work.
> My use case has to do with genetics data, and we are running into several
> situations were we need to be able to query multiple regions of a
> chromosome (or gene, or other object types). All that really boils down to
> is being able to give a number, e.g. 10234, and return documents that have
> regions containing the number. So you'd have a document with a list like
> ["10000:16090","400:8000","40123:43564"], and it should come back because
> 10234 falls between "10000:16090". If there is a better or easier way to
> do this please speak up. I'd rather not have to use a "join" on another
> index, because 1) it's more complex to set up, and 2) we might need to
> join against something else and you can only do one join at a time.
>
> Anyway… I tried creating a field type similar to a PointType just to see
> if I could get one working. I added the following jars to get it to
> compile:
> apache-solr-core-4.0.0,lucene-core-4.0.0,lucene-queries-4.0.0,apache-solr-solrj-4.0.0.
> I am running solr 4.0.0 on jetty, and put my jar file in a sharedLib
> folder, and specified it in my solr.xml (I have multiple cores).
>
> After starting up solr, I got the line that it picked up the jar:
> INFO: Adding 'file:/blah/blah/lib/CustomPlugins.jar' to classloader
>
> But I get this error about it not being able to find the
> AbstractSubTypeFieldType class.
> Here is the first bit of the trace:
>
> SEVERE: null:java.lang.NoClassDefFoundError:
> org/apache/solr/schema/AbstractSubTypeFieldType
> at java.lang.ClassLoader.defineClass1(Native Method)
> at java.lang.ClassLoader.defineClass(ClassLoader.java:791)
> at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
> at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
> at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
> ...etc…
>
>
> Any hints as to what I did wrong? I can provide source code, or a fuller
> stack trace, config settings, etc.
>
> Also, I did try to unpack the solr.war, stick my jar in WEB-INF/lib, then
> repack. However, when I did that, I get a NoClassDefFoundError for my
> plugin itself.
>
>
> Thanks,
> Kevin
>
> The information in this email, including attachments, may be confidential
> and is intended solely for the addressee(s). If you believe you received
> this email by mistake, please notify the sender by return email as soon as
> possible.
-----
Author: http://www.packtpub.com/apache-solr-3-enterprise-search-server/book
--
View this message in context: http://lucene.472066.n3.nabble.com/custom-field-type-plugin-tp4079086p4079494.html
Sent from the Solr - User mailing list archive at Nabble.com.
RE: custom field type plugin
Posted by Kevin Stone <Ke...@jax.org>.
Sorry, I accidentally hit send somehow.
Here is my ant build.xml for building the CustomPlugins jar file:
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<project basedir="." default="jar" name="CustomPlugins">
<property environment="env"/>
<property name="debuglevel" value="source,lines,vars"/>
<property name="target" value="1.6"/>
<property name="source" value="1.6"/>
<path id="CustomPlugins.classpath">
<pathelement location="bin"/>
<pathelement location="lib/apache-solr-core-4.0.0.jar"/>
<pathelement location="lib/lucene-core-4.0.0.jar"/>
<pathelement location="lib/lucene-queries-4.0.0.jar"/>
<pathelement location="lib/apache-solr-solrj-4.0.0.jar"/>
</path>
<target name="init">
<mkdir dir="bin"/>
<copy includeemptydirs="false" todir="bin">
<fileset dir="src">
<exclude name="**/*.launch"/>
<exclude name="**/*.java"/>
</fileset>
</copy>
</target>
<target name="clean">
<delete dir="bin"/>
<delete dir="dist"/>
</target>
<target depends="clean" name="cleanall"/>
<target depends="build-subprojects,build-project" name="build"/>
<target name="build-subprojects"/>
<target depends="init" name="build-project">
<echo message="building project ${ant.project.name}: ${ant.file}"/>
<javac debug="true" debuglevel="DEBUG" destdir="bin" source="${source}" target="${target}">
<src path="src"/>
<classpath refid="CustomPlugins.classpath"/>
</javac>
</target>
<target depends="build" name="jar">
<echo message="${ant.project.name}: ${ant.file}"/>
<mkdir dir="dist"/>
<jar jarfile="dist/CustomPlugins.jar" basedir="bin" includes="**/*.class">
</jar>
</target>
</project>
Here is the code for the GeneticLocation.java file. It is not complete, and might have errors in it. I used PointType as my starting point, and trimmed out what I didn't think I needed. I want to verify that I can load it now before I muck with it any further.
package org.jax.mgi.fe.solrplugin;
import java.io.IOException;
import java.util.ArrayList;
import java.util.List;
import java.util.Map;
import org.apache.lucene.document.FieldType;
import org.apache.lucene.index.IndexableField;
import org.apache.lucene.search.BooleanClause;
import org.apache.lucene.search.BooleanQuery;
import org.apache.lucene.search.Query;
import org.apache.lucene.search.SortField;
import org.apache.solr.common.SolrException;
import org.apache.solr.common.params.MapSolrParams;
import org.apache.solr.common.params.SolrParams;
import org.apache.solr.response.TextResponseWriter;
import org.apache.solr.schema.AbstractSubTypeFieldType;
import org.apache.solr.schema.IndexSchema;
import org.apache.solr.schema.SchemaField;
import org.apache.solr.search.QParser;
import org.apache.lucene.queries.function.ValueSource;
import org.apache.lucene.queries.function.valuesource.VectorValueSource;
/**
* Custom solr field type to support querying against genetic location data.
*/
public class GeneticLocation extends AbstractSubTypeFieldType
{
@Override
protected void init(IndexSchema schema, Map<String, String> args) {
SolrParams p = new MapSolrParams(args);
this.schema = schema;
super.init(schema, args);
// cache suffixes
createSuffixCache(2);
}
@Override
public boolean isPolyField() {
return true; // really only true if the field is indexed
}
@Override
public IndexableField[] createFields(SchemaField field, Object value, float boost)
{
String externalVal = value.toString();
String[] coords = externalVal.split("-");
if(coords.length!=2)
{
throw new SolrException(SolrException.ErrorCode.BAD_REQUEST,"Invalid coordinate format for "+externalVal);
}
List<IndexableField> f = new ArrayList<IndexableField>();
if (field.indexed())
{
SchemaField coordField1 = subField(field,0);
SchemaField coordField2 = subField(field,1);
f.add(coordField1.createField(coords[0],coordField1.indexed() && !coordField1.omitNorms() ? boost : 1f));
f.add(coordField2.createField(coords[1],coordField2.indexed() && !coordField2.omitNorms() ? boost : 1f));
}
if (field.stored())
{
String storedVal = externalVal; // normalize or not?
FieldType customType = new FieldType();
customType.setStored(true);
f.add(createField(field.getName(), storedVal, customType, 1f));
}
return (IndexableField[]) f.toArray();
}
@Override
public ValueSource getValueSource(SchemaField field, QParser parser) {
ArrayList<ValueSource> vs = new ArrayList<ValueSource>();
SchemaField coord1 = subField(field, 0);
vs.add(coord1.getType().getValueSource(coord1, parser));
SchemaField coord2 = subField(field, 1);
vs.add(coord2.getType().getValueSource(coord2, parser));
return new CoordTypeValueSource(field, vs);
}
/**
* It never makes sense to create a single field, so make it impossible to happen by
* throwing UnsupportedOperationException
*
*/
@Override
public IndexableField createField(SchemaField field, Object value, float boost) {
throw new UnsupportedOperationException("PointType uses multiple fields. field=" + field.getName());
}
@Override
public void write(TextResponseWriter writer, String name, IndexableField f) throws IOException {
writer.writeStr(name, f.stringValue(), true);
}
@Override
public SortField getSortField(SchemaField field, boolean top) {
throw new SolrException(SolrException.ErrorCode.BAD_REQUEST, "Sorting not supported on PointType " + field.getName());
}
@Override
public Query getFieldQuery(QParser parser, SchemaField field, String externalVal)
{
String[] coords = externalVal.split("-");
if(coords.length!=2)
{
throw new SolrException(SolrException.ErrorCode.BAD_REQUEST,"Invalid coordinate format for "+externalVal);
}
BooleanQuery bq = new BooleanQuery(true);
SchemaField coord1 = subField(field, 0);
Query tq1 = coord1.getType().getFieldQuery(parser, coord1, coords[0]);
bq.add(tq1, BooleanClause.Occur.MUST);
SchemaField coord2 = subField(field, 1);
Query tq2 = coord2.getType().getFieldQuery(parser, coord1, coords[1]);
bq.add(tq2, BooleanClause.Occur.MUST);
return bq;
}
}
class CoordTypeValueSource extends VectorValueSource
{
private final SchemaField sf;
public CoordTypeValueSource(SchemaField sf, List<ValueSource> sources) {
super(sources);
this.sf = sf;
}
@Override
public String name() {
return "point";
}
@Override
public String description() {
return name()+"("+sf.getName()+")";
}
}
________________________________________
From: Kevin Stone
Sent: Saturday, July 20, 2013 8:24 AM
To: solr-user@lucene.apache.org
Subject: RE: custom field type plugin
Thank you for the links, they really helped me understand. I see how the spatial solution works now. I think this could work as a good backup if I cannot get the custom field type working. The custom field would ideally be a bit more robust than what I mentioned before, because a region really means four pieces, a chromosome (e.g. 1-22), a start base pair, an end base pair, and the direction (forward or reverse). But if need be, the chomosome and direction can be multiplied into the base pairs to get it down to two translated numbers.
As for the upper bounds, I do have an idea, but it would be a large number, say between 1 and 10 billion depending on how I translate the values. I'll just have to try it out I guess.
Ok, now back to the custom field problem. From here on I'll spam source code and stack traces.
I started fresh, removing all places where I may have had my jar file, and popped in a fresh solr.war.
I define the plugin class in my schema like this:
<fieldType name="geneticLocation" class="org.jax.mgi.fe.solrplugin.GeneticLocation" omitNorms="true"/>
and use it here:
<field name="coordinate" type="geneticLocation" indexed="true" stored="true" />
Ok, when I start solr, I get this error saying it can't find the plugin class that is defined in my schema.
org.apache.solr.common.SolrException: Plugin init failure for [schema.xml] fieldType "geneticLocation": Error loading class 'org.jax.mgi.fe.solrplugin.GeneticLocation'
at org.apache.solr.util.plugin.AbstractPluginLoader.load(AbstractPluginLoader.java:177)
at org.apache.solr.schema.IndexSchema.readSchema(IndexSchema.java:369)
at org.apache.solr.schema.IndexSchema.<init>(IndexSchema.java:113)
at org.apache.solr.core.CoreContainer.create(CoreContainer.java:846)
at org.apache.solr.core.CoreContainer.load(CoreContainer.java:534)
at org.apache.solr.core.CoreContainer.load(CoreContainer.java:356)
...etc...
Caused by: org.apache.solr.common.SolrException: Error loading class 'org.jax.mgi.fe.solrplugin.GeneticLocation'
at org.apache.solr.core.SolrResourceLoader.findClass(SolrResourceLoader.java:436)
at org.apache.solr.core.SolrResourceLoader.newInstance(SolrResourceLoader.java:457)
at org.apache.solr.core.SolrResourceLoader.newInstance(SolrResourceLoader.java:453)
...etc...
So, that's all fine.
In my solr.xml, I define this sharedLib folder:
<solr persistent="true" sharedLib="../lib">
I shut the server down, drop in my CustomPlugins.jar file and start the server back up. And... I got a different error! It said I was missing the subFieldType or subFieldSuffix in my fieldType definition. So I added one 'subFieldSuffix="_gl". Then I restart the server thinking that I'm making progress, and I get the old error again. I pulled out the jar, did the above test again to verify that it couldn't find my plugin. Then I re-add it and restart. Nope, still this error about AbstractSubTypeFieldType. Here is the full stack trace:
SEVERE: null:java.lang.NoClassDefFoundError: org/apache/solr/schema/AbstractSubTypeFieldType
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:791)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:423)
at java.lang.ClassLoader.loadClass(ClassLoader.java:356)
at org.mortbay.jetty.webapp.WebAppClassLoader.loadClass(WebAppClassLoader.java:401)
at java.lang.ClassLoader.loadClass(ClassLoader.java:410)
at java.net.FactoryURLClassLoader.loadClass(URLClassLoader.java:789)
at java.lang.ClassLoader.loadClass(ClassLoader.java:410)
at java.net.FactoryURLClassLoader.loadClass(URLClassLoader.java:789)
at java.lang.ClassLoader.loadClass(ClassLoader.java:356)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:266)
at org.apache.solr.core.SolrResourceLoader.findClass(SolrResourceLoader.java:420)
at org.apache.solr.core.SolrResourceLoader.newInstance(SolrResourceLoader.java:457)
at org.apache.solr.core.SolrResourceLoader.newInstance(SolrResourceLoader.java:453)
at org.apache.solr.schema.FieldTypePluginLoader.create(FieldTypePluginLoader.java:81)
at org.apache.solr.schema.FieldTypePluginLoader.create(FieldTypePluginLoader.java:43)
at org.apache.solr.util.plugin.AbstractPluginLoader.load(AbstractPluginLoader.java:151)
at org.apache.solr.schema.IndexSchema.readSchema(IndexSchema.java:369)
at org.apache.solr.schema.IndexSchema.<init>(IndexSchema.java:113)
at org.apache.solr.core.CoreContainer.create(CoreContainer.java:846)
at org.apache.solr.core.CoreContainer.load(CoreContainer.java:534)
at org.apache.solr.core.CoreContainer.load(CoreContainer.java:356)
at org.apache.solr.core.CoreContainer$Initializer.initialize(CoreContainer.java:308)
at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:107)
at org.mortbay.jetty.servlet.FilterHolder.doStart(FilterHolder.java:97)
at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
at org.mortbay.jetty.servlet.ServletHandler.initialize(ServletHandler.java:713)
at org.mortbay.jetty.servlet.Context.startContext(Context.java:140)
at org.mortbay.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1282)
at org.mortbay.jetty.handler.ContextHandler.doStart(ContextHandler.java:518)
at org.mortbay.jetty.webapp.WebAppContext.doStart(WebAppContext.java:499)
at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
at org.mortbay.jetty.handler.HandlerCollection.doStart(HandlerCollection.java:152)
at org.mortbay.jetty.handler.ContextHandlerCollection.doStart(ContextHandlerCollection.java:156)
at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
at org.mortbay.jetty.handler.HandlerCollection.doStart(HandlerCollection.java:152)
at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
at org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java:130)
at org.mortbay.jetty.Server.doStart(Server.java:224)
at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
at org.mortbay.xml.XmlConfiguration.main(XmlConfiguration.java:985)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at org.mortbay.start.Main.invokeMain(Main.java:194)
at org.mortbay.start.Main.start(Main.java:534)
at org.mortbay.start.Main.start(Main.java:441)
at org.mortbay.start.Main.main(Main.java:119)
Caused by: java.lang.ClassNotFoundException: org.apache.solr.schema.AbstractSubTypeFieldType
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:423)
at java.lang.ClassLoader.loadClass(ClassLoader.java:356)
... 57 more
Here is my ant build.xml for building the CustomPlugins jar file:
________________________________________
From: Chris Hostetter [hossman_lucene@fucit.org]
Sent: Friday, July 19, 2013 10:08 PM
To: solr-user@lucene.apache.org
Subject: RE: custom field type plugin
: I can try again this weekend to get a clean environment. However, the
: order I did things in was the reverse of what you suggest. I got the
Hmmm... then i'm kind of at a loss to explain what you're describing.
need to see more details of the configs, dir structure, jar
structure, etc...
: The spatial feature looks intriguing, although I have no idea if it
: could fit my use case. It looks fairly complex a concept, but maybe it
: is all the different shapes and geometry that is confusing me. If I
: thought of my problem in terms of geometry, I would say a chromosome
: region is like a segment of a line. I would need to define multiple line
: segments and be able to query by a single point and only return
: documents that have a line segment that the single point falls on. Does
: that make sense? Is that at all doable with a spatial query?
The tricky thing about leveraging the spatial stuff for this type of
problem is that it's frequently better to *not* let yourself think in
terms of the a straightforward mapping between your problem space and
geometry.
Instead of modeling your data as documents containing multiple line
segments and trying to search for a document containing a line segment
that contains your 1D point, imagine modeling your data as documents
containing multiple 2D points, one point per "range", where the X
coordinate is the lower bound of your range, and the Y axis is the upper
bound of the range...
https://people.apache.org/~hossman/spatial-for-non-spatial-meetup-20130117/#slide8
...and to find all documents containing a range that contains a
specified input value V, you then query for all documents containing
points inside of a specially crafted bounding box based on V...
https://people.apache.org/~hossman/spatial-for-non-spatial-meetup-20130117/#slide11
..the big caveat to this approach that i failed to mention before is that
it presumes there is an absolute min/max definable for the overall range
of values you are dealing with so that you can define the bounding boxes
appropriates -- otherwise the geometery won't work.
In anycase .. it's an interesting idea i wanted to through out there for
you to consider i case it worked for you before you jumped through a tone
of hoops trying to get a new custom FieldType to work.
-Hoss
The information in this email, including attachments, may be confidential and is intended solely for the addressee(s). If you believe you received this email by mistake, please notify the sender by return email as soon as possible.
RE: custom field type plugin
Posted by Kevin Stone <Ke...@jax.org>.
Thank you for the links, they really helped me understand. I see how the spatial solution works now. I think this could work as a good backup if I cannot get the custom field type working. The custom field would ideally be a bit more robust than what I mentioned before, because a region really means four pieces, a chromosome (e.g. 1-22), a start base pair, an end base pair, and the direction (forward or reverse). But if need be, the chomosome and direction can be multiplied into the base pairs to get it down to two translated numbers.
As for the upper bounds, I do have an idea, but it would be a large number, say between 1 and 10 billion depending on how I translate the values. I'll just have to try it out I guess.
Ok, now back to the custom field problem. From here on I'll spam source code and stack traces.
I started fresh, removing all places where I may have had my jar file, and popped in a fresh solr.war.
I define the plugin class in my schema like this:
<fieldType name="geneticLocation" class="org.jax.mgi.fe.solrplugin.GeneticLocation" omitNorms="true"/>
and use it here:
<field name="coordinate" type="geneticLocation" indexed="true" stored="true" />
Ok, when I start solr, I get this error saying it can't find the plugin class that is defined in my schema.
org.apache.solr.common.SolrException: Plugin init failure for [schema.xml] fieldType "geneticLocation": Error loading class 'org.jax.mgi.fe.solrplugin.GeneticLocation'
at org.apache.solr.util.plugin.AbstractPluginLoader.load(AbstractPluginLoader.java:177)
at org.apache.solr.schema.IndexSchema.readSchema(IndexSchema.java:369)
at org.apache.solr.schema.IndexSchema.<init>(IndexSchema.java:113)
at org.apache.solr.core.CoreContainer.create(CoreContainer.java:846)
at org.apache.solr.core.CoreContainer.load(CoreContainer.java:534)
at org.apache.solr.core.CoreContainer.load(CoreContainer.java:356)
...etc...
Caused by: org.apache.solr.common.SolrException: Error loading class 'org.jax.mgi.fe.solrplugin.GeneticLocation'
at org.apache.solr.core.SolrResourceLoader.findClass(SolrResourceLoader.java:436)
at org.apache.solr.core.SolrResourceLoader.newInstance(SolrResourceLoader.java:457)
at org.apache.solr.core.SolrResourceLoader.newInstance(SolrResourceLoader.java:453)
...etc...
So, that's all fine.
In my solr.xml, I define this sharedLib folder:
<solr persistent="true" sharedLib="../lib">
I shut the server down, drop in my CustomPlugins.jar file and start the server back up. And... I got a different error! It said I was missing the subFieldType or subFieldSuffix in my fieldType definition. So I added one 'subFieldSuffix="_gl". Then I restart the server thinking that I'm making progress, and I get the old error again. I pulled out the jar, did the above test again to verify that it couldn't find my plugin. Then I re-add it and restart. Nope, still this error about AbstractSubTypeFieldType. Here is the full stack trace:
SEVERE: null:java.lang.NoClassDefFoundError: org/apache/solr/schema/AbstractSubTypeFieldType
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:791)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:423)
at java.lang.ClassLoader.loadClass(ClassLoader.java:356)
at org.mortbay.jetty.webapp.WebAppClassLoader.loadClass(WebAppClassLoader.java:401)
at java.lang.ClassLoader.loadClass(ClassLoader.java:410)
at java.net.FactoryURLClassLoader.loadClass(URLClassLoader.java:789)
at java.lang.ClassLoader.loadClass(ClassLoader.java:410)
at java.net.FactoryURLClassLoader.loadClass(URLClassLoader.java:789)
at java.lang.ClassLoader.loadClass(ClassLoader.java:356)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:266)
at org.apache.solr.core.SolrResourceLoader.findClass(SolrResourceLoader.java:420)
at org.apache.solr.core.SolrResourceLoader.newInstance(SolrResourceLoader.java:457)
at org.apache.solr.core.SolrResourceLoader.newInstance(SolrResourceLoader.java:453)
at org.apache.solr.schema.FieldTypePluginLoader.create(FieldTypePluginLoader.java:81)
at org.apache.solr.schema.FieldTypePluginLoader.create(FieldTypePluginLoader.java:43)
at org.apache.solr.util.plugin.AbstractPluginLoader.load(AbstractPluginLoader.java:151)
at org.apache.solr.schema.IndexSchema.readSchema(IndexSchema.java:369)
at org.apache.solr.schema.IndexSchema.<init>(IndexSchema.java:113)
at org.apache.solr.core.CoreContainer.create(CoreContainer.java:846)
at org.apache.solr.core.CoreContainer.load(CoreContainer.java:534)
at org.apache.solr.core.CoreContainer.load(CoreContainer.java:356)
at org.apache.solr.core.CoreContainer$Initializer.initialize(CoreContainer.java:308)
at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:107)
at org.mortbay.jetty.servlet.FilterHolder.doStart(FilterHolder.java:97)
at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
at org.mortbay.jetty.servlet.ServletHandler.initialize(ServletHandler.java:713)
at org.mortbay.jetty.servlet.Context.startContext(Context.java:140)
at org.mortbay.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1282)
at org.mortbay.jetty.handler.ContextHandler.doStart(ContextHandler.java:518)
at org.mortbay.jetty.webapp.WebAppContext.doStart(WebAppContext.java:499)
at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
at org.mortbay.jetty.handler.HandlerCollection.doStart(HandlerCollection.java:152)
at org.mortbay.jetty.handler.ContextHandlerCollection.doStart(ContextHandlerCollection.java:156)
at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
at org.mortbay.jetty.handler.HandlerCollection.doStart(HandlerCollection.java:152)
at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
at org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java:130)
at org.mortbay.jetty.Server.doStart(Server.java:224)
at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
at org.mortbay.xml.XmlConfiguration.main(XmlConfiguration.java:985)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at org.mortbay.start.Main.invokeMain(Main.java:194)
at org.mortbay.start.Main.start(Main.java:534)
at org.mortbay.start.Main.start(Main.java:441)
at org.mortbay.start.Main.main(Main.java:119)
Caused by: java.lang.ClassNotFoundException: org.apache.solr.schema.AbstractSubTypeFieldType
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:423)
at java.lang.ClassLoader.loadClass(ClassLoader.java:356)
... 57 more
Here is my ant build.xml for building the CustomPlugins jar file:
________________________________________
From: Chris Hostetter [hossman_lucene@fucit.org]
Sent: Friday, July 19, 2013 10:08 PM
To: solr-user@lucene.apache.org
Subject: RE: custom field type plugin
: I can try again this weekend to get a clean environment. However, the
: order I did things in was the reverse of what you suggest. I got the
Hmmm... then i'm kind of at a loss to explain what you're describing.
need to see more details of the configs, dir structure, jar
structure, etc...
: The spatial feature looks intriguing, although I have no idea if it
: could fit my use case. It looks fairly complex a concept, but maybe it
: is all the different shapes and geometry that is confusing me. If I
: thought of my problem in terms of geometry, I would say a chromosome
: region is like a segment of a line. I would need to define multiple line
: segments and be able to query by a single point and only return
: documents that have a line segment that the single point falls on. Does
: that make sense? Is that at all doable with a spatial query?
The tricky thing about leveraging the spatial stuff for this type of
problem is that it's frequently better to *not* let yourself think in
terms of the a straightforward mapping between your problem space and
geometry.
Instead of modeling your data as documents containing multiple line
segments and trying to search for a document containing a line segment
that contains your 1D point, imagine modeling your data as documents
containing multiple 2D points, one point per "range", where the X
coordinate is the lower bound of your range, and the Y axis is the upper
bound of the range...
https://people.apache.org/~hossman/spatial-for-non-spatial-meetup-20130117/#slide8
...and to find all documents containing a range that contains a
specified input value V, you then query for all documents containing
points inside of a specially crafted bounding box based on V...
https://people.apache.org/~hossman/spatial-for-non-spatial-meetup-20130117/#slide11
..the big caveat to this approach that i failed to mention before is that
it presumes there is an absolute min/max definable for the overall range
of values you are dealing with so that you can define the bounding boxes
appropriates -- otherwise the geometery won't work.
In anycase .. it's an interesting idea i wanted to through out there for
you to consider i case it worked for you before you jumped through a tone
of hoops trying to get a new custom FieldType to work.
-Hoss
The information in this email, including attachments, may be confidential and is intended solely for the addressee(s). If you believe you received this email by mistake, please notify the sender by return email as soon as possible.
RE: custom field type plugin
Posted by Chris Hostetter <ho...@fucit.org>.
: I can try again this weekend to get a clean environment. However, the
: order I did things in was the reverse of what you suggest. I got the
Hmmm... then i'm kind of at a loss to explain what you're describing.
need to see more details of the configs, dir structure, jar
structure, etc...
: The spatial feature looks intriguing, although I have no idea if it
: could fit my use case. It looks fairly complex a concept, but maybe it
: is all the different shapes and geometry that is confusing me. If I
: thought of my problem in terms of geometry, I would say a chromosome
: region is like a segment of a line. I would need to define multiple line
: segments and be able to query by a single point and only return
: documents that have a line segment that the single point falls on. Does
: that make sense? Is that at all doable with a spatial query?
The tricky thing about leveraging the spatial stuff for this type of
problem is that it's frequently better to *not* let yourself think in
terms of the a straightforward mapping between your problem space and
geometry.
Instead of modeling your data as documents containing multiple line
segments and trying to search for a document containing a line segment
that contains your 1D point, imagine modeling your data as documents
containing multiple 2D points, one point per "range", where the X
coordinate is the lower bound of your range, and the Y axis is the upper
bound of the range...
https://people.apache.org/~hossman/spatial-for-non-spatial-meetup-20130117/#slide8
...and to find all documents containing a range that contains a
specified input value V, you then query for all documents containing
points inside of a specially crafted bounding box based on V...
https://people.apache.org/~hossman/spatial-for-non-spatial-meetup-20130117/#slide11
..the big caveat to this approach that i failed to mention before is that
it presumes there is an absolute min/max definable for the overall range
of values you are dealing with so that you can define the bounding boxes
appropriates -- otherwise the geometery won't work.
In anycase .. it's an interesting idea i wanted to through out there for
you to consider i case it worked for you before you jumped through a tone
of hoops trying to get a new custom FieldType to work.
-Hoss
RE: custom field type plugin
Posted by Kevin Stone <Ke...@jax.org>.
I can try again this weekend to get a clean environment. However, the order I did things in was the reverse of what you suggest. I got the AbstractSubTypeFieldType error first. Then I removed my jar from the sharedLib folder, and tried the war repacking solution. That is when I got NoClassDefFoundError on my custom class.
The spatial feature looks intriguing, although I have no idea if it could fit my use case. It looks fairly complex a concept, but maybe it is all the different shapes and geometry that is confusing me. If I thought of my problem in terms of geometry, I would say a chromosome region is like a segment of a line. I would need to define multiple line segments and be able to query by a single point and only return documents that have a line segment that the single point falls on. Does that make sense? Is that at all doable with a spatial query?
-Kevin
________________________________________
From: Chris Hostetter [hossman_lucene@fucit.org]
Sent: Friday, July 19, 2013 3:15 PM
To: solr-user@lucene.apache.org
Subject: Re: custom field type plugin
: a chromosome (or gene, or other object types). All that really boils
: down to is being able to give a number, e.g. 10234, and return documents
: that have regions containing the number. So you'd have a document with a
: list like ["10000:16090","400:8000","40123:43564"], and it should come
You should take a look at some of the build in features using the spatial
types...
http://wiki.apache.org/solr/SpatialForTimeDurations
I believe David also covered this usecase in his talk in san diego...
http://www.lucenerevolution.org/2013/Lucene-Solr4-Spatial-Deep-Dive
: But I get this error about it not being able to find the AbstractSubTypeFieldType class.
: Here is the first bit of the trace:
...
: Any hints as to what I did wrong? I can provide source code, or a fuller stack trace, config settings, etc.
:
: Also, I did try to unpack the solr.war, stick my jar in WEB-INF/lib,
: then repack. However, when I did that, I get a NoClassDefFoundError for
: my plugin itself.
a fuller stack trace might help -- but the key question is what order did
you try these two approaches in? and what exactly did you fieldType
declaration look like?
my guess is that you tried repacking the war first, and maybe your
exploded war classpath is still polluted with your old jar from when you
repacked it and now you have multiple copies in the plugin classloaders
classpath. (the initial NoClassDefFoundError could have been from a
mistake in your <fieldType/> declaration)
try starting competley clean, using the stock war and sample configs and
make sure you get no errors. then try declaring your custom fieldType,
using hte fully qualified classname w/o even telling solr about your jar,
and ensure that you get a NoClassDefFoundError for your custom class -- if
you get an error about AbstractSubTypeFieldType again then you still have
a copy of your custom class somwhere in the classpath. *THEN* try adding
a <lib/> directive to load your jar to load it.
if that still doesn't work provide us with the details of your servlet
container, solr version, the full stack trace, the details of how you are
configuring your <fieldType/>, how you declared the <lib/> what your
filesystem looks like for your solrhome, war, etc...
-Hoss
The information in this email, including attachments, may be confidential and is intended solely for the addressee(s). If you believe you received this email by mistake, please notify the sender by return email as soon as possible.
Re: custom field type plugin
Posted by Chris Hostetter <ho...@fucit.org>.
: a chromosome (or gene, or other object types). All that really boils
: down to is being able to give a number, e.g. 10234, and return documents
: that have regions containing the number. So you'd have a document with a
: list like ["10000:16090","400:8000","40123:43564"], and it should come
You should take a look at some of the build in features using the spatial
types...
http://wiki.apache.org/solr/SpatialForTimeDurations
I believe David also covered this usecase in his talk in san diego...
http://www.lucenerevolution.org/2013/Lucene-Solr4-Spatial-Deep-Dive
: But I get this error about it not being able to find the AbstractSubTypeFieldType class.
: Here is the first bit of the trace:
...
: Any hints as to what I did wrong? I can provide source code, or a fuller stack trace, config settings, etc.
:
: Also, I did try to unpack the solr.war, stick my jar in WEB-INF/lib,
: then repack. However, when I did that, I get a NoClassDefFoundError for
: my plugin itself.
a fuller stack trace might help -- but the key question is what order did
you try these two approaches in? and what exactly did you fieldType
declaration look like?
my guess is that you tried repacking the war first, and maybe your
exploded war classpath is still polluted with your old jar from when you
repacked it and now you have multiple copies in the plugin classloaders
classpath. (the initial NoClassDefFoundError could have been from a
mistake in your <fieldType/> declaration)
try starting competley clean, using the stock war and sample configs and
make sure you get no errors. then try declaring your custom fieldType,
using hte fully qualified classname w/o even telling solr about your jar,
and ensure that you get a NoClassDefFoundError for your custom class -- if
you get an error about AbstractSubTypeFieldType again then you still have
a copy of your custom class somwhere in the classpath. *THEN* try adding
a <lib/> directive to load your jar to load it.
if that still doesn't work provide us with the details of your servlet
container, solr version, the full stack trace, the details of how you are
configuring your <fieldType/>, how you declared the <lib/> what your
filesystem looks like for your solrhome, war, etc...
-Hoss