You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Mark Fenbers <ma...@noaa.gov> on 2015/09/23 13:57:05 UTC
query parsing
When I submit this:
http://localhost:8983/solr/EventLog/select?q=deeper&wt=json&indent=true
then I get these (empty) results:
{
"responseHeader":{
"status":0,
"QTime":1,
"params":{
"q":"deeper",
"indent":"true",
"wt":"json"}},
"response":{"numFound":0,"start":0,"docs":[]
}}
However, if I add asterisks before *and *after "deeper", like this:
http://localhost:8983/solr/EventLog/select?q=*deeper*&wt=json&indent=true
then I get the correct set of results (shown below), as I expect. What
am I doing wrong that the query requires leading and trailing asterisks
to work correctly? If I search on existing text in the username field
instead of the default logtext field, then I don't need to use the
asterisks to get correct results. Does this mean I have a problem in my
indexing process when I used /dataimport. Or does it mean I have
something wrong in my query?
Also, notice in the results that category, logtext, and username fields
are returned as arrays, even though I do not include multiValued="true"
in the schema.xml definition. Why? Attached are my solrconfig.xml and
schema.xml. Any insights would be appreciated!
thanks,
Mark
{
"responseHeader":{
"status":0,
"QTime":9,
"params":{
"q":"*deeper*",
"indent":"true",
"wt":"json"}},
"response":{"numFound":45,"start":0,"docs":[
{
"id":"2012-07-10 13:23:39.0",
"category":[16],
"logtext":["\nHydromet Coordination Message\nOhio River
Forecast Center, Wilmington, OH\n923 AM EDT Tuesday, July 10,
2012\n\nVery slow moving front has sagged down to the southernmost
portion of the\nOhio Valley. This will keep the axis of convection along
or south of the \nTN/KY border today and tomorrow, though some very
light showers are \npossible in the northwest portion of the basin. On
Thursday increased \nsoutherly flow over the Ohio Valley will begin to
draw deeper moisture\nfarther north into the basin, but this will mainly
be after the 48-hour\nforecast cutoff.\n\nDay 1 (8am EDT Tuesday - 8am
EDT Wednesday):\nRain is forecast in southern Kentucky, southern West
Virginia, middle\nTennessee and far western Virginia. Basin average
amounts increase to the\nsouth with come areas approaching an inch.
Light amounts less than 0.10 inch\nare expected in portions of central
Indiana and Ohio. \n\nDay 2 (8am EDT Wednesday - 8am EDT Thursday):
\nRain is forecast all areas south of the Ohio River as well as eastern
\nIllinois, southern Indiana and southwest Pennsylvania. Basin average
amounts\nincrease to the southwest with areas southwest of Nashville
expecting \nover an inch. \n\nQPF from OHRFC, HPC, et al., can be seen
at weather.gov/ohrfc/Forecast.php\n$$\nFor critical after-hours support,
the OHRFC cell number is 937-725-2222.\nLink Crawford "],
"username":["crawford"],
"_version_":1512928764746530816},
{
"id":"2012-07-10 17:39:09.0",
"category":[16],
"logtext":["\nHydromet Coordination Message\nOhio River
Forecast Center, Wilmington, OH\n139 PM EDT Tuesday, July 10,
2012\n\n18Z Discussion:\nMade some changes to the first 6-hour period of
the QPF, but otherwise made\nno changes to the previous
issuance.\n\nPrevious Discussion (12Z):\nVery slow moving front has
sagged down to the southernmost portion of the\nOhio Valley. This will
keep the axis of convection along or south of the \nTN/KY border today
and tomorrow, though some very light showers are \npossible in the
northwest portion of the basin. On Thursday increased \nsoutherly flow
over the Ohio Valley will begin to draw deeper moisture\nfarther north
into the basin, but this will mainly be after the 48-hour\nforecast
cutoff.\n\nDay 1 (8am EDT Tuesday - 8am EDT Wednesday):\nRain is
forecast in southern Kentucky, southern West Virginia, middle\nTennessee
and far western Virginia. Basin average amounts increase to the\nsouth
with come areas approaching an inch. Light amounts less than 0.10
inch\nare expected in portions of central Indiana and Ohio. \n\nDay 2
(8am EDT Wednesday - 8am EDT Thursday): \nRain is forecast all areas
south of the Ohio River as well as eastern \nIllinois, southern Indiana
and southwest Pennsylvania. Basin average amounts\nincrease to the
southwest with areas southwest of Nashville expecting \nover an inch.
\n\nQPF from OHRFC, HPC, et al., can be seen at
weather.gov/ohrfc/Forecast.php\n$$\nFor critical after-hours support,
the OHRFC cell number is 937-725-2222.\nLink Crawford "],
"username":["crawford"],
"_version_":1512928764769599488},
{
"id":"2012-07-11 12:39:56.0",
"category":[16],
"logtext":["\nHydromet Coordination Message\nOhio River
Forecast Center, Wilmington, OH\n839 AM EDT Wednesday, July 11,
2012\n\nOHRFC QPF Discussion (12Z):\n\nDewpoints in the upper 60's and
70's will help fuel showers and thunderstorms \nacross the southern
third of the basin today. This deeper moisture will\nslowly ooze
northward this week as the Bermuda ridge gradually strengthens,\nthus
placing the basin under the influence of a more southwestly flow.
QPF\nis relatively tricky today as models remain in disagreement how
far\nnorth/west the deeper moisture will traverse. We biased our QPF
more toward\nthe Euro which keeps it farther south than all other
models. Despite this\nmodel being an outlier, it goes along with the
prevailing trend. \n\nDay 1 (8am EDT Wednesday - 8am EDT
Thursday):\nShowers and storms are forecast across the southern third of
the basin, and\nto a lesser extent in the higher terrain of WV and
southwest PA. Parts of\nmiddle TN could top 1\" basin-average. \n\nDay 2
(8am EDT Thursday - 8am EDT Friday): \nShowers and storms will develop
farther north than Day 1, into parts of\neastern IL, southern and
central IN, southern and central OH, and much of\nWV. Convection is also
forecast to continue across areas south of the Ohio\nRiver. Again,
heaviest amounts are forecast in middle TN and parts of\nsouthern KY
with over 1\" basin-average. \n\nQPF from OHRFC, HPC, et al., can be
seen at weather.gov/ohrfc/Forecast.php\n$$\nFor critical after-hours
support, the OHRFC cell number is 937-725-2222.\njm "],
"username":["myers"],
"_version_":1512928764810493952},
{
"id":"2012-07-11 18:18:56.0",
"category":[16],
"logtext":["\nHydromet Coordination Message\nOhio River
Forecast Center, Wilmington, OH\n218 PM EDT Wednesday, July 11,
2012\n\n18Z Update: Made some minor changes to the first period to
account for \nconvection firing across the southern and eastern basin
this afternoon.\nOtherwise, besides tagging on an additional period
Friday, no changes were\nmade to the forecast. \n\n\nPrevious Discussion
(12Z):\n\nDewpoints in the upper 60's and 70's will help fuel showers
and thunderstorms \nacross the southern third of the basin today. This
deeper moisture will\nslowly ooze northward this week as the Bermuda
ridge gradually strengthens,\nthus placing the basin under the influence
of a more southwestly flow. QPF\nis relatively tricky today as models
remain in disagreement how far\nnorth/west the deeper moisture will
traverse. We biased our QPF more toward\nthe Euro which keeps it farther
south than all other models. Despite this\nmodel being an outlier, it
goes along with the prevailing trend. \n\nDay 1 (8am EDT Wednesday - 8am
EDT Thursday):\nShowers and storms are forecast across the southern
third of the basin, and\nto a lesser extent in the higher terrain of WV
and southwest PA. Parts of\nmiddle TN could top 1\" basin-average.
\n\nDay 2 (8am EDT Thursday - 8am EDT Friday): \nShowers and storms will
develop farther north than Day 1, into parts of\neastern IL, southern
and central IN, southern and central OH, and much of\nWV. Convection is
also forecast to continue across areas south of the Ohio\nRiver. Again,
heaviest amounts are forecast in middle TN and parts of\nsouthern KY
with over 1\" basin-average. \n\nQPF from OHRFC, HPC, et al., can be
seen at weather.gov/ohrfc/Forecast.php\n$$\nFor critical after-hours
support, the OHRFC cell number is 937-725-2222."],
"username":["oper"],
"_version_":1512928764835659776},
{
"id":"2012-07-11 23:41:50.0",
"category":[16],
"logtext":["\nHydromet Coordination Message\nOhio River
Forecast Center, Wilmington, OH\n741 PM EDT Wednesday, July 11,
2012\n\nDiscussion (00Z):\n\nMoist air across the southern basin will
keep showers and thunderstorms in\nthe forecast. The deeper moisture
will move farther north as southerly \nflow returns to the Ohio Valley.
Stuck close to HPC this evening with\nminor adjustments for radar trends
in the first 6 hr period. Also cut \nback slightly on max amounts
across the SW. \n\nDay 1 (8pm EDT Wednesday - 8pm EDT
Thursday):\nPrecip amounts up to 1.25\" are forecast across middle TN.
Amounts taper \nto around 0.10\" along the Ohio River.\n\nDay 2 (8pm
EDT Thursday - 8pm EDT Friday):\nPrecip is forecast for almost the
entire basin. The highest forecast\namounts are across western
KY...where up to 1.25\" is forecast. Amounts \ntaper to 0.00\" across
the NE.\n\nQPF from OHRFC, HPC, et al., can be seen at
weather.gov/ohrfc/Forecast.php\n$$\nFor critical after-hours support,
the OHRFC cell number is 937-725-2222.\n "],
"username":["oper"],
"_version_":1512928764890185728},
{
"id":"2012-07-12 12:45:36.0",
"category":[16],
"logtext":["\nHydromet Coordination Message\nOhio River
Forecast Center, Wilmington, OH\n845 AM EDT Thursday, July 12,
2012\n\nOHRFC QPF Discussion (12Z):\n\nA stubborn front over the
southern basin will continue to produce unsettled\nweather mainly south
of the Ohio River today and tonight. A strengthening\nBermuda ridge will
nudge the deeper moisture farther north and west Friday\nand into the
weekend. A nearly stationary surface trough will interact with\nthe
excessive moisture to generate showers and thunderstorms across a good
\nportion of the middle Ohio Valley during this time period. Due to the
high \namount of uncertainty, we are going pretty conservative with our
QPF amounts. \nWe favored a SREF/Euro blend, but made plenty of
additional tweaks to cut \nback the precip over the drier northwest part
of the basin. In general, our \nQPF amounts are about half that of HPC
throughout much of the basin. \n\nOHRFC 48-hr QPF Amounts ending 12Z
Saturday: \n\nDay 1 (8am EDT Thursday - 8am EDT Friday):\nUnsettled
weather will continue mainly along and south of the Ohio River.
\nHeaviest amounts are forecast across much of KY and middle TN with
basin\naverage amounts between 0.50\" to 1.50\". More isolated activity
is expected\nnorth of the Ohio. \n\nDay 2 (8am EDT Friday - 8am EDT
Saturday):\nThe axis of heavier precipitation will shift northward into
north-central\nKY, far southern IL, southern IN, southern OH, northern
and western WV, and\nsouthwest PA. Basin-average amounts should range
from 0.25\" to 0.75\". \n\n\nQPF from OHRFC, HPC, et al., can be seen at
weather.gov/ohrfc/Forecast.php\n$$\nFor critical after-hours support,
the OHRFC cell number is 937-725-2222.\njm "],
"username":["myers"],
"_version_":1512928764967780352},
{
"id":"2012-07-12 18:02:33.0",
"category":[16],
"logtext":["\nHydromet Coordination Message\nOhio River
Forecast Center, Wilmington, OH\n202 PM EDT Thursday, July 12,
2012\n\n18Z Update: New 18Z HPC QPF package appears to be a bit more
conservative. We \nblended our first period toward their new guidance
and made some additional \ntweaks for current radar trends/convection.
Otherwise, no changes were\nneeded at this time. \n\nPrevious QPF
Discussion (12Z):\n\nA stubborn front over the southern basin will
continue to produce unsettled\nweather mainly south of the Ohio River
today and tonight. A strengthening\nBermuda ridge will nudge the deeper
moisture farther north and west Friday\nand into the weekend. A nearly
stationary surface trough will interact with\nthe excessive moisture to
generate showers and thunderstorms across a good \nportion of the middle
Ohio Valley during this time period. Due to the high \namount of
uncertainty, we are going pretty conservative with our QPF amounts. \nWe
favored a SREF/Euro blend, but made plenty of additional tweaks to cut
\nback the precip over the drier northwest part of the basin. In
general, our \nQPF amounts are about half that of HPC throughout much of
the basin. \n\nOHRFC 48-hr QPF Amounts ending 12Z Saturday: \n\nDay 1
(8am EDT Thursday - 8am EDT Friday):\nUnsettled weather will continue
mainly along and south of the Ohio River. \nHeaviest amounts are
forecast across much of KY and middle TN with basin\naverage amounts
between 0.50\" to 1.50\". More isolated activity is expected\nnorth of
the Ohio. \n\nDay 2 (8am EDT Friday - 8am EDT Saturday):\nThe axis of
heavier precipitation will shift northward into north-central\nKY, far
southern IL, southern IN, southern OH, northern and western WV,
and\nsouthwest PA. Basin-average amounts should range from 0.25\" to
0.75\". \n\nQPF from OHRFC, HPC, et al., can be seen at
weather.gov/ohrfc/Forecast.php\n$$\nFor critical after-hours support,
the OHRFC cell number is 937-725-2222.\njm "],
"username":["myers"],
"_version_":1512928765003431936},
{
"id":"2009-07-29 13:13:20.0",
"category":[16],
"logtext":["ZCZC CRWHCMTIR CES\nTTAA00 KTIR DDHHMM\n\n...FOR
INTERGOVERNMENTAL AGENCY USE ONLY...\n\nHydromet Coordination
Message\nOhio River Forecast Center, Wilmington, OH\n913 AM EDT
Wednesday, July 29, 2009\n\nTo: OHRFC WFOs\nFrom: OHRFC\n\nA cold
front slowly working its way across the basin will result in
more\nunsettled weather for much of the Ohio Valley today. Higher QPF
totals are\nforecast across the eastern half of the region where the
deeper moisture\nresides. A sizable break in the action is anticipated
tonight before the\nnext wave moves into the western basin on Thursday.
Overall, our forecast\nis very similar to HPC's QPF except for the usual
radar based adjustments\nin the first period. \n\nOHRFC QPF Amounts
through 12Z Thursday:\nA half inch or more is forecast for most of the
eastern half of the Ohio\nValley, with amounts over an inch likely for
the higher terrain. Generally\naround a quarter of an inch or less is
forecast for the western side of the\nbasin. \n\n\n$$\njm \nNNNN"],
"username":["myers"],
"_version_":1512928766474584064},
{
"id":"2010-08-28 11:04:33.0",
"category":[16],
"logtext":["TTAA00 KTIR DDHHMM\n\n...FOR INTERGOVERNMENTAL
AGENCY USE ONLY...\n\nHydromet Coordination Message\nOhio River Forecast
Center, Wilmington, OH\n704 AM EDT Saturday, August 28, 2010\n\nSome
isolated showers are possible again over the New River basin and
in\nmiddle Tennessee this afternoon. Otherwise, the Ohio Valley will
remain dry\ntoday. On Sunday, the ridge axis will shift slightly
eastward, just enough\nto allow deeper moisture to sneak into the
southwestern Ohio Valley. \nOur QPF reflects this increased chance of
showers and thunderstorms over\nmiddle TN, western KY, eastern IL, and
far southwest IN. \n \n\n$$\nFor critical after-hours support, the OHRFC
cell number is 937-725-2222.\njm "],
"username":["myers"],
"_version_":1512928767572443136},
{
"id":"2011-05-06 12:44:03.0",
"category":[16],
"logtext":["\nHydromet Coordination Message\nOhio River
Forecast Center, Wilmington, OH\n844 AM EDT Friday, May 6, 2011\n\nTO:
OHRFC HSAs\nFROM: OHRFC \n\n***The Ohio River Forecast Center remains
on 24 hour operations***\n\nOHRFC QPF Discussion:\n\nMild-mannered
system to work its way across the Ohio Valley today with\nweak cold
front currently pushing into western Ohio and middle Kentucky.\nThere
will be a brief break after this system before the Saturday \nsystem
moves into the western portion of the basin. This second
system\nfeatures slightly deeper moisture, but still rainfall amounts
should be\nmodest. Utilized HPC's guidance for the QPF today, with the
exception\nof adjustments to the first period to account current
rainfall trends.\n\nOHRFC QPF Forecast through 8am EDT Sunday:\n\nDay 1:
8am EDT Friday through 8am EDT Saturday:\nRain is forecast for most of
the Ohio Valley. Amounts will generally be\n0.10 inch or less.\n\nDay
2: 8am EDT Saturday through 8am EDT Sunday:\nRain is expected
basinwide. Most areas will see amounts in the 0.25 \nto 0.50 inch range
with amounts up to 0.75 inch in southeast Kentucky.\n\nQPF from OHRFC,
HPC, et al., can be seen at weather.gov/ohrfc/Forecast.php\n$$\nFor
critical after-hours support, the OHRFC cell number is
937-725-2222.\nLink Crawford "],
"username":["crawford"],
"_version_":1512928767842975744}]
}}
Re: query parsing
Posted by Alessandro Benedetti <be...@gmail.com>.
happy to read that, regarding the spellcheck, is a different thing, so let
us know for further details !
Cheers
2015-09-27 18:59 GMT+01:00 Mark Fenbers <ma...@noaa.gov>:
> I am delighted to announce that I have it all working again! Well, not
> all, just the searching!
>
> I deleted my core and created a new one from the command-line (solr
> create_core -c EventLog2) using the basic_configs option. Then I had to add
> my columns to the schema.xml and the dataimport handler to solrconfig.xml
> and tweak a couple of other details. But to make a long story short,
> parsing is working and I can search on terms without wrapping asterisks!!
> Yay! Thanks for the help!
>
> Spell-checking still isn't working, though, and I'm apprehensive about
> working with it today. But I will eventually. The complaint is it can't
> find ELspell, which I had defined in the old setup that I blew away, so
> I'll have to redefine it at some point! For now, I'm just gonna delight in
> having searching working again!
>
> Mark
>
>
> On 9/26/2015 11:05 PM, Erick Erickson wrote:
>
>> No need to re-install Solr, just create a new core, this time it'd
>> probably be
>> easiest to use the bin/solr create_core command. In the Solr
>> directory just type bin/solr create_core -help to see the options.
>>
>> We're pretty much trying to migrate to using bin/solr for all the
>> maintenance
>> we can, but as always the documentation lags the code.
>>
>> Yeah, things are a bit ragged. The admin UI/core UI is really a legacy
>> bit of code that has _always_ been confusing, I'm hoping we can pretty
>> much remove it at some point since it's as trappy as it is.
>>
>> Best,
>> Erick
>>
>> On Sat, Sep 26, 2015 at 12:49 PM, Mark Fenbers <ma...@noaa.gov>
>> wrote:
>>
>>> OK, a lot of dialog while I was gone for two days! I read the whole
>>> thread,
>>> but I'm a newbie to Solr, so some of the dialog was Greek to me. I
>>> understand the words, of course, but applying it so I know exactly what
>>> to
>>> do without screwing something else up is the problem. After all, that is
>>> how I got into the mess in the first place. I'm glad I have good help to
>>> untangle the knots I've made!
>>>
>>> I'd like to start over (option 1 below), but does this mean delete all my
>>> config and reinstalling Solr?? Maybe that is not a bad idea, but I will
>>> at
>>> least save off my data-config.xml as that is clearly the one thing that
>>> is
>>> probably working right. However, I did do quite a bit of editing that I
>>> would have to do again. Please advise...
>>>
>>> To be fair, I must answer Erick's question of how I created the data
>>> index
>>> in the first place, because this might be relevant...
>>>
>>> The bulk of the data is read from 9000+ text files, where each file was
>>> manually typed. Before inserting into the database, I do a little bit of
>>> processing of the text using "sed" to delete the top few and bottom few
>>> lines, and to substitute each single-quote character with a pair of
>>> single-quotes (so PostgreSQL doesn't choke). Line-feed characters are
>>> preserved as ASCII 10 (hex 0A), but there shouldn't be (and I am not
>>> aware
>>> of) any characters aside from what is on the keyboard.
>>>
>>> Next, I insert it with this command:
>>> psql -U awips -d OHRFC -c "INSERT INTO EventLogText VALUES('$postDate',
>>> '$user', '$postDate', '$entryText', '$postCatVal');"
>>>
>>> In case you are wondering about my table, it is defined in this way:
>>> CREATE TABLE eventlogtext (
>>> posttime timestamp without time zone NOT NULL, -- Timestamp of this
>>> entry's original posting
>>> username character varying(8), -- username (logname) of the original
>>> poster
>>> lastmodtime timestamp without time zone, -- Last time record was
>>> altered
>>> logtext text, -- text of the log entry
>>> category integer, -- bit-wise category value
>>> CONSTRAINT eventlogtext_pkey PRIMARY KEY (posttime)
>>> )
>>>
>>> To do the indexing, I merely use /dataimport?full-import, but it knows
>>> what
>>> to do from my data-config.xml; which is here:
>>>
>>> <dataConfig>
>>> <dataSource driver="org.postgresql.Driver"
>>> url="jdbc:postgresql://dx1f/OHRFC" user="awips" />
>>> <document>
>>> <entity name="eventlogtext" query="SELECT posttime AS id,
>>> username,
>>> logtext, category FROM eventlogtext;"
>>> deltaQuery="SELECT posttime AS id FROM eventlogtext
>>> WHERE
>>> lastmodtime > '${dataimporter.last_index_time}';">
>>> <entity name="categorytypes" query="SELECT catname FROM
>>> categorytypes WHERE catid='${eventlogtext.category}';">
>>> </entity>
>>> </entity>
>>> </document>
>>> </dataConfig>
>>>
>>> Hope this helps!
>>>
>>> Thanks,
>>> Mark
>>>
>>> On 9/24/2015 10:57 AM, Erick Erickson wrote:
>>>
>>>> Geraint:
>>>>
>>>> Good Catch! I totally missed that. So all of our focus on schema.xml has
>>>> been... totally irrelevant. Now that you pointed that out, there's also
>>>> the
>>>> addition: add-unknown-fields-to-the-schema, which indicates you started
>>>> this up in "schemaless" mode.
>>>>
>>>> In short, solr is trying to guess what your field types should be and
>>>> guessing wrong (again and again and again). This is the classic weakness
>>>> of
>>>> schemaless. It's great for indexing stuff fast, but if it guesses wrong
>>>> you're stuck.
>>>>
>>>>
>>>> So to the original problem: I'd start over and either
>>>> 1> use the regular setup, not schemaless
>>>> or
>>>> 2> use the _managed_ schema API to explicitly add fields and fieldTypes
>>>> to
>>>> the managed schema
>>>>
>>>> Best,
>>>> Erick
>>>>
>>>> On Thu, Sep 24, 2015 at 2:02 AM, Duck Geraint (ext) GBJH <
>>>> Geraint.Duck@syngenta.com> wrote:
>>>>
>>>> Okay, so maybe I'm missing something here (I'm still relatively new to
>>>>> Solr myself), but am I right in thinking the following is still in your
>>>>> solrconfig.xml file:
>>>>>
>>>>> <schemaFactory class="ManagedIndexSchemaFactory">
>>>>> <bool name="mutable">true</bool>
>>>>> <str name="managedSchemaResourceName">managed-schema</str>
>>>>> </schemaFactory>
>>>>>
>>>>> If so, wouldn't using a managed schema make several of your field
>>>>> definitions inside the schema.xml file semi-redundant?
>>>>>
>>>>> Regards,
>>>>> Geraint
>>>>>
>>>>>
>>>>> Geraint Duck
>>>>> Data Scientist
>>>>> Toxicology and Health Sciences
>>>>> Syngenta UK
>>>>> Email: geraint.duck@syngenta.com
>>>>>
>>>>>
>>>>> -----Original Message-----
>>>>> From: Alessandro Benedetti [mailto:benedetti.alex85@gmail.com]
>>>>> Sent: 24 September 2015 09:23
>>>>> To: solr-user@lucene.apache.org
>>>>> Subject: Re: query parsing
>>>>>
>>>>> I would focus on this :
>>>>>
>>>>> "
>>>>>
>>>>> 5> now kick off the DIH job and look again.
>>>>>>
>>>>>> Now it shows a histogram, but most of the "terms" are long -- the full
>>>>> texts of (the table.column) eventlogtext.logtext, including the
>>>>> whitespace
>>>>> (with %0A used for newline characters)... So, it appears it is not
>>>>> being
>>>>> tokenized properly, correct?"
>>>>> Can you open from your Solr ui , the schema xml and show us the
>>>>> snippets
>>>>> for that field that seems to not tokenise ?
>>>>> Can you show us ( even a screenshot is fine) the schema browser page
>>>>> related ?
>>>>> Could be a problem of encoding ?
>>>>> Following Erick details about the analysis, what are your results ?
>>>>>
>>>>> Cheers
>>>>>
>>>>> 2015-09-24 8:04 GMT+01:00 Upayavira <uv...@odoko.co.uk>:
>>>>>
>>>>> typically, the index dir is inside the data dir. Delete the index dir
>>>>>> and you should be good. If there is a tlog next to it, you might want
>>>>>> to delete that also.
>>>>>>
>>>>>> If you dont have a data dir, i wonder whether you set the data dir
>>>>>> when creating your core or collection. Typically the instance dir and
>>>>>> data dir aren't needed.
>>>>>>
>>>>>> Upayavira
>>>>>>
>>>>>> On Wed, Sep 23, 2015, at 10:46 PM, Erick Erickson wrote:
>>>>>>
>>>>>>> OK, this is bizarre. You'd have had to set up SolrCloud by
>>>>>>> specifying the -zkRun command when you start Solr or the -zkHost;
>>>>>>> highly unlikely. On the admin page there would be a "cloud" link on
>>>>>>> the left side, I really doubt one's there.
>>>>>>>
>>>>>>> You should have a data directory, it should be the parent of the
>>>>>>> index and tlog directories. As of sanity check try looking at the
>>>>>>> analysis page.
>>>>>>> Type
>>>>>>> a bunch of words in the left hand side indexing box and uncheck the
>>>>>>> verbose box. As you can tell I'm grasping at straws. I'm still
>>>>>>> puzzled why you don't have a "data" directory here, but that
>>>>>>> shouldn't really matter. How did you create this index? I don't mean
>>>>>>> data import handler more how did you create the core that you're
>>>>>>> indexing to?
>>>>>>>
>>>>>>> Best,
>>>>>>> Erick
>>>>>>>
>>>>>>> On Wed, Sep 23, 2015 at 10:16 AM, Mark Fenbers
>>>>>>> <ma...@noaa.gov>
>>>>>>> wrote:
>>>>>>>
>>>>>>> On 9/23/2015 12:30 PM, Erick Erickson wrote:
>>>>>>>>
>>>>>>>> Then my next guess is you're not pointing at the index you think
>>>>>>>>> you
>>>>>>>>>
>>>>>>>> are
>>>>>>
>>>>>>> when you 'rm -rf data'
>>>>>>>>>
>>>>>>>>> Just ignore the Elall field for now I should think, although get
>>>>>>>>> rid
>>>>>>>>>
>>>>>>>> of it
>>>>>>
>>>>>>> if you don't think you need it.
>>>>>>>>>
>>>>>>>>> DIH should be irrelevant here.
>>>>>>>>>
>>>>>>>>> So let's back up.
>>>>>>>>> 1> go ahead and "rm -fr data" (with Solr stopped).
>>>>>>>>>
>>>>>>>>> I have no "data" dir. Did you mean "index" dir? I removed 3
>>>>>>>> index directories (2 for spelling):
>>>>>>>> cd /localapps/dev/eventLog; rm -rfv index solr/spFile solr/spIndex
>>>>>>>>
>>>>>>>> 2> start Solr
>>>>>>>>> 3> do NOT re-index.
>>>>>>>>> 4> look at your index via the schema-browser. Of course there
>>>>>>>>> 4> should
>>>>>>>>>
>>>>>>>> be
>>>>>>
>>>>>>> nothing there!
>>>>>>>>>
>>>>>>>>> Correct! It said "there is no term info :("
>>>>>>>>
>>>>>>>> 5> now kick off the DIH job and look again.
>>>>>>>>>
>>>>>>>>> Now it shows a histogram, but most of the "terms" are long -- the
>>>>>>>> full texts of (the table.column) eventlogtext.logtext, including
>>>>>>>> the
>>>>>>>>
>>>>>>> whitespace
>>>>>>
>>>>>>> (with %0A used for newline characters)... So, it appears it is
>>>>>>>> not
>>>>>>>>
>>>>>>> being
>>>>>>
>>>>>>> tokenized properly, correct?
>>>>>>>>
>>>>>>>> Your logtext field should have only single tokens. The fact that
>>>>>>>>> you
>>>>>>>>>
>>>>>>>> have
>>>>>>
>>>>>>> some very
>>>>>>>>> long tokens presumably with whitespace) indicates that you aren't
>>>>>>>>>
>>>>>>>> really
>>>>>>
>>>>>>> blowing
>>>>>>>>> the index away between indexing.
>>>>>>>>>
>>>>>>>>> Well, I did this time for sure. I verified that initially,
>>>>>>>> because it showed there was no term info until I DIH'd again.
>>>>>>>>
>>>>>>>> Are you perhaps in Solr Cloud with more than one replica?
>>>>>>>>>
>>>>>>>>> Not that I know of, but being new to Solr, there could be things
>>>>>>>> going
>>>>>>>>
>>>>>>> on
>>>>>>
>>>>>>> that I'm not aware of. How can I tell? I certainly didn't set
>>>>>>>>
>>>>>>> anything up
>>>>>>
>>>>>>> for solrCloud deliberately.
>>>>>>>>
>>>>>>>> In that case you
>>>>>>>>> might be getting the index replicated on startup assuming you
>>>>>>>>> didn't blow away all replicas. If you are in SolrCloud, I'd just
>>>>>>>>> delete the collection and start over, after insuring that you'd
>>>>>>>>> pushed the configset up to Zookeeper.
>>>>>>>>>
>>>>>>>>> BTW, I always look at the schema.xml file from the Solr admin
>>>>>>>>> window
>>>>>>>>>
>>>>>>>> just
>>>>>>
>>>>>>> as
>>>>>>>>> a sanity check in these situations.
>>>>>>>>>
>>>>>>>>> Good idea! But the one shown in the browser is identical to the
>>>>>>>> one
>>>>>>>>
>>>>>>> I've
>>>>>>
>>>>>>> been editing! So that's not an issue.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>> --
>>>>> --------------------------
>>>>>
>>>>> Benedetti Alessandro
>>>>> Visiting card - http://about.me/alessandro_benedetti
>>>>> Blog - http://alexbenedetti.blogspot.co.uk
>>>>>
>>>>> "Tyger, tyger burning bright
>>>>> In the forests of the night,
>>>>> What immortal hand or eye
>>>>> Could frame thy fearful symmetry?"
>>>>>
>>>>> William Blake - Songs of Experience -1794 England
>>>>> ________________________________
>>>>>
>>>>>
>>>>> Syngenta Limited, Registered in England No 2710846;Registered Office :
>>>>> Syngenta Limited, European Regional Centre, Priestley Road, Surrey
>>>>> Research
>>>>> Park, Guildford, Surrey, GU2 7YH, United Kingdom
>>>>> ________________________________
>>>>> This message may contain confidential information. If you are not
>>>>> the
>>>>> designated recipient, please notify the sender immediately, and delete
>>>>> the
>>>>> original and any copies. Any use of the message by you is prohibited.
>>>>>
>>>>>
>
--
--------------------------
Benedetti Alessandro
Visiting card - http://about.me/alessandro_benedetti
Blog - http://alexbenedetti.blogspot.co.uk
"Tyger, tyger burning bright
In the forests of the night,
What immortal hand or eye
Could frame thy fearful symmetry?"
William Blake - Songs of Experience -1794 England
Re: query parsing
Posted by Mark Fenbers <ma...@noaa.gov>.
I am delighted to announce that I have it all working again! Well, not
all, just the searching!
I deleted my core and created a new one from the command-line (solr
create_core -c EventLog2) using the basic_configs option. Then I had to
add my columns to the schema.xml and the dataimport handler to
solrconfig.xml and tweak a couple of other details. But to make a long
story short, parsing is working and I can search on terms without
wrapping asterisks!! Yay! Thanks for the help!
Spell-checking still isn't working, though, and I'm apprehensive about
working with it today. But I will eventually. The complaint is it
can't find ELspell, which I had defined in the old setup that I blew
away, so I'll have to redefine it at some point! For now, I'm just
gonna delight in having searching working again!
Mark
On 9/26/2015 11:05 PM, Erick Erickson wrote:
> No need to re-install Solr, just create a new core, this time it'd probably be
> easiest to use the bin/solr create_core command. In the Solr
> directory just type bin/solr create_core -help to see the options.
>
> We're pretty much trying to migrate to using bin/solr for all the maintenance
> we can, but as always the documentation lags the code.
>
> Yeah, things are a bit ragged. The admin UI/core UI is really a legacy
> bit of code that has _always_ been confusing, I'm hoping we can pretty
> much remove it at some point since it's as trappy as it is.
>
> Best,
> Erick
>
> On Sat, Sep 26, 2015 at 12:49 PM, Mark Fenbers <ma...@noaa.gov> wrote:
>> OK, a lot of dialog while I was gone for two days! I read the whole thread,
>> but I'm a newbie to Solr, so some of the dialog was Greek to me. I
>> understand the words, of course, but applying it so I know exactly what to
>> do without screwing something else up is the problem. After all, that is
>> how I got into the mess in the first place. I'm glad I have good help to
>> untangle the knots I've made!
>>
>> I'd like to start over (option 1 below), but does this mean delete all my
>> config and reinstalling Solr?? Maybe that is not a bad idea, but I will at
>> least save off my data-config.xml as that is clearly the one thing that is
>> probably working right. However, I did do quite a bit of editing that I
>> would have to do again. Please advise...
>>
>> To be fair, I must answer Erick's question of how I created the data index
>> in the first place, because this might be relevant...
>>
>> The bulk of the data is read from 9000+ text files, where each file was
>> manually typed. Before inserting into the database, I do a little bit of
>> processing of the text using "sed" to delete the top few and bottom few
>> lines, and to substitute each single-quote character with a pair of
>> single-quotes (so PostgreSQL doesn't choke). Line-feed characters are
>> preserved as ASCII 10 (hex 0A), but there shouldn't be (and I am not aware
>> of) any characters aside from what is on the keyboard.
>>
>> Next, I insert it with this command:
>> psql -U awips -d OHRFC -c "INSERT INTO EventLogText VALUES('$postDate',
>> '$user', '$postDate', '$entryText', '$postCatVal');"
>>
>> In case you are wondering about my table, it is defined in this way:
>> CREATE TABLE eventlogtext (
>> posttime timestamp without time zone NOT NULL, -- Timestamp of this
>> entry's original posting
>> username character varying(8), -- username (logname) of the original
>> poster
>> lastmodtime timestamp without time zone, -- Last time record was altered
>> logtext text, -- text of the log entry
>> category integer, -- bit-wise category value
>> CONSTRAINT eventlogtext_pkey PRIMARY KEY (posttime)
>> )
>>
>> To do the indexing, I merely use /dataimport?full-import, but it knows what
>> to do from my data-config.xml; which is here:
>>
>> <dataConfig>
>> <dataSource driver="org.postgresql.Driver"
>> url="jdbc:postgresql://dx1f/OHRFC" user="awips" />
>> <document>
>> <entity name="eventlogtext" query="SELECT posttime AS id, username,
>> logtext, category FROM eventlogtext;"
>> deltaQuery="SELECT posttime AS id FROM eventlogtext WHERE
>> lastmodtime > '${dataimporter.last_index_time}';">
>> <entity name="categorytypes" query="SELECT catname FROM
>> categorytypes WHERE catid='${eventlogtext.category}';">
>> </entity>
>> </entity>
>> </document>
>> </dataConfig>
>>
>> Hope this helps!
>>
>> Thanks,
>> Mark
>>
>> On 9/24/2015 10:57 AM, Erick Erickson wrote:
>>> Geraint:
>>>
>>> Good Catch! I totally missed that. So all of our focus on schema.xml has
>>> been... totally irrelevant. Now that you pointed that out, there's also
>>> the
>>> addition: add-unknown-fields-to-the-schema, which indicates you started
>>> this up in "schemaless" mode.
>>>
>>> In short, solr is trying to guess what your field types should be and
>>> guessing wrong (again and again and again). This is the classic weakness
>>> of
>>> schemaless. It's great for indexing stuff fast, but if it guesses wrong
>>> you're stuck.
>>>
>>>
>>> So to the original problem: I'd start over and either
>>> 1> use the regular setup, not schemaless
>>> or
>>> 2> use the _managed_ schema API to explicitly add fields and fieldTypes to
>>> the managed schema
>>>
>>> Best,
>>> Erick
>>>
>>> On Thu, Sep 24, 2015 at 2:02 AM, Duck Geraint (ext) GBJH <
>>> Geraint.Duck@syngenta.com> wrote:
>>>
>>>> Okay, so maybe I'm missing something here (I'm still relatively new to
>>>> Solr myself), but am I right in thinking the following is still in your
>>>> solrconfig.xml file:
>>>>
>>>> <schemaFactory class="ManagedIndexSchemaFactory">
>>>> <bool name="mutable">true</bool>
>>>> <str name="managedSchemaResourceName">managed-schema</str>
>>>> </schemaFactory>
>>>>
>>>> If so, wouldn't using a managed schema make several of your field
>>>> definitions inside the schema.xml file semi-redundant?
>>>>
>>>> Regards,
>>>> Geraint
>>>>
>>>>
>>>> Geraint Duck
>>>> Data Scientist
>>>> Toxicology and Health Sciences
>>>> Syngenta UK
>>>> Email: geraint.duck@syngenta.com
>>>>
>>>>
>>>> -----Original Message-----
>>>> From: Alessandro Benedetti [mailto:benedetti.alex85@gmail.com]
>>>> Sent: 24 September 2015 09:23
>>>> To: solr-user@lucene.apache.org
>>>> Subject: Re: query parsing
>>>>
>>>> I would focus on this :
>>>>
>>>> "
>>>>
>>>>> 5> now kick off the DIH job and look again.
>>>>>
>>>> Now it shows a histogram, but most of the "terms" are long -- the full
>>>> texts of (the table.column) eventlogtext.logtext, including the
>>>> whitespace
>>>> (with %0A used for newline characters)... So, it appears it is not being
>>>> tokenized properly, correct?"
>>>> Can you open from your Solr ui , the schema xml and show us the snippets
>>>> for that field that seems to not tokenise ?
>>>> Can you show us ( even a screenshot is fine) the schema browser page
>>>> related ?
>>>> Could be a problem of encoding ?
>>>> Following Erick details about the analysis, what are your results ?
>>>>
>>>> Cheers
>>>>
>>>> 2015-09-24 8:04 GMT+01:00 Upayavira <uv...@odoko.co.uk>:
>>>>
>>>>> typically, the index dir is inside the data dir. Delete the index dir
>>>>> and you should be good. If there is a tlog next to it, you might want
>>>>> to delete that also.
>>>>>
>>>>> If you dont have a data dir, i wonder whether you set the data dir
>>>>> when creating your core or collection. Typically the instance dir and
>>>>> data dir aren't needed.
>>>>>
>>>>> Upayavira
>>>>>
>>>>> On Wed, Sep 23, 2015, at 10:46 PM, Erick Erickson wrote:
>>>>>> OK, this is bizarre. You'd have had to set up SolrCloud by
>>>>>> specifying the -zkRun command when you start Solr or the -zkHost;
>>>>>> highly unlikely. On the admin page there would be a "cloud" link on
>>>>>> the left side, I really doubt one's there.
>>>>>>
>>>>>> You should have a data directory, it should be the parent of the
>>>>>> index and tlog directories. As of sanity check try looking at the
>>>>>> analysis page.
>>>>>> Type
>>>>>> a bunch of words in the left hand side indexing box and uncheck the
>>>>>> verbose box. As you can tell I'm grasping at straws. I'm still
>>>>>> puzzled why you don't have a "data" directory here, but that
>>>>>> shouldn't really matter. How did you create this index? I don't mean
>>>>>> data import handler more how did you create the core that you're
>>>>>> indexing to?
>>>>>>
>>>>>> Best,
>>>>>> Erick
>>>>>>
>>>>>> On Wed, Sep 23, 2015 at 10:16 AM, Mark Fenbers
>>>>>> <ma...@noaa.gov>
>>>>>> wrote:
>>>>>>
>>>>>>> On 9/23/2015 12:30 PM, Erick Erickson wrote:
>>>>>>>
>>>>>>>> Then my next guess is you're not pointing at the index you think
>>>>>>>> you
>>>>> are
>>>>>>>> when you 'rm -rf data'
>>>>>>>>
>>>>>>>> Just ignore the Elall field for now I should think, although get
>>>>>>>> rid
>>>>> of it
>>>>>>>> if you don't think you need it.
>>>>>>>>
>>>>>>>> DIH should be irrelevant here.
>>>>>>>>
>>>>>>>> So let's back up.
>>>>>>>> 1> go ahead and "rm -fr data" (with Solr stopped).
>>>>>>>>
>>>>>>> I have no "data" dir. Did you mean "index" dir? I removed 3
>>>>>>> index directories (2 for spelling):
>>>>>>> cd /localapps/dev/eventLog; rm -rfv index solr/spFile solr/spIndex
>>>>>>>
>>>>>>>> 2> start Solr
>>>>>>>> 3> do NOT re-index.
>>>>>>>> 4> look at your index via the schema-browser. Of course there
>>>>>>>> 4> should
>>>>> be
>>>>>>>> nothing there!
>>>>>>>>
>>>>>>> Correct! It said "there is no term info :("
>>>>>>>
>>>>>>>> 5> now kick off the DIH job and look again.
>>>>>>>>
>>>>>>> Now it shows a histogram, but most of the "terms" are long -- the
>>>>>>> full texts of (the table.column) eventlogtext.logtext, including
>>>>>>> the
>>>>> whitespace
>>>>>>> (with %0A used for newline characters)... So, it appears it is
>>>>>>> not
>>>>> being
>>>>>>> tokenized properly, correct?
>>>>>>>
>>>>>>>> Your logtext field should have only single tokens. The fact that
>>>>>>>> you
>>>>> have
>>>>>>>> some very
>>>>>>>> long tokens presumably with whitespace) indicates that you aren't
>>>>> really
>>>>>>>> blowing
>>>>>>>> the index away between indexing.
>>>>>>>>
>>>>>>> Well, I did this time for sure. I verified that initially,
>>>>>>> because it showed there was no term info until I DIH'd again.
>>>>>>>
>>>>>>>> Are you perhaps in Solr Cloud with more than one replica?
>>>>>>>>
>>>>>>> Not that I know of, but being new to Solr, there could be things
>>>>>>> going
>>>>> on
>>>>>>> that I'm not aware of. How can I tell? I certainly didn't set
>>>>> anything up
>>>>>>> for solrCloud deliberately.
>>>>>>>
>>>>>>>> In that case you
>>>>>>>> might be getting the index replicated on startup assuming you
>>>>>>>> didn't blow away all replicas. If you are in SolrCloud, I'd just
>>>>>>>> delete the collection and start over, after insuring that you'd
>>>>>>>> pushed the configset up to Zookeeper.
>>>>>>>>
>>>>>>>> BTW, I always look at the schema.xml file from the Solr admin
>>>>>>>> window
>>>>> just
>>>>>>>> as
>>>>>>>> a sanity check in these situations.
>>>>>>>>
>>>>>>> Good idea! But the one shown in the browser is identical to the
>>>>>>> one
>>>>> I've
>>>>>>> been editing! So that's not an issue.
>>>>>>>
>>>>>>>
>>>>
>>>> --
>>>> --------------------------
>>>>
>>>> Benedetti Alessandro
>>>> Visiting card - http://about.me/alessandro_benedetti
>>>> Blog - http://alexbenedetti.blogspot.co.uk
>>>>
>>>> "Tyger, tyger burning bright
>>>> In the forests of the night,
>>>> What immortal hand or eye
>>>> Could frame thy fearful symmetry?"
>>>>
>>>> William Blake - Songs of Experience -1794 England
>>>> ________________________________
>>>>
>>>>
>>>> Syngenta Limited, Registered in England No 2710846;Registered Office :
>>>> Syngenta Limited, European Regional Centre, Priestley Road, Surrey
>>>> Research
>>>> Park, Guildford, Surrey, GU2 7YH, United Kingdom
>>>> ________________________________
>>>> This message may contain confidential information. If you are not the
>>>> designated recipient, please notify the sender immediately, and delete
>>>> the
>>>> original and any copies. Any use of the message by you is prohibited.
>>>>
Re: query parsing
Posted by Erick Erickson <er...@gmail.com>.
No need to re-install Solr, just create a new core, this time it'd probably be
easiest to use the bin/solr create_core command. In the Solr
directory just type bin/solr create_core -help to see the options.
We're pretty much trying to migrate to using bin/solr for all the maintenance
we can, but as always the documentation lags the code.
Yeah, things are a bit ragged. The admin UI/core UI is really a legacy
bit of code that has _always_ been confusing, I'm hoping we can pretty
much remove it at some point since it's as trappy as it is.
Best,
Erick
On Sat, Sep 26, 2015 at 12:49 PM, Mark Fenbers <ma...@noaa.gov> wrote:
> OK, a lot of dialog while I was gone for two days! I read the whole thread,
> but I'm a newbie to Solr, so some of the dialog was Greek to me. I
> understand the words, of course, but applying it so I know exactly what to
> do without screwing something else up is the problem. After all, that is
> how I got into the mess in the first place. I'm glad I have good help to
> untangle the knots I've made!
>
> I'd like to start over (option 1 below), but does this mean delete all my
> config and reinstalling Solr?? Maybe that is not a bad idea, but I will at
> least save off my data-config.xml as that is clearly the one thing that is
> probably working right. However, I did do quite a bit of editing that I
> would have to do again. Please advise...
>
> To be fair, I must answer Erick's question of how I created the data index
> in the first place, because this might be relevant...
>
> The bulk of the data is read from 9000+ text files, where each file was
> manually typed. Before inserting into the database, I do a little bit of
> processing of the text using "sed" to delete the top few and bottom few
> lines, and to substitute each single-quote character with a pair of
> single-quotes (so PostgreSQL doesn't choke). Line-feed characters are
> preserved as ASCII 10 (hex 0A), but there shouldn't be (and I am not aware
> of) any characters aside from what is on the keyboard.
>
> Next, I insert it with this command:
> psql -U awips -d OHRFC -c "INSERT INTO EventLogText VALUES('$postDate',
> '$user', '$postDate', '$entryText', '$postCatVal');"
>
> In case you are wondering about my table, it is defined in this way:
> CREATE TABLE eventlogtext (
> posttime timestamp without time zone NOT NULL, -- Timestamp of this
> entry's original posting
> username character varying(8), -- username (logname) of the original
> poster
> lastmodtime timestamp without time zone, -- Last time record was altered
> logtext text, -- text of the log entry
> category integer, -- bit-wise category value
> CONSTRAINT eventlogtext_pkey PRIMARY KEY (posttime)
> )
>
> To do the indexing, I merely use /dataimport?full-import, but it knows what
> to do from my data-config.xml; which is here:
>
> <dataConfig>
> <dataSource driver="org.postgresql.Driver"
> url="jdbc:postgresql://dx1f/OHRFC" user="awips" />
> <document>
> <entity name="eventlogtext" query="SELECT posttime AS id, username,
> logtext, category FROM eventlogtext;"
> deltaQuery="SELECT posttime AS id FROM eventlogtext WHERE
> lastmodtime > '${dataimporter.last_index_time}';">
> <entity name="categorytypes" query="SELECT catname FROM
> categorytypes WHERE catid='${eventlogtext.category}';">
> </entity>
> </entity>
> </document>
> </dataConfig>
>
> Hope this helps!
>
> Thanks,
> Mark
>
> On 9/24/2015 10:57 AM, Erick Erickson wrote:
>>
>> Geraint:
>>
>> Good Catch! I totally missed that. So all of our focus on schema.xml has
>> been... totally irrelevant. Now that you pointed that out, there's also
>> the
>> addition: add-unknown-fields-to-the-schema, which indicates you started
>> this up in "schemaless" mode.
>>
>> In short, solr is trying to guess what your field types should be and
>> guessing wrong (again and again and again). This is the classic weakness
>> of
>> schemaless. It's great for indexing stuff fast, but if it guesses wrong
>> you're stuck.
>>
>>
>> So to the original problem: I'd start over and either
>> 1> use the regular setup, not schemaless
>> or
>> 2> use the _managed_ schema API to explicitly add fields and fieldTypes to
>> the managed schema
>>
>> Best,
>> Erick
>>
>> On Thu, Sep 24, 2015 at 2:02 AM, Duck Geraint (ext) GBJH <
>> Geraint.Duck@syngenta.com> wrote:
>>
>>> Okay, so maybe I'm missing something here (I'm still relatively new to
>>> Solr myself), but am I right in thinking the following is still in your
>>> solrconfig.xml file:
>>>
>>> <schemaFactory class="ManagedIndexSchemaFactory">
>>> <bool name="mutable">true</bool>
>>> <str name="managedSchemaResourceName">managed-schema</str>
>>> </schemaFactory>
>>>
>>> If so, wouldn't using a managed schema make several of your field
>>> definitions inside the schema.xml file semi-redundant?
>>>
>>> Regards,
>>> Geraint
>>>
>>>
>>> Geraint Duck
>>> Data Scientist
>>> Toxicology and Health Sciences
>>> Syngenta UK
>>> Email: geraint.duck@syngenta.com
>>>
>>>
>>> -----Original Message-----
>>> From: Alessandro Benedetti [mailto:benedetti.alex85@gmail.com]
>>> Sent: 24 September 2015 09:23
>>> To: solr-user@lucene.apache.org
>>> Subject: Re: query parsing
>>>
>>> I would focus on this :
>>>
>>> "
>>>
>>>> 5> now kick off the DIH job and look again.
>>>>
>>> Now it shows a histogram, but most of the "terms" are long -- the full
>>> texts of (the table.column) eventlogtext.logtext, including the
>>> whitespace
>>> (with %0A used for newline characters)... So, it appears it is not being
>>> tokenized properly, correct?"
>>> Can you open from your Solr ui , the schema xml and show us the snippets
>>> for that field that seems to not tokenise ?
>>> Can you show us ( even a screenshot is fine) the schema browser page
>>> related ?
>>> Could be a problem of encoding ?
>>> Following Erick details about the analysis, what are your results ?
>>>
>>> Cheers
>>>
>>> 2015-09-24 8:04 GMT+01:00 Upayavira <uv...@odoko.co.uk>:
>>>
>>>> typically, the index dir is inside the data dir. Delete the index dir
>>>> and you should be good. If there is a tlog next to it, you might want
>>>> to delete that also.
>>>>
>>>> If you dont have a data dir, i wonder whether you set the data dir
>>>> when creating your core or collection. Typically the instance dir and
>>>> data dir aren't needed.
>>>>
>>>> Upayavira
>>>>
>>>> On Wed, Sep 23, 2015, at 10:46 PM, Erick Erickson wrote:
>>>>>
>>>>> OK, this is bizarre. You'd have had to set up SolrCloud by
>>>>> specifying the -zkRun command when you start Solr or the -zkHost;
>>>>> highly unlikely. On the admin page there would be a "cloud" link on
>>>>> the left side, I really doubt one's there.
>>>>>
>>>>> You should have a data directory, it should be the parent of the
>>>>> index and tlog directories. As of sanity check try looking at the
>>>>> analysis page.
>>>>> Type
>>>>> a bunch of words in the left hand side indexing box and uncheck the
>>>>> verbose box. As you can tell I'm grasping at straws. I'm still
>>>>> puzzled why you don't have a "data" directory here, but that
>>>>> shouldn't really matter. How did you create this index? I don't mean
>>>>> data import handler more how did you create the core that you're
>>>>> indexing to?
>>>>>
>>>>> Best,
>>>>> Erick
>>>>>
>>>>> On Wed, Sep 23, 2015 at 10:16 AM, Mark Fenbers
>>>>> <ma...@noaa.gov>
>>>>> wrote:
>>>>>
>>>>>> On 9/23/2015 12:30 PM, Erick Erickson wrote:
>>>>>>
>>>>>>> Then my next guess is you're not pointing at the index you think
>>>>>>> you
>>>>
>>>> are
>>>>>>>
>>>>>>> when you 'rm -rf data'
>>>>>>>
>>>>>>> Just ignore the Elall field for now I should think, although get
>>>>>>> rid
>>>>
>>>> of it
>>>>>>>
>>>>>>> if you don't think you need it.
>>>>>>>
>>>>>>> DIH should be irrelevant here.
>>>>>>>
>>>>>>> So let's back up.
>>>>>>> 1> go ahead and "rm -fr data" (with Solr stopped).
>>>>>>>
>>>>>> I have no "data" dir. Did you mean "index" dir? I removed 3
>>>>>> index directories (2 for spelling):
>>>>>> cd /localapps/dev/eventLog; rm -rfv index solr/spFile solr/spIndex
>>>>>>
>>>>>>> 2> start Solr
>>>>>>> 3> do NOT re-index.
>>>>>>> 4> look at your index via the schema-browser. Of course there
>>>>>>> 4> should
>>>>
>>>> be
>>>>>>>
>>>>>>> nothing there!
>>>>>>>
>>>>>> Correct! It said "there is no term info :("
>>>>>>
>>>>>>> 5> now kick off the DIH job and look again.
>>>>>>>
>>>>>> Now it shows a histogram, but most of the "terms" are long -- the
>>>>>> full texts of (the table.column) eventlogtext.logtext, including
>>>>>> the
>>>>
>>>> whitespace
>>>>>>
>>>>>> (with %0A used for newline characters)... So, it appears it is
>>>>>> not
>>>>
>>>> being
>>>>>>
>>>>>> tokenized properly, correct?
>>>>>>
>>>>>>> Your logtext field should have only single tokens. The fact that
>>>>>>> you
>>>>
>>>> have
>>>>>>>
>>>>>>> some very
>>>>>>> long tokens presumably with whitespace) indicates that you aren't
>>>>
>>>> really
>>>>>>>
>>>>>>> blowing
>>>>>>> the index away between indexing.
>>>>>>>
>>>>>> Well, I did this time for sure. I verified that initially,
>>>>>> because it showed there was no term info until I DIH'd again.
>>>>>>
>>>>>>> Are you perhaps in Solr Cloud with more than one replica?
>>>>>>>
>>>>>> Not that I know of, but being new to Solr, there could be things
>>>>>> going
>>>>
>>>> on
>>>>>>
>>>>>> that I'm not aware of. How can I tell? I certainly didn't set
>>>>
>>>> anything up
>>>>>>
>>>>>> for solrCloud deliberately.
>>>>>>
>>>>>>> In that case you
>>>>>>> might be getting the index replicated on startup assuming you
>>>>>>> didn't blow away all replicas. If you are in SolrCloud, I'd just
>>>>>>> delete the collection and start over, after insuring that you'd
>>>>>>> pushed the configset up to Zookeeper.
>>>>>>>
>>>>>>> BTW, I always look at the schema.xml file from the Solr admin
>>>>>>> window
>>>>
>>>> just
>>>>>>>
>>>>>>> as
>>>>>>> a sanity check in these situations.
>>>>>>>
>>>>>> Good idea! But the one shown in the browser is identical to the
>>>>>> one
>>>>
>>>> I've
>>>>>>
>>>>>> been editing! So that's not an issue.
>>>>>>
>>>>>>
>>>
>>>
>>> --
>>> --------------------------
>>>
>>> Benedetti Alessandro
>>> Visiting card - http://about.me/alessandro_benedetti
>>> Blog - http://alexbenedetti.blogspot.co.uk
>>>
>>> "Tyger, tyger burning bright
>>> In the forests of the night,
>>> What immortal hand or eye
>>> Could frame thy fearful symmetry?"
>>>
>>> William Blake - Songs of Experience -1794 England
>>> ________________________________
>>>
>>>
>>> Syngenta Limited, Registered in England No 2710846;Registered Office :
>>> Syngenta Limited, European Regional Centre, Priestley Road, Surrey
>>> Research
>>> Park, Guildford, Surrey, GU2 7YH, United Kingdom
>>> ________________________________
>>> This message may contain confidential information. If you are not the
>>> designated recipient, please notify the sender immediately, and delete
>>> the
>>> original and any copies. Any use of the message by you is prohibited.
>>>
>
Re: query parsing
Posted by Mark Fenbers <ma...@noaa.gov>.
OK, a lot of dialog while I was gone for two days! I read the whole
thread, but I'm a newbie to Solr, so some of the dialog was Greek to
me. I understand the words, of course, but applying it so I know
exactly what to do without screwing something else up is the problem.
After all, that is how I got into the mess in the first place. I'm glad
I have good help to untangle the knots I've made!
I'd like to start over (option 1 below), but does this mean delete all
my config and reinstalling Solr?? Maybe that is not a bad idea, but I
will at least save off my data-config.xml as that is clearly the one
thing that is probably working right. However, I did do quite a bit of
editing that I would have to do again. Please advise...
To be fair, I must answer Erick's question of how I created the data
index in the first place, because this might be relevant...
The bulk of the data is read from 9000+ text files, where each file was
manually typed. Before inserting into the database, I do a little bit
of processing of the text using "sed" to delete the top few and bottom
few lines, and to substitute each single-quote character with a pair of
single-quotes (so PostgreSQL doesn't choke). Line-feed characters are
preserved as ASCII 10 (hex 0A), but there shouldn't be (and I am not
aware of) any characters aside from what is on the keyboard.
Next, I insert it with this command:
psql -U awips -d OHRFC -c "INSERT INTO EventLogText VALUES('$postDate',
'$user', '$postDate', '$entryText', '$postCatVal');"
In case you are wondering about my table, it is defined in this way:
CREATE TABLE eventlogtext (
posttime timestamp without time zone NOT NULL, -- Timestamp of this
entry's original posting
username character varying(8), -- username (logname) of the original
poster
lastmodtime timestamp without time zone, -- Last time record was altered
logtext text, -- text of the log entry
category integer, -- bit-wise category value
CONSTRAINT eventlogtext_pkey PRIMARY KEY (posttime)
)
To do the indexing, I merely use /dataimport?full-import, but it knows
what to do from my data-config.xml; which is here:
<dataConfig>
<dataSource driver="org.postgresql.Driver"
url="jdbc:postgresql://dx1f/OHRFC" user="awips" />
<document>
<entity name="eventlogtext" query="SELECT posttime AS id,
username, logtext, category FROM eventlogtext;"
deltaQuery="SELECT posttime AS id FROM eventlogtext
WHERE lastmodtime > '${dataimporter.last_index_time}';">
<entity name="categorytypes" query="SELECT catname FROM
categorytypes WHERE catid='${eventlogtext.category}';">
</entity>
</entity>
</document>
</dataConfig>
Hope this helps!
Thanks,
Mark
On 9/24/2015 10:57 AM, Erick Erickson wrote:
> Geraint:
>
> Good Catch! I totally missed that. So all of our focus on schema.xml has
> been... totally irrelevant. Now that you pointed that out, there's also the
> addition: add-unknown-fields-to-the-schema, which indicates you started
> this up in "schemaless" mode.
>
> In short, solr is trying to guess what your field types should be and
> guessing wrong (again and again and again). This is the classic weakness of
> schemaless. It's great for indexing stuff fast, but if it guesses wrong
> you're stuck.
>
>
> So to the original problem: I'd start over and either
> 1> use the regular setup, not schemaless
> or
> 2> use the _managed_ schema API to explicitly add fields and fieldTypes to
> the managed schema
>
> Best,
> Erick
>
> On Thu, Sep 24, 2015 at 2:02 AM, Duck Geraint (ext) GBJH <
> Geraint.Duck@syngenta.com> wrote:
>
>> Okay, so maybe I'm missing something here (I'm still relatively new to
>> Solr myself), but am I right in thinking the following is still in your
>> solrconfig.xml file:
>>
>> <schemaFactory class="ManagedIndexSchemaFactory">
>> <bool name="mutable">true</bool>
>> <str name="managedSchemaResourceName">managed-schema</str>
>> </schemaFactory>
>>
>> If so, wouldn't using a managed schema make several of your field
>> definitions inside the schema.xml file semi-redundant?
>>
>> Regards,
>> Geraint
>>
>>
>> Geraint Duck
>> Data Scientist
>> Toxicology and Health Sciences
>> Syngenta UK
>> Email: geraint.duck@syngenta.com
>>
>>
>> -----Original Message-----
>> From: Alessandro Benedetti [mailto:benedetti.alex85@gmail.com]
>> Sent: 24 September 2015 09:23
>> To: solr-user@lucene.apache.org
>> Subject: Re: query parsing
>>
>> I would focus on this :
>>
>> "
>>
>>> 5> now kick off the DIH job and look again.
>>>
>> Now it shows a histogram, but most of the "terms" are long -- the full
>> texts of (the table.column) eventlogtext.logtext, including the whitespace
>> (with %0A used for newline characters)... So, it appears it is not being
>> tokenized properly, correct?"
>> Can you open from your Solr ui , the schema xml and show us the snippets
>> for that field that seems to not tokenise ?
>> Can you show us ( even a screenshot is fine) the schema browser page
>> related ?
>> Could be a problem of encoding ?
>> Following Erick details about the analysis, what are your results ?
>>
>> Cheers
>>
>> 2015-09-24 8:04 GMT+01:00 Upayavira <uv...@odoko.co.uk>:
>>
>>> typically, the index dir is inside the data dir. Delete the index dir
>>> and you should be good. If there is a tlog next to it, you might want
>>> to delete that also.
>>>
>>> If you dont have a data dir, i wonder whether you set the data dir
>>> when creating your core or collection. Typically the instance dir and
>>> data dir aren't needed.
>>>
>>> Upayavira
>>>
>>> On Wed, Sep 23, 2015, at 10:46 PM, Erick Erickson wrote:
>>>> OK, this is bizarre. You'd have had to set up SolrCloud by
>>>> specifying the -zkRun command when you start Solr or the -zkHost;
>>>> highly unlikely. On the admin page there would be a "cloud" link on
>>>> the left side, I really doubt one's there.
>>>>
>>>> You should have a data directory, it should be the parent of the
>>>> index and tlog directories. As of sanity check try looking at the
>>>> analysis page.
>>>> Type
>>>> a bunch of words in the left hand side indexing box and uncheck the
>>>> verbose box. As you can tell I'm grasping at straws. I'm still
>>>> puzzled why you don't have a "data" directory here, but that
>>>> shouldn't really matter. How did you create this index? I don't mean
>>>> data import handler more how did you create the core that you're
>>>> indexing to?
>>>>
>>>> Best,
>>>> Erick
>>>>
>>>> On Wed, Sep 23, 2015 at 10:16 AM, Mark Fenbers
>>>> <ma...@noaa.gov>
>>>> wrote:
>>>>
>>>>> On 9/23/2015 12:30 PM, Erick Erickson wrote:
>>>>>
>>>>>> Then my next guess is you're not pointing at the index you think
>>>>>> you
>>> are
>>>>>> when you 'rm -rf data'
>>>>>>
>>>>>> Just ignore the Elall field for now I should think, although get
>>>>>> rid
>>> of it
>>>>>> if you don't think you need it.
>>>>>>
>>>>>> DIH should be irrelevant here.
>>>>>>
>>>>>> So let's back up.
>>>>>> 1> go ahead and "rm -fr data" (with Solr stopped).
>>>>>>
>>>>> I have no "data" dir. Did you mean "index" dir? I removed 3
>>>>> index directories (2 for spelling):
>>>>> cd /localapps/dev/eventLog; rm -rfv index solr/spFile solr/spIndex
>>>>>
>>>>>> 2> start Solr
>>>>>> 3> do NOT re-index.
>>>>>> 4> look at your index via the schema-browser. Of course there
>>>>>> 4> should
>>> be
>>>>>> nothing there!
>>>>>>
>>>>> Correct! It said "there is no term info :("
>>>>>
>>>>>> 5> now kick off the DIH job and look again.
>>>>>>
>>>>> Now it shows a histogram, but most of the "terms" are long -- the
>>>>> full texts of (the table.column) eventlogtext.logtext, including
>>>>> the
>>> whitespace
>>>>> (with %0A used for newline characters)... So, it appears it is
>>>>> not
>>> being
>>>>> tokenized properly, correct?
>>>>>
>>>>>> Your logtext field should have only single tokens. The fact that
>>>>>> you
>>> have
>>>>>> some very
>>>>>> long tokens presumably with whitespace) indicates that you aren't
>>> really
>>>>>> blowing
>>>>>> the index away between indexing.
>>>>>>
>>>>> Well, I did this time for sure. I verified that initially,
>>>>> because it showed there was no term info until I DIH'd again.
>>>>>
>>>>>> Are you perhaps in Solr Cloud with more than one replica?
>>>>>>
>>>>> Not that I know of, but being new to Solr, there could be things
>>>>> going
>>> on
>>>>> that I'm not aware of. How can I tell? I certainly didn't set
>>> anything up
>>>>> for solrCloud deliberately.
>>>>>
>>>>>> In that case you
>>>>>> might be getting the index replicated on startup assuming you
>>>>>> didn't blow away all replicas. If you are in SolrCloud, I'd just
>>>>>> delete the collection and start over, after insuring that you'd
>>>>>> pushed the configset up to Zookeeper.
>>>>>>
>>>>>> BTW, I always look at the schema.xml file from the Solr admin
>>>>>> window
>>> just
>>>>>> as
>>>>>> a sanity check in these situations.
>>>>>>
>>>>> Good idea! But the one shown in the browser is identical to the
>>>>> one
>>> I've
>>>>> been editing! So that's not an issue.
>>>>>
>>>>>
>>
>>
>> --
>> --------------------------
>>
>> Benedetti Alessandro
>> Visiting card - http://about.me/alessandro_benedetti
>> Blog - http://alexbenedetti.blogspot.co.uk
>>
>> "Tyger, tyger burning bright
>> In the forests of the night,
>> What immortal hand or eye
>> Could frame thy fearful symmetry?"
>>
>> William Blake - Songs of Experience -1794 England
>> ________________________________
>>
>>
>> Syngenta Limited, Registered in England No 2710846;Registered Office :
>> Syngenta Limited, European Regional Centre, Priestley Road, Surrey Research
>> Park, Guildford, Surrey, GU2 7YH, United Kingdom
>> ________________________________
>> This message may contain confidential information. If you are not the
>> designated recipient, please notify the sender immediately, and delete the
>> original and any copies. Any use of the message by you is prohibited.
>>
Re: query parsing
Posted by Erick Erickson <er...@gmail.com>.
Geraint:
Good Catch! I totally missed that. So all of our focus on schema.xml has
been... totally irrelevant. Now that you pointed that out, there's also the
addition: add-unknown-fields-to-the-schema, which indicates you started
this up in "schemaless" mode.
In short, solr is trying to guess what your field types should be and
guessing wrong (again and again and again). This is the classic weakness of
schemaless. It's great for indexing stuff fast, but if it guesses wrong
you're stuck.
So to the original problem: I'd start over and either
1> use the regular setup, not schemaless
or
2> use the _managed_ schema API to explicitly add fields and fieldTypes to
the managed schema
Best,
Erick
On Thu, Sep 24, 2015 at 2:02 AM, Duck Geraint (ext) GBJH <
Geraint.Duck@syngenta.com> wrote:
> Okay, so maybe I'm missing something here (I'm still relatively new to
> Solr myself), but am I right in thinking the following is still in your
> solrconfig.xml file:
>
> <schemaFactory class="ManagedIndexSchemaFactory">
> <bool name="mutable">true</bool>
> <str name="managedSchemaResourceName">managed-schema</str>
> </schemaFactory>
>
> If so, wouldn't using a managed schema make several of your field
> definitions inside the schema.xml file semi-redundant?
>
> Regards,
> Geraint
>
>
> Geraint Duck
> Data Scientist
> Toxicology and Health Sciences
> Syngenta UK
> Email: geraint.duck@syngenta.com
>
>
> -----Original Message-----
> From: Alessandro Benedetti [mailto:benedetti.alex85@gmail.com]
> Sent: 24 September 2015 09:23
> To: solr-user@lucene.apache.org
> Subject: Re: query parsing
>
> I would focus on this :
>
> "
>
> > 5> now kick off the DIH job and look again.
> >
> Now it shows a histogram, but most of the "terms" are long -- the full
> texts of (the table.column) eventlogtext.logtext, including the whitespace
> (with %0A used for newline characters)... So, it appears it is not being
> tokenized properly, correct?"
> Can you open from your Solr ui , the schema xml and show us the snippets
> for that field that seems to not tokenise ?
> Can you show us ( even a screenshot is fine) the schema browser page
> related ?
> Could be a problem of encoding ?
> Following Erick details about the analysis, what are your results ?
>
> Cheers
>
> 2015-09-24 8:04 GMT+01:00 Upayavira <uv...@odoko.co.uk>:
>
> > typically, the index dir is inside the data dir. Delete the index dir
> > and you should be good. If there is a tlog next to it, you might want
> > to delete that also.
> >
> > If you dont have a data dir, i wonder whether you set the data dir
> > when creating your core or collection. Typically the instance dir and
> > data dir aren't needed.
> >
> > Upayavira
> >
> > On Wed, Sep 23, 2015, at 10:46 PM, Erick Erickson wrote:
> > > OK, this is bizarre. You'd have had to set up SolrCloud by
> > > specifying the -zkRun command when you start Solr or the -zkHost;
> > > highly unlikely. On the admin page there would be a "cloud" link on
> > > the left side, I really doubt one's there.
> > >
> > > You should have a data directory, it should be the parent of the
> > > index and tlog directories. As of sanity check try looking at the
> > > analysis page.
> > > Type
> > > a bunch of words in the left hand side indexing box and uncheck the
> > > verbose box. As you can tell I'm grasping at straws. I'm still
> > > puzzled why you don't have a "data" directory here, but that
> > > shouldn't really matter. How did you create this index? I don't mean
> > > data import handler more how did you create the core that you're
> > > indexing to?
> > >
> > > Best,
> > > Erick
> > >
> > > On Wed, Sep 23, 2015 at 10:16 AM, Mark Fenbers
> > > <ma...@noaa.gov>
> > > wrote:
> > >
> > > > On 9/23/2015 12:30 PM, Erick Erickson wrote:
> > > >
> > > >> Then my next guess is you're not pointing at the index you think
> > > >> you
> > are
> > > >> when you 'rm -rf data'
> > > >>
> > > >> Just ignore the Elall field for now I should think, although get
> > > >> rid
> > of it
> > > >> if you don't think you need it.
> > > >>
> > > >> DIH should be irrelevant here.
> > > >>
> > > >> So let's back up.
> > > >> 1> go ahead and "rm -fr data" (with Solr stopped).
> > > >>
> > > > I have no "data" dir. Did you mean "index" dir? I removed 3
> > > > index directories (2 for spelling):
> > > > cd /localapps/dev/eventLog; rm -rfv index solr/spFile solr/spIndex
> > > >
> > > >> 2> start Solr
> > > >> 3> do NOT re-index.
> > > >> 4> look at your index via the schema-browser. Of course there
> > > >> 4> should
> > be
> > > >> nothing there!
> > > >>
> > > > Correct! It said "there is no term info :("
> > > >
> > > >> 5> now kick off the DIH job and look again.
> > > >>
> > > > Now it shows a histogram, but most of the "terms" are long -- the
> > > > full texts of (the table.column) eventlogtext.logtext, including
> > > > the
> > whitespace
> > > > (with %0A used for newline characters)... So, it appears it is
> > > > not
> > being
> > > > tokenized properly, correct?
> > > >
> > > >> Your logtext field should have only single tokens. The fact that
> > > >> you
> > have
> > > >> some very
> > > >> long tokens presumably with whitespace) indicates that you aren't
> > really
> > > >> blowing
> > > >> the index away between indexing.
> > > >>
> > > > Well, I did this time for sure. I verified that initially,
> > > > because it showed there was no term info until I DIH'd again.
> > > >
> > > >> Are you perhaps in Solr Cloud with more than one replica?
> > > >>
> > > > Not that I know of, but being new to Solr, there could be things
> > > > going
> > on
> > > > that I'm not aware of. How can I tell? I certainly didn't set
> > anything up
> > > > for solrCloud deliberately.
> > > >
> > > >> In that case you
> > > >> might be getting the index replicated on startup assuming you
> > > >> didn't blow away all replicas. If you are in SolrCloud, I'd just
> > > >> delete the collection and start over, after insuring that you'd
> > > >> pushed the configset up to Zookeeper.
> > > >>
> > > >> BTW, I always look at the schema.xml file from the Solr admin
> > > >> window
> > just
> > > >> as
> > > >> a sanity check in these situations.
> > > >>
> > > > Good idea! But the one shown in the browser is identical to the
> > > > one
> > I've
> > > > been editing! So that's not an issue.
> > > >
> > > >
> >
>
>
>
> --
> --------------------------
>
> Benedetti Alessandro
> Visiting card - http://about.me/alessandro_benedetti
> Blog - http://alexbenedetti.blogspot.co.uk
>
> "Tyger, tyger burning bright
> In the forests of the night,
> What immortal hand or eye
> Could frame thy fearful symmetry?"
>
> William Blake - Songs of Experience -1794 England
> ________________________________
>
>
> Syngenta Limited, Registered in England No 2710846;Registered Office :
> Syngenta Limited, European Regional Centre, Priestley Road, Surrey Research
> Park, Guildford, Surrey, GU2 7YH, United Kingdom
> ________________________________
> This message may contain confidential information. If you are not the
> designated recipient, please notify the sender immediately, and delete the
> original and any copies. Any use of the message by you is prohibited.
>
RE: query parsing
Posted by "Duck Geraint (ext) GBJH" <Ge...@SYNGENTA.COM>.
Okay, so maybe I'm missing something here (I'm still relatively new to Solr myself), but am I right in thinking the following is still in your solrconfig.xml file:
<schemaFactory class="ManagedIndexSchemaFactory">
<bool name="mutable">true</bool>
<str name="managedSchemaResourceName">managed-schema</str>
</schemaFactory>
If so, wouldn't using a managed schema make several of your field definitions inside the schema.xml file semi-redundant?
Regards,
Geraint
Geraint Duck
Data Scientist
Toxicology and Health Sciences
Syngenta UK
Email: geraint.duck@syngenta.com
-----Original Message-----
From: Alessandro Benedetti [mailto:benedetti.alex85@gmail.com]
Sent: 24 September 2015 09:23
To: solr-user@lucene.apache.org
Subject: Re: query parsing
I would focus on this :
"
> 5> now kick off the DIH job and look again.
>
Now it shows a histogram, but most of the "terms" are long -- the full texts of (the table.column) eventlogtext.logtext, including the whitespace (with %0A used for newline characters)... So, it appears it is not being tokenized properly, correct?"
Can you open from your Solr ui , the schema xml and show us the snippets for that field that seems to not tokenise ?
Can you show us ( even a screenshot is fine) the schema browser page related ?
Could be a problem of encoding ?
Following Erick details about the analysis, what are your results ?
Cheers
2015-09-24 8:04 GMT+01:00 Upayavira <uv...@odoko.co.uk>:
> typically, the index dir is inside the data dir. Delete the index dir
> and you should be good. If there is a tlog next to it, you might want
> to delete that also.
>
> If you dont have a data dir, i wonder whether you set the data dir
> when creating your core or collection. Typically the instance dir and
> data dir aren't needed.
>
> Upayavira
>
> On Wed, Sep 23, 2015, at 10:46 PM, Erick Erickson wrote:
> > OK, this is bizarre. You'd have had to set up SolrCloud by
> > specifying the -zkRun command when you start Solr or the -zkHost;
> > highly unlikely. On the admin page there would be a "cloud" link on
> > the left side, I really doubt one's there.
> >
> > You should have a data directory, it should be the parent of the
> > index and tlog directories. As of sanity check try looking at the
> > analysis page.
> > Type
> > a bunch of words in the left hand side indexing box and uncheck the
> > verbose box. As you can tell I'm grasping at straws. I'm still
> > puzzled why you don't have a "data" directory here, but that
> > shouldn't really matter. How did you create this index? I don't mean
> > data import handler more how did you create the core that you're
> > indexing to?
> >
> > Best,
> > Erick
> >
> > On Wed, Sep 23, 2015 at 10:16 AM, Mark Fenbers
> > <ma...@noaa.gov>
> > wrote:
> >
> > > On 9/23/2015 12:30 PM, Erick Erickson wrote:
> > >
> > >> Then my next guess is you're not pointing at the index you think
> > >> you
> are
> > >> when you 'rm -rf data'
> > >>
> > >> Just ignore the Elall field for now I should think, although get
> > >> rid
> of it
> > >> if you don't think you need it.
> > >>
> > >> DIH should be irrelevant here.
> > >>
> > >> So let's back up.
> > >> 1> go ahead and "rm -fr data" (with Solr stopped).
> > >>
> > > I have no "data" dir. Did you mean "index" dir? I removed 3
> > > index directories (2 for spelling):
> > > cd /localapps/dev/eventLog; rm -rfv index solr/spFile solr/spIndex
> > >
> > >> 2> start Solr
> > >> 3> do NOT re-index.
> > >> 4> look at your index via the schema-browser. Of course there
> > >> 4> should
> be
> > >> nothing there!
> > >>
> > > Correct! It said "there is no term info :("
> > >
> > >> 5> now kick off the DIH job and look again.
> > >>
> > > Now it shows a histogram, but most of the "terms" are long -- the
> > > full texts of (the table.column) eventlogtext.logtext, including
> > > the
> whitespace
> > > (with %0A used for newline characters)... So, it appears it is
> > > not
> being
> > > tokenized properly, correct?
> > >
> > >> Your logtext field should have only single tokens. The fact that
> > >> you
> have
> > >> some very
> > >> long tokens presumably with whitespace) indicates that you aren't
> really
> > >> blowing
> > >> the index away between indexing.
> > >>
> > > Well, I did this time for sure. I verified that initially,
> > > because it showed there was no term info until I DIH'd again.
> > >
> > >> Are you perhaps in Solr Cloud with more than one replica?
> > >>
> > > Not that I know of, but being new to Solr, there could be things
> > > going
> on
> > > that I'm not aware of. How can I tell? I certainly didn't set
> anything up
> > > for solrCloud deliberately.
> > >
> > >> In that case you
> > >> might be getting the index replicated on startup assuming you
> > >> didn't blow away all replicas. If you are in SolrCloud, I'd just
> > >> delete the collection and start over, after insuring that you'd
> > >> pushed the configset up to Zookeeper.
> > >>
> > >> BTW, I always look at the schema.xml file from the Solr admin
> > >> window
> just
> > >> as
> > >> a sanity check in these situations.
> > >>
> > > Good idea! But the one shown in the browser is identical to the
> > > one
> I've
> > > been editing! So that's not an issue.
> > >
> > >
>
--
--------------------------
Benedetti Alessandro
Visiting card - http://about.me/alessandro_benedetti
Blog - http://alexbenedetti.blogspot.co.uk
"Tyger, tyger burning bright
In the forests of the night,
What immortal hand or eye
Could frame thy fearful symmetry?"
William Blake - Songs of Experience -1794 England
________________________________
Syngenta Limited, Registered in England No 2710846;Registered Office : Syngenta Limited, European Regional Centre, Priestley Road, Surrey Research Park, Guildford, Surrey, GU2 7YH, United Kingdom
________________________________
This message may contain confidential information. If you are not the designated recipient, please notify the sender immediately, and delete the original and any copies. Any use of the message by you is prohibited.
Re: query parsing
Posted by Alessandro Benedetti <be...@gmail.com>.
I would focus on this :
"
> 5> now kick off the DIH job and look again.
>
Now it shows a histogram, but most of the "terms" are long -- the full
texts of (the table.column) eventlogtext.logtext, including the whitespace
(with %0A used for newline characters)... So, it appears it is not being
tokenized properly, correct?"
Can you open from your Solr ui , the schema xml and show us the snippets
for that field that seems to not tokenise ?
Can you show us ( even a screenshot is fine) the schema browser page
related ?
Could be a problem of encoding ?
Following Erick details about the analysis, what are your results ?
Cheers
2015-09-24 8:04 GMT+01:00 Upayavira <uv...@odoko.co.uk>:
> typically, the index dir is inside the data dir. Delete the index dir
> and you should be good. If there is a tlog next to it, you might want to
> delete that also.
>
> If you dont have a data dir, i wonder whether you set the data dir when
> creating your core or collection. Typically the instance dir and data
> dir aren't needed.
>
> Upayavira
>
> On Wed, Sep 23, 2015, at 10:46 PM, Erick Erickson wrote:
> > OK, this is bizarre. You'd have had to set up SolrCloud by specifying the
> > -zkRun command when you start Solr or the -zkHost; highly unlikely. On
> > the
> > admin page there would be a "cloud" link on the left side, I really doubt
> > one's there.
> >
> > You should have a data directory, it should be the parent of the index
> > and
> > tlog directories. As of sanity check try looking at the analysis page.
> > Type
> > a bunch of words in the left hand side indexing box and uncheck the
> > verbose
> > box. As you can tell I'm grasping at straws. I'm still puzzled why you
> > don't have a "data" directory here, but that shouldn't really matter. How
> > did you create this index? I don't mean data import handler more how did
> > you create the core that you're indexing to?
> >
> > Best,
> > Erick
> >
> > On Wed, Sep 23, 2015 at 10:16 AM, Mark Fenbers <ma...@noaa.gov>
> > wrote:
> >
> > > On 9/23/2015 12:30 PM, Erick Erickson wrote:
> > >
> > >> Then my next guess is you're not pointing at the index you think you
> are
> > >> when you 'rm -rf data'
> > >>
> > >> Just ignore the Elall field for now I should think, although get rid
> of it
> > >> if you don't think you need it.
> > >>
> > >> DIH should be irrelevant here.
> > >>
> > >> So let's back up.
> > >> 1> go ahead and "rm -fr data" (with Solr stopped).
> > >>
> > > I have no "data" dir. Did you mean "index" dir? I removed 3 index
> > > directories (2 for spelling):
> > > cd /localapps/dev/eventLog; rm -rfv index solr/spFile solr/spIndex
> > >
> > >> 2> start Solr
> > >> 3> do NOT re-index.
> > >> 4> look at your index via the schema-browser. Of course there should
> be
> > >> nothing there!
> > >>
> > > Correct! It said "there is no term info :("
> > >
> > >> 5> now kick off the DIH job and look again.
> > >>
> > > Now it shows a histogram, but most of the "terms" are long -- the full
> > > texts of (the table.column) eventlogtext.logtext, including the
> whitespace
> > > (with %0A used for newline characters)... So, it appears it is not
> being
> > > tokenized properly, correct?
> > >
> > >> Your logtext field should have only single tokens. The fact that you
> have
> > >> some very
> > >> long tokens presumably with whitespace) indicates that you aren't
> really
> > >> blowing
> > >> the index away between indexing.
> > >>
> > > Well, I did this time for sure. I verified that initially, because it
> > > showed there was no term info until I DIH'd again.
> > >
> > >> Are you perhaps in Solr Cloud with more than one replica?
> > >>
> > > Not that I know of, but being new to Solr, there could be things going
> on
> > > that I'm not aware of. How can I tell? I certainly didn't set
> anything up
> > > for solrCloud deliberately.
> > >
> > >> In that case you
> > >> might be getting the index replicated on startup assuming you didn't
> > >> blow away all replicas. If you are in SolrCloud, I'd just delete the
> > >> collection and
> > >> start over, after insuring that you'd pushed the configset up to
> > >> Zookeeper.
> > >>
> > >> BTW, I always look at the schema.xml file from the Solr admin window
> just
> > >> as
> > >> a sanity check in these situations.
> > >>
> > > Good idea! But the one shown in the browser is identical to the one
> I've
> > > been editing! So that's not an issue.
> > >
> > >
>
--
--------------------------
Benedetti Alessandro
Visiting card - http://about.me/alessandro_benedetti
Blog - http://alexbenedetti.blogspot.co.uk
"Tyger, tyger burning bright
In the forests of the night,
What immortal hand or eye
Could frame thy fearful symmetry?"
William Blake - Songs of Experience -1794 England
Re: query parsing
Posted by Upayavira <uv...@odoko.co.uk>.
typically, the index dir is inside the data dir. Delete the index dir
and you should be good. If there is a tlog next to it, you might want to
delete that also.
If you dont have a data dir, i wonder whether you set the data dir when
creating your core or collection. Typically the instance dir and data
dir aren't needed.
Upayavira
On Wed, Sep 23, 2015, at 10:46 PM, Erick Erickson wrote:
> OK, this is bizarre. You'd have had to set up SolrCloud by specifying the
> -zkRun command when you start Solr or the -zkHost; highly unlikely. On
> the
> admin page there would be a "cloud" link on the left side, I really doubt
> one's there.
>
> You should have a data directory, it should be the parent of the index
> and
> tlog directories. As of sanity check try looking at the analysis page.
> Type
> a bunch of words in the left hand side indexing box and uncheck the
> verbose
> box. As you can tell I'm grasping at straws. I'm still puzzled why you
> don't have a "data" directory here, but that shouldn't really matter. How
> did you create this index? I don't mean data import handler more how did
> you create the core that you're indexing to?
>
> Best,
> Erick
>
> On Wed, Sep 23, 2015 at 10:16 AM, Mark Fenbers <ma...@noaa.gov>
> wrote:
>
> > On 9/23/2015 12:30 PM, Erick Erickson wrote:
> >
> >> Then my next guess is you're not pointing at the index you think you are
> >> when you 'rm -rf data'
> >>
> >> Just ignore the Elall field for now I should think, although get rid of it
> >> if you don't think you need it.
> >>
> >> DIH should be irrelevant here.
> >>
> >> So let's back up.
> >> 1> go ahead and "rm -fr data" (with Solr stopped).
> >>
> > I have no "data" dir. Did you mean "index" dir? I removed 3 index
> > directories (2 for spelling):
> > cd /localapps/dev/eventLog; rm -rfv index solr/spFile solr/spIndex
> >
> >> 2> start Solr
> >> 3> do NOT re-index.
> >> 4> look at your index via the schema-browser. Of course there should be
> >> nothing there!
> >>
> > Correct! It said "there is no term info :("
> >
> >> 5> now kick off the DIH job and look again.
> >>
> > Now it shows a histogram, but most of the "terms" are long -- the full
> > texts of (the table.column) eventlogtext.logtext, including the whitespace
> > (with %0A used for newline characters)... So, it appears it is not being
> > tokenized properly, correct?
> >
> >> Your logtext field should have only single tokens. The fact that you have
> >> some very
> >> long tokens presumably with whitespace) indicates that you aren't really
> >> blowing
> >> the index away between indexing.
> >>
> > Well, I did this time for sure. I verified that initially, because it
> > showed there was no term info until I DIH'd again.
> >
> >> Are you perhaps in Solr Cloud with more than one replica?
> >>
> > Not that I know of, but being new to Solr, there could be things going on
> > that I'm not aware of. How can I tell? I certainly didn't set anything up
> > for solrCloud deliberately.
> >
> >> In that case you
> >> might be getting the index replicated on startup assuming you didn't
> >> blow away all replicas. If you are in SolrCloud, I'd just delete the
> >> collection and
> >> start over, after insuring that you'd pushed the configset up to
> >> Zookeeper.
> >>
> >> BTW, I always look at the schema.xml file from the Solr admin window just
> >> as
> >> a sanity check in these situations.
> >>
> > Good idea! But the one shown in the browser is identical to the one I've
> > been editing! So that's not an issue.
> >
> >
Re: query parsing
Posted by Erick Erickson <er...@gmail.com>.
OK, this is bizarre. You'd have had to set up SolrCloud by specifying the
-zkRun command when you start Solr or the -zkHost; highly unlikely. On the
admin page there would be a "cloud" link on the left side, I really doubt
one's there.
You should have a data directory, it should be the parent of the index and
tlog directories. As of sanity check try looking at the analysis page. Type
a bunch of words in the left hand side indexing box and uncheck the verbose
box. As you can tell I'm grasping at straws. I'm still puzzled why you
don't have a "data" directory here, but that shouldn't really matter. How
did you create this index? I don't mean data import handler more how did
you create the core that you're indexing to?
Best,
Erick
On Wed, Sep 23, 2015 at 10:16 AM, Mark Fenbers <ma...@noaa.gov>
wrote:
> On 9/23/2015 12:30 PM, Erick Erickson wrote:
>
>> Then my next guess is you're not pointing at the index you think you are
>> when you 'rm -rf data'
>>
>> Just ignore the Elall field for now I should think, although get rid of it
>> if you don't think you need it.
>>
>> DIH should be irrelevant here.
>>
>> So let's back up.
>> 1> go ahead and "rm -fr data" (with Solr stopped).
>>
> I have no "data" dir. Did you mean "index" dir? I removed 3 index
> directories (2 for spelling):
> cd /localapps/dev/eventLog; rm -rfv index solr/spFile solr/spIndex
>
>> 2> start Solr
>> 3> do NOT re-index.
>> 4> look at your index via the schema-browser. Of course there should be
>> nothing there!
>>
> Correct! It said "there is no term info :("
>
>> 5> now kick off the DIH job and look again.
>>
> Now it shows a histogram, but most of the "terms" are long -- the full
> texts of (the table.column) eventlogtext.logtext, including the whitespace
> (with %0A used for newline characters)... So, it appears it is not being
> tokenized properly, correct?
>
>> Your logtext field should have only single tokens. The fact that you have
>> some very
>> long tokens presumably with whitespace) indicates that you aren't really
>> blowing
>> the index away between indexing.
>>
> Well, I did this time for sure. I verified that initially, because it
> showed there was no term info until I DIH'd again.
>
>> Are you perhaps in Solr Cloud with more than one replica?
>>
> Not that I know of, but being new to Solr, there could be things going on
> that I'm not aware of. How can I tell? I certainly didn't set anything up
> for solrCloud deliberately.
>
>> In that case you
>> might be getting the index replicated on startup assuming you didn't
>> blow away all replicas. If you are in SolrCloud, I'd just delete the
>> collection and
>> start over, after insuring that you'd pushed the configset up to
>> Zookeeper.
>>
>> BTW, I always look at the schema.xml file from the Solr admin window just
>> as
>> a sanity check in these situations.
>>
> Good idea! But the one shown in the browser is identical to the one I've
> been editing! So that's not an issue.
>
>
Re: query parsing
Posted by Mark Fenbers <ma...@noaa.gov>.
On 9/23/2015 12:30 PM, Erick Erickson wrote:
> Then my next guess is you're not pointing at the index you think you are
> when you 'rm -rf data'
>
> Just ignore the Elall field for now I should think, although get rid of it
> if you don't think you need it.
>
> DIH should be irrelevant here.
>
> So let's back up.
> 1> go ahead and "rm -fr data" (with Solr stopped).
I have no "data" dir. Did you mean "index" dir? I removed 3 index
directories (2 for spelling):
cd /localapps/dev/eventLog; rm -rfv index solr/spFile solr/spIndex
> 2> start Solr
> 3> do NOT re-index.
> 4> look at your index via the schema-browser. Of course there should be
> nothing there!
Correct! It said "there is no term info :("
> 5> now kick off the DIH job and look again.
Now it shows a histogram, but most of the "terms" are long -- the full
texts of (the table.column) eventlogtext.logtext, including the
whitespace (with %0A used for newline characters)... So, it appears it
is not being tokenized properly, correct?
> Your logtext field should have only single tokens. The fact that you have
> some very
> long tokens presumably with whitespace) indicates that you aren't really
> blowing
> the index away between indexing.
Well, I did this time for sure. I verified that initially, because it
showed there was no term info until I DIH'd again.
> Are you perhaps in Solr Cloud with more than one replica?
Not that I know of, but being new to Solr, there could be things going
on that I'm not aware of. How can I tell? I certainly didn't set
anything up for solrCloud deliberately.
> In that case you
> might be getting the index replicated on startup assuming you didn't
> blow away all replicas. If you are in SolrCloud, I'd just delete the
> collection and
> start over, after insuring that you'd pushed the configset up to Zookeeper.
>
> BTW, I always look at the schema.xml file from the Solr admin window just as
> a sanity check in these situations.
Good idea! But the one shown in the browser is identical to the one
I've been editing! So that's not an issue.
Re: query parsing
Posted by Erick Erickson <er...@gmail.com>.
Then my next guess is you're not pointing at the index you think you are
when you 'rm -rf data'
Just ignore the Elall field for now I should think, although get rid of it
if you don't think you need it.
DIH should be irrelevant here.
So let's back up.
1> go ahead and "rm -fr data" (with Solr stopped).
2> start Solr
3> do NOT re-index.
4> look at your index via the schema-browser. Of course there should be
nothing there!
5> now kick off the DIH job and look again.
Your logtext field should have only single tokens. The fact that you have
some very
long tokens presumably with whitespace) indicates that you aren't really
blowing
the index away between indexing.
Are you perhaps in Solr Cloud with more than one replica? In that case you
might be getting the index replicated on startup assuming you didn't
blow away all replicas. If you are in SolrCloud, I'd just delete the
collection and
start over, after insuring that you'd pushed the configset up to Zookeeper.
BTW, I always look at the schema.xml file from the Solr admin window just as
a sanity check in these situations.
Best,
Erick
On Wed, Sep 23, 2015 at 9:22 AM, Mark Fenbers <ma...@noaa.gov> wrote:
> On 9/23/2015 11:28 AM, Erick Erickson wrote:
>
>> This is totally weird.
>>
>> Don't only re-index your old docs, find the data directory and
>> rm -rf data (with Solr stopped) and re-index.
>>
> I pretty much do that. The thing is: I don't have a data directory
> anywhere! Most of my stuff is in /localapps/dev/EventLog/solr/, but I *do*
> have a /localapps/dev/EventLog/index/ directory where the main index
> resides. I'd like to move that into /localapps/dev/EventLog/solr/ so that
> I can keep all Solr-related files under one parent dir, but I can't find
> where the configuration for that is...
>
> Perhaps I should also share what start command I'm using (in case it is
> wrong!):
>
> /localapps/dev/solr-5.3.0/bin/solr start -s /localapps/dev/EventLog
>
>> re: the analysis page Alessandro mentioned.
>> Go to the Solr admin UI (http://localhost:8983/solr). You'll
>> see a drop-down on the left that lets you select a core,
>> select the appropriate one.
>>
>> Now you'll see a bunch of new choices. The "analysis" section
>> is what Alessandro is referencing. That shows you _exactly_ what
>> effects your analysis chain has at index and query time.
>>
>> On the same page, you'll find "schema browser". Take a look at
>> your logtext field and hit the "load term info" button. You should
>> see a bunch of single-word tokens listed. If you see really long ones,
>> then your index is hosed and you should start by blowing away
>> the data directory....
>>
> I wish I could show a screen capture! But according to your symptoms, my
> index is hosed (I see very few single-word tokens and lots of really long
> ones.) I have no data directory to blow away, though. I've blown away
> /localapps/dev/EventLog/index/ before, but that has had no effect on the
> problem.
>
> Am I indexing improperly perhaps? I'm using /dataimport. Here is my
> data-config.xml, which hasn't been giving me any obvious trouble. Import
> seems successful. And I can get correct search results so long as I wrap
> my search text in asterisks...
>
> <?xml version="1.0"?>
> <dataConfig>
> <dataSource user="awips" url="jdbc:postgresql://dx1f/OHRFC"
> driver="org.postgresql.Driver"/>
> <document>
> <entity deltaQuery="SELECT posttime AS id FROM eventlogtext
> WHERE lastmodtime > '${dataimporter.last_index_time}';" query="SELECT
> posttime AS id, username, logtext, category FROM eventlogtext;"
> name="eventlogtext">
> <entity query="SELECT catname FROM categorytypes WHERE
> catid='${eventlogtext.category}';" name="categorytypes"> </entity>
> </entity>
> </document>
> </dataConfig>
>
> Because this symptom is totally explained by searching on a "string"
>> rather than a "text" type. But your definition is clearly a tokenized text
>> type so I'm mystified.
>>
>> The ELall field is a red herring. The debug output shows you're searching
>> on the logtext field, this line is the relevant one:
>> "parsedquery_toString":"logtext:deeper",
>>
> Should I just get rid of "ELall"? I only created it with the intent to be
> able to search on "fenbers" and get hits if "fenbers" occurred in either
> place, the logtext field or the username field.
>
> thanks,
> Mark
>
>
Re: query parsing
Posted by Mark Fenbers <ma...@noaa.gov>.
On 9/23/2015 11:28 AM, Erick Erickson wrote:
> This is totally weird.
>
> Don't only re-index your old docs, find the data directory and
> rm -rf data (with Solr stopped) and re-index.
I pretty much do that. The thing is: I don't have a data directory
anywhere! Most of my stuff is in /localapps/dev/EventLog/solr/, but I
*do* have a /localapps/dev/EventLog/index/ directory where the main
index resides. I'd like to move that into /localapps/dev/EventLog/solr/
so that I can keep all Solr-related files under one parent dir, but I
can't find where the configuration for that is...
Perhaps I should also share what start command I'm using (in case it is
wrong!):
/localapps/dev/solr-5.3.0/bin/solr start -s /localapps/dev/EventLog
> re: the analysis page Alessandro mentioned.
> Go to the Solr admin UI (http://localhost:8983/solr). You'll
> see a drop-down on the left that lets you select a core,
> select the appropriate one.
>
> Now you'll see a bunch of new choices. The "analysis" section
> is what Alessandro is referencing. That shows you _exactly_ what
> effects your analysis chain has at index and query time.
>
> On the same page, you'll find "schema browser". Take a look at
> your logtext field and hit the "load term info" button. You should
> see a bunch of single-word tokens listed. If you see really long ones,
> then your index is hosed and you should start by blowing away
> the data directory....
I wish I could show a screen capture! But according to your symptoms,
my index is hosed (I see very few single-word tokens and lots of really
long ones.) I have no data directory to blow away, though. I've blown
away /localapps/dev/EventLog/index/ before, but that has had no effect
on the problem.
Am I indexing improperly perhaps? I'm using /dataimport. Here is my
data-config.xml, which hasn't been giving me any obvious trouble.
Import seems successful. And I can get correct search results so long
as I wrap my search text in asterisks...
<?xml version="1.0"?>
<dataConfig>
<dataSource user="awips" url="jdbc:postgresql://dx1f/OHRFC"
driver="org.postgresql.Driver"/>
<document>
<entity deltaQuery="SELECT posttime AS id FROM eventlogtext
WHERE lastmodtime > '${dataimporter.last_index_time}';" query="SELECT
posttime AS id, username, logtext, category FROM eventlogtext;"
name="eventlogtext">
<entity query="SELECT catname FROM categorytypes WHERE
catid='${eventlogtext.category}';" name="categorytypes"> </entity>
</entity>
</document>
</dataConfig>
> Because this symptom is totally explained by searching on a "string"
> rather than a "text" type. But your definition is clearly a tokenized text
> type so I'm mystified.
>
> The ELall field is a red herring. The debug output shows you're searching
> on the logtext field, this line is the relevant one:
> "parsedquery_toString":"logtext:deeper",
Should I just get rid of "ELall"? I only created it with the intent to
be able to search on "fenbers" and get hits if "fenbers" occurred in
either place, the logtext field or the username field.
thanks,
Mark
Re: query parsing
Posted by Erick Erickson <er...@gmail.com>.
This is totally weird.
Don't only re-index your old docs, find the data directory and
rm -rf data (with Solr stopped) and re-index.
re: the analysis page Alessandro mentioned.
Go to the Solr admin UI (http://localhost:8983/solr). You'll
see a drop-down on the left that lets you select a core,
select the appropriate one.
Now you'll see a bunch of new choices. The "analysis" section
is what Alessandro is referencing. That shows you _exactly_ what
effects your analysis chain has at index and query time.
On the same page, you'll find "schema browser". Take a look at
your logtext field and hit the "load term info" button. You should
see a bunch of single-word tokens listed. If you see really long ones,
then your index is hosed and you should start by blowing away
the data directory....
Because this symptom is totally explained by searching on a "string"
rather than a "text" type. But your definition is clearly a tokenized text
type so I'm mystified.
The ELall field is a red herring. The debug output shows you're searching
on the logtext field, this line is the relevant one:
"parsedquery_toString":"logtext:deeper",
Best,
Erick
On Wed, Sep 23, 2015 at 8:07 AM, Mark Fenbers <ma...@noaa.gov> wrote:
> On 9/23/2015 10:21 AM, Alessandro Benedetti wrote:
>
>> mmmmm so those 2 are the queries at the minute :
>>
>> 1) logtext:deeper
>> 2) logtext:*deeper*
>>
>> According to your schema, the log text field is of type "text_en".
>> This should be completely fine.
>> Have you ever changed your schema on run ? without re-indexing your old
>> docs ?
>>
> I might forget sometimes, but usually, when I make changes to
> solrconfig.xml or schema.xml, then I delete the main index and the
> spellchecker indexes, and then restart solr, then do /dataimport again.
>
>> What happens if you use your analysis tool ( both query and index time)
>> with the term deeper ?
>>
> Can you clarify what you want me to do here? What do you want me to put
> in the (Index) text box and in the (Query) text box and what do I select in
> the fieldType drop-list? When I put "deeper" into both text boxes and
> select text_en from the drop list, I get several results, but I don't know
> what the output means.
>
> thanks,
> Mark
>
Re: query parsing
Posted by Mark Fenbers <ma...@noaa.gov>.
On 9/23/2015 10:21 AM, Alessandro Benedetti wrote:
> mmmmm so those 2 are the queries at the minute :
>
> 1) logtext:deeper
> 2) logtext:*deeper*
>
> According to your schema, the log text field is of type "text_en".
> This should be completely fine.
> Have you ever changed your schema on run ? without re-indexing your old
> docs ?
I might forget sometimes, but usually, when I make changes to
solrconfig.xml or schema.xml, then I delete the main index and the
spellchecker indexes, and then restart solr, then do /dataimport again.
> What happens if you use your analysis tool ( both query and index time)
> with the term deeper ?
Can you clarify what you want me to do here? What do you want me to put
in the (Index) text box and in the (Query) text box and what do I select
in the fieldType drop-list? When I put "deeper" into both text boxes
and select text_en from the drop list, I get several results, but I
don't know what the output means.
thanks,
Mark
Re: query parsing
Posted by Alessandro Benedetti <be...@gmail.com>.
mmmmm so those 2 are the queries at the minute :
1) logtext:deeper
2) logtext:*deeper*
According to your schema, the log text field is of type "text_en".
This should be completely fine.
Have you ever changed your schema on run ? without re-indexing your old
docs ?
What happens if you use your analysis tool ( both query and index time)
with the term deeper ?
Cheers
2015-09-23 15:10 GMT+01:00 Mark Fenbers <ma...@noaa.gov>:
> Mugeesh, I believe you are on the right path and I was eager to try out
> your suggestion. So my schema.xml now contains this snippet (changes
> indicated by ~):
>
> <field name="id" type="date" indexed="true" stored="true" required="true"
> />
> ~ <field name="logtext" type="logtype" indexed="true"
> stored="true" required="true" />
> <field name="category" type="int" indexed="true" stored="true"
> required="true" />
> <field name="username" type="string" indexed="true" stored="true"
> required="true" />
> ~ <field name="ELall" type="logtype" indexed="true" stored="true"
> multiValued="true" />
> <copyField source="logtext" dest="ELall" />
> <copyField source="username" dest="ELall" />
>
> ~ <fieldType name="logtype" class="solr.TextField">
> ~ <analyzer type="index">
> ~ <tokenizer class="solr.StandardTokenizerFactory" />
> ~ <filter class="solr.StandardFilterFactory" />
> ~ </analyzer>
> ~ </fieldType>
>
> but my results are the same -- that my search yields 0 results unless I
> wrap the search word with asterisks.
>
> Alessandro, below are the results (with and without the asterisks) with
> debug turned on. I don't know what much of the debug info means. Is it
> giving you more clues?
>
>
> http://localhost:8983/solr/EventLog/select?q=deeper&wt=json&indent=true&debugQuery=true
>
> {
> "responseHeader":{
> "status":0,
> "QTime":2,
> "params":{
> "q":"deeper",
> "indent":"true",
> "wt":"json",
> "debugQuery":"true"}},
> "response":{"numFound":0,"start":0,"docs":[]
> },
> "debug":{
> "rawquerystring":"deeper",
> "querystring":"deeper",
> "parsedquery":"logtext:deeper",
> "parsedquery_toString":"logtext:deeper",
> "explain":{},
> "QParser":"LuceneQParser",
> "timing":{
> "time":1.0,
> "prepare":{
> "time":0.0,
> "query":{
> "time":0.0},
> "facet":{
> "time":0.0},
> "facet_module":{
> "time":0.0},
> "mlt":{
> "time":0.0},
> "highlight":{
> "time":0.0},
> "stats":{
> "time":0.0},
> "expand":{
> "time":0.0},
> "debug":{
> "time":0.0}},
> "process":{
> "time":0.0,
> "query":{
> "time":0.0},
> "facet":{
> "time":0.0},
> "facet_module":{
> "time":0.0},
> "mlt":{
> "time":0.0},
> "highlight":{
> "time":0.0},
> "stats":{
> "time":0.0},
> "expand":{
> "time":0.0},
> "debug":{
> "time":0.0}}}}}
>
>
> http://localhost:8983/solr/EventLog/select?q=*deeper*&wt=json&indent=true&debugQuery=true
>
> {
> "responseHeader":{
> "status":0,
> "QTime":460,
> "params":{
> "q":"*deeper*",
> "indent":"true",
> "wt":"json",
> "debugQuery":"true"}},
>
> "response":{"numFound":45,"start":0,"docs":[
> {
> "id":"2012-07-10 13:23:39.0",
> "category":[16],
> "logtext":["\nHydromet Coordination Message\nOhio River Forecast
> Center, Wilmington, OH\n923 AM EDT Tuesday, July 10, 2012\n\nVery slow
> moving front has sagged down to the southernmost portion of the\nOhio
> Valley. This will keep the axis of convection along or south of the \nTN/KY
> border today and tomorrow, though some very light showers are \npossible in
> the northwest portion of the basin. On Thursday increased \nsoutherly flow
> over the Ohio Valley will begin to draw deeper moisture\nfarther north into
> the basin, but this will mainly be after the 48-hour\nforecast
> cutoff.\n\nDay 1 (8am EDT Tuesday - 8am EDT Wednesday):\nRain is forecast
> in southern Kentucky, southern West Virginia, middle\nTennessee and far
> western Virginia. Basin average amounts increase to the\nsouth with come
> areas approaching an inch. Light amounts less than 0.10 inch\nare expected
> in portions of central Indiana and Ohio. \n\nDay 2 (8am EDT Wednesday - 8am
> EDT Thursday): \nRain is forecast all areas south of the Ohio River as well
> as eastern \nIllinois, southern Indiana and southwest Pennsylvania. Basin
> average amounts\nincrease to the southwest with areas southwest of
> Nashville expecting \nover an inch. \n\nQPF from OHRFC, HPC, et al., can be
> seen at weather.gov/ohrfc/Forecast.php\n$$\nFor
> <http://weather.gov/ohrfc/Forecast.php%5Cn$$%5CnFor> critical after-hours
> support, the OHRFC cell number is 937-725-2222.\nLink Crawford "],
> "username":["crawford"],
> "_version_":1512928764746530816},
> {
> "id":"2012-07-10 17:39:09.0",
> "category":[16],
> "logtext":["\nHydromet Coordination Message\nOhio River Forecast
> Center, Wilmington, OH\n139 PM EDT Tuesday, July 10, 2012\n\n18Z
> Discussion:\nMade some changes to the first 6-hour period of the QPF, but
> otherwise made\nno changes to the previous issuance.\n\nPrevious Discussion
> (12Z):\nVery slow moving front has sagged down to the southernmost portion
> of the\nOhio Valley. This will keep the axis of convection along or south
> of the \nTN/KY border today and tomorrow, though some very light showers
> are \npossible in the northwest portion of the basin. On Thursday increased
> \nsoutherly flow over the Ohio Valley will begin to draw deeper
> moisture\nfarther north into the basin, but this will mainly be after the
> 48-hour\nforecast cutoff.\n\nDay 1 (8am EDT Tuesday - 8am EDT
> Wednesday):\nRain is forecast in southern Kentucky, southern West Virginia,
> middle\nTennessee and far western Virginia. Basin average amounts increase
> to the\nsouth with come areas approaching an inch. Light amounts less than
> 0.10 inch\nare expected in portions of central Indiana and Ohio. \n\nDay 2
> (8am EDT Wednesday - 8am EDT Thursday): \nRain is forecast all areas south
> of the Ohio River as well as eastern \nIllinois, southern Indiana and
> southwest Pennsylvania. Basin average amounts\nincrease to the southwest
> with areas southwest of Nashville expecting \nover an inch. \n\nQPF from
> OHRFC, HPC, et al., can be seen at weather.gov/ohrfc/Forecast.php\n$$\nFor
> <http://weather.gov/ohrfc/Forecast.php%5Cn$$%5CnFor> critical after-hours
> support, the OHRFC cell number is 937-725-2222.\nLink Crawford "],
> "username":["crawford"],
> "_version_":1512928764769599488},
> {
> "id":"2012-07-11 12:39:56.0",
> "category":[16],
> "logtext":["\nHydromet Coordination Message\nOhio River Forecast
> Center, Wilmington, OH\n839 AM EDT Wednesday, July 11, 2012\n\nOHRFC QPF
> Discussion (12Z):\n\nDewpoints in the upper 60's and 70's will help fuel
> showers and thunderstorms \nacross the southern third of the basin today.
> This deeper moisture will\nslowly ooze northward this week as the Bermuda
> ridge gradually strengthens,\nthus placing the basin under the influence of
> a more southwestly flow. QPF\nis relatively tricky today as models remain
> in disagreement how far\nnorth/west the deeper moisture will traverse. We
> biased our QPF more toward\nthe Euro which keeps it farther south than all
> other models. Despite this\nmodel being an outlier, it goes along with the
> prevailing trend. \n\nDay 1 (8am EDT Wednesday - 8am EDT
> Thursday):\nShowers and storms are forecast across the southern third of
> the basin, and\nto a lesser extent in the higher terrain of WV and
> southwest PA. Parts of\nmiddle TN could top 1\" basin-average. \n\nDay 2
> (8am EDT Thursday - 8am EDT Friday): \nShowers and storms will develop
> farther north than Day 1, into parts of\neastern IL, southern and central
> IN, southern and central OH, and much of\nWV. Convection is also forecast
> to continue across areas south of the Ohio\nRiver. Again, heaviest amounts
> are forecast in middle TN and parts of\nsouthern KY with over 1\"
> basin-average. \n\nQPF from OHRFC, HPC, et al., can be seen at
> weather.gov/ohrfc/Forecast.php\n$$\nFor
> <http://weather.gov/ohrfc/Forecast.php%5Cn$$%5CnFor> critical after-hours
> support, the OHRFC cell number is 937-725-2222.\njm "],
> "username":["myers"],
> "_version_":1512928764810493952},
> {
> "id":"2012-07-11 18:18:56.0",
> "category":[16],
> "logtext":["\nHydromet Coordination Message\nOhio River Forecast
> Center, Wilmington, OH\n218 PM EDT Wednesday, July 11, 2012\n\n18Z Update:
> Made some minor changes to the first period to account for \nconvection
> firing across the southern and eastern basin this afternoon.\nOtherwise,
> besides tagging on an additional period Friday, no changes were\nmade to
> the forecast. \n\n\nPrevious Discussion (12Z):\n\nDewpoints in the upper
> 60's and 70's will help fuel showers and thunderstorms \nacross the
> southern third of the basin today. This deeper moisture will\nslowly ooze
> northward this week as the Bermuda ridge gradually strengthens,\nthus
> placing the basin under the influence of a more southwestly flow. QPF\nis
> relatively tricky today as models remain in disagreement how
> far\nnorth/west the deeper moisture will traverse. We biased our QPF more
> toward\nthe Euro which keeps it farther south than all other models.
> Despite this\nmodel being an outlier, it goes along with the prevailing
> trend. \n\nDay 1 (8am EDT Wednesday - 8am EDT Thursday):\nShowers and
> storms are forecast across the southern third of the basin, and\nto a
> lesser extent in the higher terrain of WV and southwest PA. Parts
> of\nmiddle TN could top 1\" basin-average. \n\nDay 2 (8am EDT Thursday -
> 8am EDT Friday): \nShowers and storms will develop farther north than Day
> 1, into parts of\neastern IL, southern and central IN, southern and central
> OH, and much of\nWV. Convection is also forecast to continue across areas
> south of the Ohio\nRiver. Again, heaviest amounts are forecast in middle TN
> and parts of\nsouthern KY with over 1\" basin-average. \n\nQPF from OHRFC,
> HPC, et al., can be seen at weather.gov/ohrfc/Forecast.php\n$$\nFor
> <http://weather.gov/ohrfc/Forecast.php%5Cn$$%5CnFor> critical after-hours
> support, the OHRFC cell number is 937-725-2222."],
> "username":["oper"],
> "_version_":1512928764835659776},
> {
> "id":"2012-07-11 23:41:50.0",
> "category":[16],
> "logtext":["\nHydromet Coordination Message\nOhio River Forecast
> Center, Wilmington, OH\n741 PM EDT Wednesday, July 11, 2012\n\nDiscussion
> (00Z):\n\nMoist air across the southern basin will keep showers and
> thunderstorms in\nthe forecast. The deeper moisture will move farther
> north as southerly \nflow returns to the Ohio Valley. Stuck close to HPC
> this evening with\nminor adjustments for radar trends in the first 6 hr
> period. Also cut \nback slightly on max amounts across the SW. \n\nDay 1
> (8pm EDT Wednesday - 8pm EDT Thursday):\nPrecip amounts up to 1.25\" are
> forecast across middle TN. Amounts taper \nto around 0.10\" along the Ohio
> River.\n\nDay 2 (8pm EDT Thursday - 8pm EDT Friday):\nPrecip is forecast
> for almost the entire basin. The highest forecast\namounts are across
> western KY...where up to 1.25\" is forecast. Amounts \ntaper to 0.00\"
> across the NE.\n\nQPF from OHRFC, HPC, et al., can be seen at
> weather.gov/ohrfc/Forecast.php\n$$\nFor
> <http://weather.gov/ohrfc/Forecast.php%5Cn$$%5CnFor> critical after-hours
> support, the OHRFC cell number is 937-725-2222.\n "],
> "username":["oper"],
> "_version_":1512928764890185728},
> {
> "id":"2012-07-12 12:45:36.0",
> "category":[16],
> "logtext":["\nHydromet Coordination Message\nOhio River Forecast
> Center, Wilmington, OH\n845 AM EDT Thursday, July 12, 2012\n\nOHRFC QPF
> Discussion (12Z):\n\nA stubborn front over the southern basin will continue
> to produce unsettled\nweather mainly south of the Ohio River today and
> tonight. A strengthening\nBermuda ridge will nudge the deeper moisture
> farther north and west Friday\nand into the weekend. A nearly stationary
> surface trough will interact with\nthe excessive moisture to generate
> showers and thunderstorms across a good \nportion of the middle Ohio Valley
> during this time period. Due to the high \namount of uncertainty, we are
> going pretty conservative with our QPF amounts. \nWe favored a SREF/Euro
> blend, but made plenty of additional tweaks to cut \nback the precip over
> the drier northwest part of the basin. In general, our \nQPF amounts are
> about half that of HPC throughout much of the basin. \n\nOHRFC 48-hr QPF
> Amounts ending 12Z Saturday: \n\nDay 1 (8am EDT Thursday - 8am EDT
> Friday):\nUnsettled weather will continue mainly along and south of the
> Ohio River. \nHeaviest amounts are forecast across much of KY and middle TN
> with basin\naverage amounts between 0.50\" to 1.50\". More isolated
> activity is expected\nnorth of the Ohio. \n\nDay 2 (8am EDT Friday - 8am
> EDT Saturday):\nThe axis of heavier precipitation will shift northward into
> north-central\nKY, far southern IL, southern IN, southern OH, northern and
> western WV, and\nsouthwest PA. Basin-average amounts should range from
> 0.25\" to 0.75\". \n\n\nQPF from OHRFC, HPC, et al., can be seen at
> weather.gov/ohrfc/Forecast.php\n$$\nFor
> <http://weather.gov/ohrfc/Forecast.php%5Cn$$%5CnFor> critical after-hours
> support, the OHRFC cell number is 937-725-2222.\njm "],
> "username":["myers"],
> "_version_":1512928764967780352},
> {
> "id":"2012-07-12 18:02:33.0",
> "category":[16],
> "logtext":["\nHydromet Coordination Message\nOhio River Forecast
> Center, Wilmington, OH\n202 PM EDT Thursday, July 12, 2012\n\n18Z Update:
> New 18Z HPC QPF package appears to be a bit more conservative. We \nblended
> our first period toward their new guidance and made some additional
> \ntweaks for current radar trends/convection. Otherwise, no changes
> were\nneeded at this time. \n\nPrevious QPF Discussion (12Z):\n\nA stubborn
> front over the southern basin will continue to produce unsettled\nweather
> mainly south of the Ohio River today and tonight. A strengthening\nBermuda
> ridge will nudge the deeper moisture farther north and west Friday\nand
> into the weekend. A nearly stationary surface trough will interact
> with\nthe excessive moisture to generate showers and thunderstorms across a
> good \nportion of the middle Ohio Valley during this time period. Due to
> the high \namount of uncertainty, we are going pretty conservative with our
> QPF amounts. \nWe favored a SREF/Euro blend, but made plenty of additional
> tweaks to cut \nback the precip over the drier northwest part of the basin.
> In general, our \nQPF amounts are about half that of HPC throughout much of
> the basin. \n\nOHRFC 48-hr QPF Amounts ending 12Z Saturday: \n\nDay 1 (8am
> EDT Thursday - 8am EDT Friday):\nUnsettled weather will continue mainly
> along and south of the Ohio River. \nHeaviest amounts are forecast across
> much of KY and middle TN with basin\naverage amounts between 0.50\" to
> 1.50\". More isolated activity is expected\nnorth of the Ohio. \n\nDay 2
> (8am EDT Friday - 8am EDT Saturday):\nThe axis of heavier precipitation
> will shift northward into north-central\nKY, far southern IL, southern IN,
> southern OH, northern and western WV, and\nsouthwest PA. Basin-average
> amounts should range from 0.25\" to 0.75\". \n\nQPF from OHRFC, HPC, et
> al., can be seen at weather.gov/ohrfc/Forecast.php\n$$\nFor
> <http://weather.gov/ohrfc/Forecast.php%5Cn$$%5CnFor> critical after-hours
> support, the OHRFC cell number is 937-725-2222.\njm "],
> "username":["myers"],
> "_version_":1512928765003431936},
> {
> "id":"2009-07-29 13:13:20.0",
> "category":[16],
> "logtext":["ZCZC CRWHCMTIR CES\nTTAA00 KTIR DDHHMM\n\n...FOR
> INTERGOVERNMENTAL AGENCY USE ONLY...\n\nHydromet Coordination Message\nOhio
> River Forecast Center, Wilmington, OH\n913 AM EDT Wednesday, July 29,
> 2009\n\nTo: OHRFC WFOs\nFrom: OHRFC\n\nA cold front slowly working its
> way across the basin will result in more\nunsettled weather for much of the
> Ohio Valley today. Higher QPF totals are\nforecast across the eastern half
> of the region where the deeper moisture\nresides. A sizable break in the
> action is anticipated tonight before the\nnext wave moves into the western
> basin on Thursday. Overall, our forecast\nis very similar to HPC's QPF
> except for the usual radar based adjustments\nin the first period.
> \n\nOHRFC QPF Amounts through 12Z Thursday:\nA half inch or more is
> forecast for most of the eastern half of the Ohio\nValley, with amounts
> over an inch likely for the higher terrain. Generally\naround a quarter of
> an inch or less is forecast for the western side of the\nbasin.
> \n\n\n$$\njm \nNNNN"],
> "username":["myers"],
> "_version_":1512928766474584064},
> {
> "id":"2010-08-28 11:04:33.0",
> "category":[16],
> "logtext":["TTAA00 KTIR DDHHMM\n\n...FOR INTERGOVERNMENTAL AGENCY
> USE ONLY...\n\nHydromet Coordination Message\nOhio River Forecast Center,
> Wilmington, OH\n704 AM EDT Saturday, August 28, 2010\n\nSome isolated
> showers are possible again over the New River basin and in\nmiddle
> Tennessee this afternoon. Otherwise, the Ohio Valley will remain
> dry\ntoday. On Sunday, the ridge axis will shift slightly eastward, just
> enough\nto allow deeper moisture to sneak into the southwestern Ohio
> Valley. \nOur QPF reflects this increased chance of showers and
> thunderstorms over\nmiddle TN, western KY, eastern IL, and far southwest
> IN. \n \n\n$$\nFor critical after-hours support, the OHRFC cell number is
> 937-725-2222.\njm "],
> "username":["myers"],
> "_version_":1512928767572443136},
> {
> "id":"2011-05-06 12:44:03.0",
> "category":[16],
> "logtext":["\nHydromet Coordination Message\nOhio River Forecast
> Center, Wilmington, OH\n844 AM EDT Friday, May 6, 2011\n\nTO: OHRFC
> HSAs\nFROM: OHRFC \n\n***The Ohio River Forecast Center remains on 24 hour
> operations***\n\nOHRFC QPF Discussion:\n\nMild-mannered system to work its
> way across the Ohio Valley today with\nweak cold front currently pushing
> into western Ohio and middle Kentucky.\nThere will be a brief break after
> this system before the Saturday \nsystem moves into the western portion of
> the basin. This second system\nfeatures slightly deeper moisture, but still
> rainfall amounts should be\nmodest. Utilized HPC's guidance for the QPF
> today, with the exception\nof adjustments to the first period to account
> current rainfall trends.\n\nOHRFC QPF Forecast through 8am EDT
> Sunday:\n\nDay 1: 8am EDT Friday through 8am EDT Saturday:\nRain is
> forecast for most of the Ohio Valley. Amounts will generally be\n0.10 inch
> or less.\n\nDay 2: 8am EDT Saturday through 8am EDT Sunday:\nRain is
> expected basinwide. Most areas will see amounts in the 0.25 \nto 0.50 inch
> range with amounts up to 0.75 inch in southeast Kentucky.\n\nQPF from
> OHRFC, HPC, et al., can be seen at weather.gov/ohrfc/Forecast.php\n$$\nFor
> <http://weather.gov/ohrfc/Forecast.php%5Cn$$%5CnFor> critical after-hours
> support, the OHRFC cell number is 937-725-2222.\nLink Crawford "],
> "username":["crawford"],
> "_version_":1512928767842975744}]
> },
> "debug":{
> "rawquerystring":"*deeper*",
> "querystring":"*deeper*",
> "parsedquery":"logtext:*deeper*",
> "parsedquery_toString":"logtext:*deeper*",
> "explain":{
> "2012-07-10 13:23:39.0":"\n1.0 = logtext:*deeper*, product of:\n
> 1.0 = boost\n 1.0 = queryNorm\n",
> "2012-07-10 17:39:09.0":"\n1.0 = logtext:*deeper*, product of:\n
> 1.0 = boost\n 1.0 = queryNorm\n",
> "2012-07-11 12:39:56.0":"\n1.0 = logtext:*deeper*, product of:\n
> 1.0 = boost\n 1.0 = queryNorm\n",
> "2012-07-11 18:18:56.0":"\n1.0 = logtext:*deeper*, product of:\n
> 1.0 = boost\n 1.0 = queryNorm\n",
> "2012-07-11 23:41:50.0":"\n1.0 = logtext:*deeper*, product of:\n
> 1.0 = boost\n 1.0 = queryNorm\n",
> "2012-07-12 12:45:36.0":"\n1.0 = logtext:*deeper*, product of:\n
> 1.0 = boost\n 1.0 = queryNorm\n",
> "2012-07-12 18:02:33.0":"\n1.0 = logtext:*deeper*, product of:\n
> 1.0 = boost\n 1.0 = queryNorm\n",
> "2009-07-29 13:13:20.0":"\n1.0 = logtext:*deeper*, product of:\n
> 1.0 = boost\n 1.0 = queryNorm\n",
> "2010-08-28 11:04:33.0":"\n1.0 = logtext:*deeper*, product of:\n
> 1.0 = boost\n 1.0 = queryNorm\n",
> "2011-05-06 12:44:03.0":"\n1.0 = logtext:*deeper*, product of:\n
> 1.0 = boost\n 1.0 = queryNorm\n"},
> "QParser":"LuceneQParser",
> "timing":{
> "time":460.0,
> "prepare":{
> "time":8.0,
> "query":{
> "time":8.0},
> "facet":{
> "time":0.0},
> "facet_module":{
> "time":0.0},
> "mlt":{
> "time":0.0},
> "highlight":{
> "time":0.0},
> "stats":{
> "time":0.0},
> "expand":{
> "time":0.0},
> "debug":{
> "time":0.0}},
> "process":{
> "time":451.0,
> "query":{
> "time":0.0},
> "facet":{
> "time":0.0},
> "facet_module":{
> "time":0.0},
> "mlt":{
> "time":0.0},
> "highlight":{
> "time":0.0},
> "stats":{
> "time":0.0},
> "expand":{
> "time":0.0},
> "debug":{
> "time":450.0}}}}}
>
>
>
>
>
>
> On 9/23/2015 8:54 AM, Mugeesh Husain wrote:
>
>> Hi Mark,
>>
>> Search is not coming properly becuase you have taken "ELall" field as a
>> text type which is not define properly.
>>
>> you have to modify the schema.xml with these chance.
>>
>>
>> <fieldType name="logtype" class="solr.TextField">
>> <analyzer type="index">
>> <tokenizer class="solr.StandardTokenizerFactory"/>
>> <filter class="solr.StandardFilterFactory"/>
>> </analyzer>
>> </fieldType>
>>
>> <field name="logtext" type="logtype" indexed="true" stored="true"
>> required="true"/>
>> <field name="ELall" type="logtype" indexed="true" stored="true"
>> multiValued="true"/>
>>
>>
>>
>> --
>> View this message in context:
>> http://lucene.472066.n3.nabble.com/query-parsing-tp4230778p4230793.html
>> Sent from the Solr - User mailing list archive at Nabble.com.
>>
>>
>
--
--------------------------
Benedetti Alessandro
Visiting card - http://about.me/alessandro_benedetti
Blog - http://alexbenedetti.blogspot.co.uk
"Tyger, tyger burning bright
In the forests of the night,
What immortal hand or eye
Could frame thy fearful symmetry?"
William Blake - Songs of Experience -1794 England
Re: query parsing
Posted by Mark Fenbers <ma...@noaa.gov>.
Mugeesh, I believe you are on the right path and I was eager to try out
your suggestion. So my schema.xml now contains this snippet (changes
indicated by ~):
<field name="id" type="date" indexed="true" stored="true"
required="true" />
~ <field name="logtext" type="logtype" indexed="true"
stored="true" required="true" />
<field name="category" type="int" indexed="true" stored="true"
required="true" />
<field name="username" type="string" indexed="true" stored="true"
required="true" />
~ <field name="ELall" type="logtype" indexed="true"
stored="true" multiValued="true" />
<copyField source="logtext" dest="ELall" />
<copyField source="username" dest="ELall" />
~ <fieldType name="logtype" class="solr.TextField">
~ <analyzer type="index">
~ <tokenizer class="solr.StandardTokenizerFactory" />
~ <filter class="solr.StandardFilterFactory" />
~ </analyzer>
~ </fieldType>
but my results are the same -- that my search yields 0 results unless I
wrap the search word with asterisks.
Alessandro, below are the results (with and without the asterisks) with
debug turned on. I don't know what much of the debug info means. Is it
giving you more clues?
http://localhost:8983/solr/EventLog/select?q=deeper&wt=json&indent=true&debugQuery=true
{
"responseHeader":{
"status":0,
"QTime":2,
"params":{
"q":"deeper",
"indent":"true",
"wt":"json",
"debugQuery":"true"}},
"response":{"numFound":0,"start":0,"docs":[]
},
"debug":{
"rawquerystring":"deeper",
"querystring":"deeper",
"parsedquery":"logtext:deeper",
"parsedquery_toString":"logtext:deeper",
"explain":{},
"QParser":"LuceneQParser",
"timing":{
"time":1.0,
"prepare":{
"time":0.0,
"query":{
"time":0.0},
"facet":{
"time":0.0},
"facet_module":{
"time":0.0},
"mlt":{
"time":0.0},
"highlight":{
"time":0.0},
"stats":{
"time":0.0},
"expand":{
"time":0.0},
"debug":{
"time":0.0}},
"process":{
"time":0.0,
"query":{
"time":0.0},
"facet":{
"time":0.0},
"facet_module":{
"time":0.0},
"mlt":{
"time":0.0},
"highlight":{
"time":0.0},
"stats":{
"time":0.0},
"expand":{
"time":0.0},
"debug":{
"time":0.0}}}}}
http://localhost:8983/solr/EventLog/select?q=*deeper*&wt=json&indent=true&debugQuery=true
{
"responseHeader":{
"status":0,
"QTime":460,
"params":{
"q":"*deeper*",
"indent":"true",
"wt":"json",
"debugQuery":"true"}},
"response":{"numFound":45,"start":0,"docs":[
{
"id":"2012-07-10 13:23:39.0",
"category":[16],
"logtext":["\nHydromet Coordination Message\nOhio River
Forecast Center, Wilmington, OH\n923 AM EDT Tuesday, July 10,
2012\n\nVery slow moving front has sagged down to the southernmost
portion of the\nOhio Valley. This will keep the axis of convection along
or south of the \nTN/KY border today and tomorrow, though some very
light showers are \npossible in the northwest portion of the basin. On
Thursday increased \nsoutherly flow over the Ohio Valley will begin to
draw deeper moisture\nfarther north into the basin, but this will mainly
be after the 48-hour\nforecast cutoff.\n\nDay 1 (8am EDT Tuesday - 8am
EDT Wednesday):\nRain is forecast in southern Kentucky, southern West
Virginia, middle\nTennessee and far western Virginia. Basin average
amounts increase to the\nsouth with come areas approaching an inch.
Light amounts less than 0.10 inch\nare expected in portions of central
Indiana and Ohio. \n\nDay 2 (8am EDT Wednesday - 8am EDT Thursday):
\nRain is forecast all areas south of the Ohio River as well as eastern
\nIllinois, southern Indiana and southwest Pennsylvania. Basin average
amounts\nincrease to the southwest with areas southwest of Nashville
expecting \nover an inch. \n\nQPF from OHRFC, HPC, et al., can be seen
at weather.gov/ohrfc/Forecast.php\n$$\nFor critical after-hours support,
the OHRFC cell number is 937-725-2222.\nLink Crawford "],
"username":["crawford"],
"_version_":1512928764746530816},
{
"id":"2012-07-10 17:39:09.0",
"category":[16],
"logtext":["\nHydromet Coordination Message\nOhio River
Forecast Center, Wilmington, OH\n139 PM EDT Tuesday, July 10,
2012\n\n18Z Discussion:\nMade some changes to the first 6-hour period of
the QPF, but otherwise made\nno changes to the previous
issuance.\n\nPrevious Discussion (12Z):\nVery slow moving front has
sagged down to the southernmost portion of the\nOhio Valley. This will
keep the axis of convection along or south of the \nTN/KY border today
and tomorrow, though some very light showers are \npossible in the
northwest portion of the basin. On Thursday increased \nsoutherly flow
over the Ohio Valley will begin to draw deeper moisture\nfarther north
into the basin, but this will mainly be after the 48-hour\nforecast
cutoff.\n\nDay 1 (8am EDT Tuesday - 8am EDT Wednesday):\nRain is
forecast in southern Kentucky, southern West Virginia, middle\nTennessee
and far western Virginia. Basin average amounts increase to the\nsouth
with come areas approaching an inch. Light amounts less than 0.10
inch\nare expected in portions of central Indiana and Ohio. \n\nDay 2
(8am EDT Wednesday - 8am EDT Thursday): \nRain is forecast all areas
south of the Ohio River as well as eastern \nIllinois, southern Indiana
and southwest Pennsylvania. Basin average amounts\nincrease to the
southwest with areas southwest of Nashville expecting \nover an inch.
\n\nQPF from OHRFC, HPC, et al., can be seen at
weather.gov/ohrfc/Forecast.php\n$$\nFor critical after-hours support,
the OHRFC cell number is 937-725-2222.\nLink Crawford "],
"username":["crawford"],
"_version_":1512928764769599488},
{
"id":"2012-07-11 12:39:56.0",
"category":[16],
"logtext":["\nHydromet Coordination Message\nOhio River
Forecast Center, Wilmington, OH\n839 AM EDT Wednesday, July 11,
2012\n\nOHRFC QPF Discussion (12Z):\n\nDewpoints in the upper 60's and
70's will help fuel showers and thunderstorms \nacross the southern
third of the basin today. This deeper moisture will\nslowly ooze
northward this week as the Bermuda ridge gradually strengthens,\nthus
placing the basin under the influence of a more southwestly flow.
QPF\nis relatively tricky today as models remain in disagreement how
far\nnorth/west the deeper moisture will traverse. We biased our QPF
more toward\nthe Euro which keeps it farther south than all other
models. Despite this\nmodel being an outlier, it goes along with the
prevailing trend. \n\nDay 1 (8am EDT Wednesday - 8am EDT
Thursday):\nShowers and storms are forecast across the southern third of
the basin, and\nto a lesser extent in the higher terrain of WV and
southwest PA. Parts of\nmiddle TN could top 1\" basin-average. \n\nDay 2
(8am EDT Thursday - 8am EDT Friday): \nShowers and storms will develop
farther north than Day 1, into parts of\neastern IL, southern and
central IN, southern and central OH, and much of\nWV. Convection is also
forecast to continue across areas south of the Ohio\nRiver. Again,
heaviest amounts are forecast in middle TN and parts of\nsouthern KY
with over 1\" basin-average. \n\nQPF from OHRFC, HPC, et al., can be
seen at weather.gov/ohrfc/Forecast.php\n$$\nFor critical after-hours
support, the OHRFC cell number is 937-725-2222.\njm "],
"username":["myers"],
"_version_":1512928764810493952},
{
"id":"2012-07-11 18:18:56.0",
"category":[16],
"logtext":["\nHydromet Coordination Message\nOhio River
Forecast Center, Wilmington, OH\n218 PM EDT Wednesday, July 11,
2012\n\n18Z Update: Made some minor changes to the first period to
account for \nconvection firing across the southern and eastern basin
this afternoon.\nOtherwise, besides tagging on an additional period
Friday, no changes were\nmade to the forecast. \n\n\nPrevious Discussion
(12Z):\n\nDewpoints in the upper 60's and 70's will help fuel showers
and thunderstorms \nacross the southern third of the basin today. This
deeper moisture will\nslowly ooze northward this week as the Bermuda
ridge gradually strengthens,\nthus placing the basin under the influence
of a more southwestly flow. QPF\nis relatively tricky today as models
remain in disagreement how far\nnorth/west the deeper moisture will
traverse. We biased our QPF more toward\nthe Euro which keeps it farther
south than all other models. Despite this\nmodel being an outlier, it
goes along with the prevailing trend. \n\nDay 1 (8am EDT Wednesday - 8am
EDT Thursday):\nShowers and storms are forecast across the southern
third of the basin, and\nto a lesser extent in the higher terrain of WV
and southwest PA. Parts of\nmiddle TN could top 1\" basin-average.
\n\nDay 2 (8am EDT Thursday - 8am EDT Friday): \nShowers and storms will
develop farther north than Day 1, into parts of\neastern IL, southern
and central IN, southern and central OH, and much of\nWV. Convection is
also forecast to continue across areas south of the Ohio\nRiver. Again,
heaviest amounts are forecast in middle TN and parts of\nsouthern KY
with over 1\" basin-average. \n\nQPF from OHRFC, HPC, et al., can be
seen at weather.gov/ohrfc/Forecast.php\n$$\nFor critical after-hours
support, the OHRFC cell number is 937-725-2222."],
"username":["oper"],
"_version_":1512928764835659776},
{
"id":"2012-07-11 23:41:50.0",
"category":[16],
"logtext":["\nHydromet Coordination Message\nOhio River
Forecast Center, Wilmington, OH\n741 PM EDT Wednesday, July 11,
2012\n\nDiscussion (00Z):\n\nMoist air across the southern basin will
keep showers and thunderstorms in\nthe forecast. The deeper moisture
will move farther north as southerly \nflow returns to the Ohio Valley.
Stuck close to HPC this evening with\nminor adjustments for radar trends
in the first 6 hr period. Also cut \nback slightly on max amounts
across the SW. \n\nDay 1 (8pm EDT Wednesday - 8pm EDT
Thursday):\nPrecip amounts up to 1.25\" are forecast across middle TN.
Amounts taper \nto around 0.10\" along the Ohio River.\n\nDay 2 (8pm
EDT Thursday - 8pm EDT Friday):\nPrecip is forecast for almost the
entire basin. The highest forecast\namounts are across western
KY...where up to 1.25\" is forecast. Amounts \ntaper to 0.00\" across
the NE.\n\nQPF from OHRFC, HPC, et al., can be seen at
weather.gov/ohrfc/Forecast.php\n$$\nFor critical after-hours support,
the OHRFC cell number is 937-725-2222.\n "],
"username":["oper"],
"_version_":1512928764890185728},
{
"id":"2012-07-12 12:45:36.0",
"category":[16],
"logtext":["\nHydromet Coordination Message\nOhio River
Forecast Center, Wilmington, OH\n845 AM EDT Thursday, July 12,
2012\n\nOHRFC QPF Discussion (12Z):\n\nA stubborn front over the
southern basin will continue to produce unsettled\nweather mainly south
of the Ohio River today and tonight. A strengthening\nBermuda ridge will
nudge the deeper moisture farther north and west Friday\nand into the
weekend. A nearly stationary surface trough will interact with\nthe
excessive moisture to generate showers and thunderstorms across a good
\nportion of the middle Ohio Valley during this time period. Due to the
high \namount of uncertainty, we are going pretty conservative with our
QPF amounts. \nWe favored a SREF/Euro blend, but made plenty of
additional tweaks to cut \nback the precip over the drier northwest part
of the basin. In general, our \nQPF amounts are about half that of HPC
throughout much of the basin. \n\nOHRFC 48-hr QPF Amounts ending 12Z
Saturday: \n\nDay 1 (8am EDT Thursday - 8am EDT Friday):\nUnsettled
weather will continue mainly along and south of the Ohio River.
\nHeaviest amounts are forecast across much of KY and middle TN with
basin\naverage amounts between 0.50\" to 1.50\". More isolated activity
is expected\nnorth of the Ohio. \n\nDay 2 (8am EDT Friday - 8am EDT
Saturday):\nThe axis of heavier precipitation will shift northward into
north-central\nKY, far southern IL, southern IN, southern OH, northern
and western WV, and\nsouthwest PA. Basin-average amounts should range
from 0.25\" to 0.75\". \n\n\nQPF from OHRFC, HPC, et al., can be seen at
weather.gov/ohrfc/Forecast.php\n$$\nFor critical after-hours support,
the OHRFC cell number is 937-725-2222.\njm "],
"username":["myers"],
"_version_":1512928764967780352},
{
"id":"2012-07-12 18:02:33.0",
"category":[16],
"logtext":["\nHydromet Coordination Message\nOhio River
Forecast Center, Wilmington, OH\n202 PM EDT Thursday, July 12,
2012\n\n18Z Update: New 18Z HPC QPF package appears to be a bit more
conservative. We \nblended our first period toward their new guidance
and made some additional \ntweaks for current radar trends/convection.
Otherwise, no changes were\nneeded at this time. \n\nPrevious QPF
Discussion (12Z):\n\nA stubborn front over the southern basin will
continue to produce unsettled\nweather mainly south of the Ohio River
today and tonight. A strengthening\nBermuda ridge will nudge the deeper
moisture farther north and west Friday\nand into the weekend. A nearly
stationary surface trough will interact with\nthe excessive moisture to
generate showers and thunderstorms across a good \nportion of the middle
Ohio Valley during this time period. Due to the high \namount of
uncertainty, we are going pretty conservative with our QPF amounts. \nWe
favored a SREF/Euro blend, but made plenty of additional tweaks to cut
\nback the precip over the drier northwest part of the basin. In
general, our \nQPF amounts are about half that of HPC throughout much of
the basin. \n\nOHRFC 48-hr QPF Amounts ending 12Z Saturday: \n\nDay 1
(8am EDT Thursday - 8am EDT Friday):\nUnsettled weather will continue
mainly along and south of the Ohio River. \nHeaviest amounts are
forecast across much of KY and middle TN with basin\naverage amounts
between 0.50\" to 1.50\". More isolated activity is expected\nnorth of
the Ohio. \n\nDay 2 (8am EDT Friday - 8am EDT Saturday):\nThe axis of
heavier precipitation will shift northward into north-central\nKY, far
southern IL, southern IN, southern OH, northern and western WV,
and\nsouthwest PA. Basin-average amounts should range from 0.25\" to
0.75\". \n\nQPF from OHRFC, HPC, et al., can be seen at
weather.gov/ohrfc/Forecast.php\n$$\nFor critical after-hours support,
the OHRFC cell number is 937-725-2222.\njm "],
"username":["myers"],
"_version_":1512928765003431936},
{
"id":"2009-07-29 13:13:20.0",
"category":[16],
"logtext":["ZCZC CRWHCMTIR CES\nTTAA00 KTIR DDHHMM\n\n...FOR
INTERGOVERNMENTAL AGENCY USE ONLY...\n\nHydromet Coordination
Message\nOhio River Forecast Center, Wilmington, OH\n913 AM EDT
Wednesday, July 29, 2009\n\nTo: OHRFC WFOs\nFrom: OHRFC\n\nA cold
front slowly working its way across the basin will result in
more\nunsettled weather for much of the Ohio Valley today. Higher QPF
totals are\nforecast across the eastern half of the region where the
deeper moisture\nresides. A sizable break in the action is anticipated
tonight before the\nnext wave moves into the western basin on Thursday.
Overall, our forecast\nis very similar to HPC's QPF except for the usual
radar based adjustments\nin the first period. \n\nOHRFC QPF Amounts
through 12Z Thursday:\nA half inch or more is forecast for most of the
eastern half of the Ohio\nValley, with amounts over an inch likely for
the higher terrain. Generally\naround a quarter of an inch or less is
forecast for the western side of the\nbasin. \n\n\n$$\njm \nNNNN"],
"username":["myers"],
"_version_":1512928766474584064},
{
"id":"2010-08-28 11:04:33.0",
"category":[16],
"logtext":["TTAA00 KTIR DDHHMM\n\n...FOR INTERGOVERNMENTAL
AGENCY USE ONLY...\n\nHydromet Coordination Message\nOhio River Forecast
Center, Wilmington, OH\n704 AM EDT Saturday, August 28, 2010\n\nSome
isolated showers are possible again over the New River basin and
in\nmiddle Tennessee this afternoon. Otherwise, the Ohio Valley will
remain dry\ntoday. On Sunday, the ridge axis will shift slightly
eastward, just enough\nto allow deeper moisture to sneak into the
southwestern Ohio Valley. \nOur QPF reflects this increased chance of
showers and thunderstorms over\nmiddle TN, western KY, eastern IL, and
far southwest IN. \n \n\n$$\nFor critical after-hours support, the OHRFC
cell number is 937-725-2222.\njm "],
"username":["myers"],
"_version_":1512928767572443136},
{
"id":"2011-05-06 12:44:03.0",
"category":[16],
"logtext":["\nHydromet Coordination Message\nOhio River
Forecast Center, Wilmington, OH\n844 AM EDT Friday, May 6, 2011\n\nTO:
OHRFC HSAs\nFROM: OHRFC \n\n***The Ohio River Forecast Center remains
on 24 hour operations***\n\nOHRFC QPF Discussion:\n\nMild-mannered
system to work its way across the Ohio Valley today with\nweak cold
front currently pushing into western Ohio and middle Kentucky.\nThere
will be a brief break after this system before the Saturday \nsystem
moves into the western portion of the basin. This second
system\nfeatures slightly deeper moisture, but still rainfall amounts
should be\nmodest. Utilized HPC's guidance for the QPF today, with the
exception\nof adjustments to the first period to account current
rainfall trends.\n\nOHRFC QPF Forecast through 8am EDT Sunday:\n\nDay 1:
8am EDT Friday through 8am EDT Saturday:\nRain is forecast for most of
the Ohio Valley. Amounts will generally be\n0.10 inch or less.\n\nDay
2: 8am EDT Saturday through 8am EDT Sunday:\nRain is expected
basinwide. Most areas will see amounts in the 0.25 \nto 0.50 inch range
with amounts up to 0.75 inch in southeast Kentucky.\n\nQPF from OHRFC,
HPC, et al., can be seen at weather.gov/ohrfc/Forecast.php\n$$\nFor
critical after-hours support, the OHRFC cell number is
937-725-2222.\nLink Crawford "],
"username":["crawford"],
"_version_":1512928767842975744}]
},
"debug":{
"rawquerystring":"*deeper*",
"querystring":"*deeper*",
"parsedquery":"logtext:*deeper*",
"parsedquery_toString":"logtext:*deeper*",
"explain":{
"2012-07-10 13:23:39.0":"\n1.0 = logtext:*deeper*, product of:\n
1.0 = boost\n 1.0 = queryNorm\n",
"2012-07-10 17:39:09.0":"\n1.0 = logtext:*deeper*, product of:\n
1.0 = boost\n 1.0 = queryNorm\n",
"2012-07-11 12:39:56.0":"\n1.0 = logtext:*deeper*, product of:\n
1.0 = boost\n 1.0 = queryNorm\n",
"2012-07-11 18:18:56.0":"\n1.0 = logtext:*deeper*, product of:\n
1.0 = boost\n 1.0 = queryNorm\n",
"2012-07-11 23:41:50.0":"\n1.0 = logtext:*deeper*, product of:\n
1.0 = boost\n 1.0 = queryNorm\n",
"2012-07-12 12:45:36.0":"\n1.0 = logtext:*deeper*, product of:\n
1.0 = boost\n 1.0 = queryNorm\n",
"2012-07-12 18:02:33.0":"\n1.0 = logtext:*deeper*, product of:\n
1.0 = boost\n 1.0 = queryNorm\n",
"2009-07-29 13:13:20.0":"\n1.0 = logtext:*deeper*, product of:\n
1.0 = boost\n 1.0 = queryNorm\n",
"2010-08-28 11:04:33.0":"\n1.0 = logtext:*deeper*, product of:\n
1.0 = boost\n 1.0 = queryNorm\n",
"2011-05-06 12:44:03.0":"\n1.0 = logtext:*deeper*, product of:\n
1.0 = boost\n 1.0 = queryNorm\n"},
"QParser":"LuceneQParser",
"timing":{
"time":460.0,
"prepare":{
"time":8.0,
"query":{
"time":8.0},
"facet":{
"time":0.0},
"facet_module":{
"time":0.0},
"mlt":{
"time":0.0},
"highlight":{
"time":0.0},
"stats":{
"time":0.0},
"expand":{
"time":0.0},
"debug":{
"time":0.0}},
"process":{
"time":451.0,
"query":{
"time":0.0},
"facet":{
"time":0.0},
"facet_module":{
"time":0.0},
"mlt":{
"time":0.0},
"highlight":{
"time":0.0},
"stats":{
"time":0.0},
"expand":{
"time":0.0},
"debug":{
"time":450.0}}}}}
On 9/23/2015 8:54 AM, Mugeesh Husain wrote:
> Hi Mark,
>
> Search is not coming properly becuase you have taken "ELall" field as a
> text type which is not define properly.
>
> you have to modify the schema.xml with these chance.
>
>
> <fieldType name="logtype" class="solr.TextField">
> <analyzer type="index">
> <tokenizer class="solr.StandardTokenizerFactory"/>
> <filter class="solr.StandardFilterFactory"/>
> </analyzer>
> </fieldType>
>
>
> <field name="logtext" type="logtype" indexed="true" stored="true"
> required="true"/>
> <field name="ELall" type="logtype" indexed="true" stored="true"
> multiValued="true"/>
>
>
>
> --
> View this message in context: http://lucene.472066.n3.nabble.com/query-parsing-tp4230778p4230793.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
Re: query parsing
Posted by Mugeesh Husain <mu...@gmail.com>.
Hi Mark,
Search is not coming properly becuase you have taken "ELall" field as a
text type which is not define properly.
you have to modify the schema.xml with these chance.
<fieldType name="logtype" class="solr.TextField">
<analyzer type="index">
<tokenizer class="solr.StandardTokenizerFactory"/>
<filter class="solr.StandardFilterFactory"/>
</analyzer>
</fieldType>
<field name="logtext" type="logtype" indexed="true" stored="true"
required="true"/>
<field name="ELall" type="logtype" indexed="true" stored="true"
multiValued="true"/>
--
View this message in context: http://lucene.472066.n3.nabble.com/query-parsing-tp4230778p4230793.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: query parsing
Posted by Alessandro Benedetti <be...@gmail.com>.
If you go to the Analysis tool, indexing and query time , what can you see
for your "deeper" query text and your field content ?
( using the log text field ) ?
Have you verified the current tokens in the index for that field ?
I quickly went through your config files, and they look ok, but it is quite
weird what happens to you.
Can you please run the query with debugQuery=true and post the results ?
Cheers
2015-09-23 12:57 GMT+01:00 Mark Fenbers <ma...@noaa.gov>:
> When I submit this:
>
> http://localhost:8983/solr/EventLog/select?q=deeper&wt=json&indent=true
>
> then I get these (empty) results:
> {
> "responseHeader":{
> "status":0,
> "QTime":1,
> "params":{
> "q":"deeper",
> "indent":"true",
> "wt":"json"}},
> "response":{"numFound":0,"start":0,"docs":[]
> }}
>
> However, if I add asterisks before *and *after "deeper", like this:
>
> http://localhost:8983/solr/EventLog/select?q=*deeper*&wt=json&indent=true
>
> then I get the correct set of results (shown below), as I expect. What am
> I doing wrong that the query requires leading and trailing asterisks to
> work correctly? If I search on existing text in the username field instead
> of the default logtext field, then I don't need to use the asterisks to get
> correct results. Does this mean I have a problem in my indexing process
> when I used /dataimport. Or does it mean I have something wrong in my query?
>
> Also, notice in the results that category, logtext, and username fields
> are returned as arrays, even though I do not include multiValued="true" in
> the schema.xml definition. Why? Attached are my solrconfig.xml and
> schema.xml. Any insights would be appreciated!
>
> thanks,
> Mark
>
> {
> "responseHeader":{
> "status":0,
> "QTime":9,
> "params":{
> "q":"*deeper*",
> "indent":"true",
> "wt":"json"}},
> "response":{"numFound":45,"start":0,"docs":[
> {
> "id":"2012-07-10 13:23:39.0",
> "category":[16],
> "logtext":["\nHydromet Coordination Message\nOhio River Forecast
> Center, Wilmington, OH\n923 AM EDT Tuesday, July 10, 2012\n\nVery slow
> moving front has sagged down to the southernmost portion of the\nOhio
> Valley. This will keep the axis of convection along or south of the \nTN/KY
> border today and tomorrow, though some very light showers are \npossible in
> the northwest portion of the basin. On Thursday increased \nsoutherly flow
> over the Ohio Valley will begin to draw deeper moisture\nfarther north into
> the basin, but this will mainly be after the 48-hour\nforecast
> cutoff.\n\nDay 1 (8am EDT Tuesday - 8am EDT Wednesday):\nRain is forecast
> in southern Kentucky, southern West Virginia, middle\nTennessee and far
> western Virginia. Basin average amounts increase to the\nsouth with come
> areas approaching an inch. Light amounts less than 0.10 inch\nare expected
> in portions of central Indiana and Ohio. \n\nDay 2 (8am EDT Wednesday - 8am
> EDT Thursday): \nRain is forecast all areas south of the Ohio River as well
> as eastern \nIllinois, southern Indiana and southwest Pennsylvania. Basin
> average amounts\nincrease to the southwest with areas southwest of
> Nashville expecting \nover an inch. \n\nQPF from OHRFC, HPC, et al., can be
> seen at weather.gov/ohrfc/Forecast.php\n$$\nFor
> <http://weather.gov/ohrfc/Forecast.php%5Cn$$%5CnFor> critical after-hours
> support, the OHRFC cell number is 937-725-2222.\nLink Crawford "],
> "username":["crawford"],
> "_version_":1512928764746530816},
> {
> "id":"2012-07-10 17:39:09.0",
> "category":[16],
> "logtext":["\nHydromet Coordination Message\nOhio River Forecast
> Center, Wilmington, OH\n139 PM EDT Tuesday, July 10, 2012\n\n18Z
> Discussion:\nMade some changes to the first 6-hour period of the QPF, but
> otherwise made\nno changes to the previous issuance.\n\nPrevious Discussion
> (12Z):\nVery slow moving front has sagged down to the southernmost portion
> of the\nOhio Valley. This will keep the axis of convection along or south
> of the \nTN/KY border today and tomorrow, though some very light showers
> are \npossible in the northwest portion of the basin. On Thursday increased
> \nsoutherly flow over the Ohio Valley will begin to draw deeper
> moisture\nfarther north into the basin, but this will mainly be after the
> 48-hour\nforecast cutoff.\n\nDay 1 (8am EDT Tuesday - 8am EDT
> Wednesday):\nRain is forecast in southern Kentucky, southern West Virginia,
> middle\nTennessee and far western Virginia. Basin average amounts increase
> to the\nsouth with come areas approaching an inch. Light amounts less than
> 0.10 inch\nare expected in portions of central Indiana and Ohio. \n\nDay 2
> (8am EDT Wednesday - 8am EDT Thursday): \nRain is forecast all areas south
> of the Ohio River as well as eastern \nIllinois, southern Indiana and
> southwest Pennsylvania. Basin average amounts\nincrease to the southwest
> with areas southwest of Nashville expecting \nover an inch. \n\nQPF from
> OHRFC, HPC, et al., can be seen at weather.gov/ohrfc/Forecast.php\n$$\nFor
> <http://weather.gov/ohrfc/Forecast.php%5Cn$$%5CnFor> critical after-hours
> support, the OHRFC cell number is 937-725-2222.\nLink Crawford "],
> "username":["crawford"],
> "_version_":1512928764769599488},
> {
> "id":"2012-07-11 12:39:56.0",
> "category":[16],
> "logtext":["\nHydromet Coordination Message\nOhio River Forecast
> Center, Wilmington, OH\n839 AM EDT Wednesday, July 11, 2012\n\nOHRFC QPF
> Discussion (12Z):\n\nDewpoints in the upper 60's and 70's will help fuel
> showers and thunderstorms \nacross the southern third of the basin today.
> This deeper moisture will\nslowly ooze northward this week as the Bermuda
> ridge gradually strengthens,\nthus placing the basin under the influence of
> a more southwestly flow. QPF\nis relatively tricky today as models remain
> in disagreement how far\nnorth/west the deeper moisture will traverse. We
> biased our QPF more toward\nthe Euro which keeps it farther south than all
> other models. Despite this\nmodel being an outlier, it goes along with the
> prevailing trend. \n\nDay 1 (8am EDT Wednesday - 8am EDT
> Thursday):\nShowers and storms are forecast across the southern third of
> the basin, and\nto a lesser extent in the higher terrain of WV and
> southwest PA. Parts of\nmiddle TN could top 1\" basin-average. \n\nDay 2
> (8am EDT Thursday - 8am EDT Friday): \nShowers and storms will develop
> farther north than Day 1, into parts of\neastern IL, southern and central
> IN, southern and central OH, and much of\nWV. Convection is also forecast
> to continue across areas south of the Ohio\nRiver. Again, heaviest amounts
> are forecast in middle TN and parts of\nsouthern KY with over 1\"
> basin-average. \n\nQPF from OHRFC, HPC, et al., can be seen at
> weather.gov/ohrfc/Forecast.php\n$$\nFor
> <http://weather.gov/ohrfc/Forecast.php%5Cn$$%5CnFor> critical after-hours
> support, the OHRFC cell number is 937-725-2222.\njm "],
> "username":["myers"],
> "_version_":1512928764810493952},
> {
> "id":"2012-07-11 18:18:56.0",
> "category":[16],
> "logtext":["\nHydromet Coordination Message\nOhio River Forecast
> Center, Wilmington, OH\n218 PM EDT Wednesday, July 11, 2012\n\n18Z Update:
> Made some minor changes to the first period to account for \nconvection
> firing across the southern and eastern basin this afternoon.\nOtherwise,
> besides tagging on an additional period Friday, no changes were\nmade to
> the forecast. \n\n\nPrevious Discussion (12Z):\n\nDewpoints in the upper
> 60's and 70's will help fuel showers and thunderstorms \nacross the
> southern third of the basin today. This deeper moisture will\nslowly ooze
> northward this week as the Bermuda ridge gradually strengthens,\nthus
> placing the basin under the influence of a more southwestly flow. QPF\nis
> relatively tricky today as models remain in disagreement how
> far\nnorth/west the deeper moisture will traverse. We biased our QPF more
> toward\nthe Euro which keeps it farther south than all other models.
> Despite this\nmodel being an outlier, it goes along with the prevailing
> trend. \n\nDay 1 (8am EDT Wednesday - 8am EDT Thursday):\nShowers and
> storms are forecast across the southern third of the basin, and\nto a
> lesser extent in the higher terrain of WV and southwest PA. Parts
> of\nmiddle TN could top 1\" basin-average. \n\nDay 2 (8am EDT Thursday -
> 8am EDT Friday): \nShowers and storms will develop farther north than Day
> 1, into parts of\neastern IL, southern and central IN, southern and central
> OH, and much of\nWV. Convection is also forecast to continue across areas
> south of the Ohio\nRiver. Again, heaviest amounts are forecast in middle TN
> and parts of\nsouthern KY with over 1\" basin-average. \n\nQPF from OHRFC,
> HPC, et al., can be seen at weather.gov/ohrfc/Forecast.php\n$$\nFor
> <http://weather.gov/ohrfc/Forecast.php%5Cn$$%5CnFor> critical after-hours
> support, the OHRFC cell number is 937-725-2222."],
> "username":["oper"],
> "_version_":1512928764835659776},
> {
> "id":"2012-07-11 23:41:50.0",
> "category":[16],
> "logtext":["\nHydromet Coordination Message\nOhio River Forecast
> Center, Wilmington, OH\n741 PM EDT Wednesday, July 11, 2012\n\nDiscussion
> (00Z):\n\nMoist air across the southern basin will keep showers and
> thunderstorms in\nthe forecast. The deeper moisture will move farther
> north as southerly \nflow returns to the Ohio Valley. Stuck close to HPC
> this evening with\nminor adjustments for radar trends in the first 6 hr
> period. Also cut \nback slightly on max amounts across the SW. \n\nDay 1
> (8pm EDT Wednesday - 8pm EDT Thursday):\nPrecip amounts up to 1.25\" are
> forecast across middle TN. Amounts taper \nto around 0.10\" along the Ohio
> River.\n\nDay 2 (8pm EDT Thursday - 8pm EDT Friday):\nPrecip is forecast
> for almost the entire basin. The highest forecast\namounts are across
> western KY...where up to 1.25\" is forecast. Amounts \ntaper to 0.00\"
> across the NE.\n\nQPF from OHRFC, HPC, et al., can be seen at
> weather.gov/ohrfc/Forecast.php\n$$\nFor
> <http://weather.gov/ohrfc/Forecast.php%5Cn$$%5CnFor> critical after-hours
> support, the OHRFC cell number is 937-725-2222.\n "],
> "username":["oper"],
> "_version_":1512928764890185728},
> {
> "id":"2012-07-12 12:45:36.0",
> "category":[16],
> "logtext":["\nHydromet Coordination Message\nOhio River Forecast
> Center, Wilmington, OH\n845 AM EDT Thursday, July 12, 2012\n\nOHRFC QPF
> Discussion (12Z):\n\nA stubborn front over the southern basin will continue
> to produce unsettled\nweather mainly south of the Ohio River today and
> tonight. A strengthening\nBermuda ridge will nudge the deeper moisture
> farther north and west Friday\nand into the weekend. A nearly stationary
> surface trough will interact with\nthe excessive moisture to generate
> showers and thunderstorms across a good \nportion of the middle Ohio Valley
> during this time period. Due to the high \namount of uncertainty, we are
> going pretty conservative with our QPF amounts. \nWe favored a SREF/Euro
> blend, but made plenty of additional tweaks to cut \nback the precip over
> the drier northwest part of the basin. In general, our \nQPF amounts are
> about half that of HPC throughout much of the basin. \n\nOHRFC 48-hr QPF
> Amounts ending 12Z Saturday: \n\nDay 1 (8am EDT Thursday - 8am EDT
> Friday):\nUnsettled weather will continue mainly along and south of the
> Ohio River. \nHeaviest amounts are forecast across much of KY and middle TN
> with basin\naverage amounts between 0.50\" to 1.50\". More isolated
> activity is expected\nnorth of the Ohio. \n\nDay 2 (8am EDT Friday - 8am
> EDT Saturday):\nThe axis of heavier precipitation will shift northward into
> north-central\nKY, far southern IL, southern IN, southern OH, northern and
> western WV, and\nsouthwest PA. Basin-average amounts should range from
> 0.25\" to 0.75\". \n\n\nQPF from OHRFC, HPC, et al., can be seen at
> weather.gov/ohrfc/Forecast.php\n$$\nFor
> <http://weather.gov/ohrfc/Forecast.php%5Cn$$%5CnFor> critical after-hours
> support, the OHRFC cell number is 937-725-2222.\njm "],
> "username":["myers"],
> "_version_":1512928764967780352},
> {
> "id":"2012-07-12 18:02:33.0",
> "category":[16],
> "logtext":["\nHydromet Coordination Message\nOhio River Forecast
> Center, Wilmington, OH\n202 PM EDT Thursday, July 12, 2012\n\n18Z Update:
> New 18Z HPC QPF package appears to be a bit more conservative. We \nblended
> our first period toward their new guidance and made some additional
> \ntweaks for current radar trends/convection. Otherwise, no changes
> were\nneeded at this time. \n\nPrevious QPF Discussion (12Z):\n\nA stubborn
> front over the southern basin will continue to produce unsettled\nweather
> mainly south of the Ohio River today and tonight. A strengthening\nBermuda
> ridge will nudge the deeper moisture farther north and west Friday\nand
> into the weekend. A nearly stationary surface trough will interact
> with\nthe excessive moisture to generate showers and thunderstorms across a
> good \nportion of the middle Ohio Valley during this time period. Due to
> the high \namount of uncertainty, we are going pretty conservative with our
> QPF amounts. \nWe favored a SREF/Euro blend, but made plenty of additional
> tweaks to cut \nback the precip over the drier northwest part of the basin.
> In general, our \nQPF amounts are about half that of HPC throughout much of
> the basin. \n\nOHRFC 48-hr QPF Amounts ending 12Z Saturday: \n\nDay 1 (8am
> EDT Thursday - 8am EDT Friday):\nUnsettled weather will continue mainly
> along and south of the Ohio River. \nHeaviest amounts are forecast across
> much of KY and middle TN with basin\naverage amounts between 0.50\" to
> 1.50\". More isolated activity is expected\nnorth of the Ohio. \n\nDay 2
> (8am EDT Friday - 8am EDT Saturday):\nThe axis of heavier precipitation
> will shift northward into north-central\nKY, far southern IL, southern IN,
> southern OH, northern and western WV, and\nsouthwest PA. Basin-average
> amounts should range from 0.25\" to 0.75\". \n\nQPF from OHRFC, HPC, et
> al., can be seen at weather.gov/ohrfc/Forecast.php\n$$\nFor
> <http://weather.gov/ohrfc/Forecast.php%5Cn$$%5CnFor> critical after-hours
> support, the OHRFC cell number is 937-725-2222.\njm "],
> "username":["myers"],
> "_version_":1512928765003431936},
> {
> "id":"2009-07-29 13:13:20.0",
> "category":[16],
> "logtext":["ZCZC CRWHCMTIR CES\nTTAA00 KTIR DDHHMM\n\n...FOR
> INTERGOVERNMENTAL AGENCY USE ONLY...\n\nHydromet Coordination Message\nOhio
> River Forecast Center, Wilmington, OH\n913 AM EDT Wednesday, July 29,
> 2009\n\nTo: OHRFC WFOs\nFrom: OHRFC\n\nA cold front slowly working its
> way across the basin will result in more\nunsettled weather for much of the
> Ohio Valley today. Higher QPF totals are\nforecast across the eastern half
> of the region where the deeper moisture\nresides. A sizable break in the
> action is anticipated tonight before the\nnext wave moves into the western
> basin on Thursday. Overall, our forecast\nis very similar to HPC's QPF
> except for the usual radar based adjustments\nin the first period.
> \n\nOHRFC QPF Amounts through 12Z Thursday:\nA half inch or more is
> forecast for most of the eastern half of the Ohio\nValley, with amounts
> over an inch likely for the higher terrain. Generally\naround a quarter of
> an inch or less is forecast for the western side of the\nbasin.
> \n\n\n$$\njm \nNNNN"],
> "username":["myers"],
> "_version_":1512928766474584064},
> {
> "id":"2010-08-28 11:04:33.0",
> "category":[16],
> "logtext":["TTAA00 KTIR DDHHMM\n\n...FOR INTERGOVERNMENTAL AGENCY
> USE ONLY...\n\nHydromet Coordination Message\nOhio River Forecast Center,
> Wilmington, OH\n704 AM EDT Saturday, August 28, 2010\n\nSome isolated
> showers are possible again over the New River basin and in\nmiddle
> Tennessee this afternoon. Otherwise, the Ohio Valley will remain
> dry\ntoday. On Sunday, the ridge axis will shift slightly eastward, just
> enough\nto allow deeper moisture to sneak into the southwestern Ohio
> Valley. \nOur QPF reflects this increased chance of showers and
> thunderstorms over\nmiddle TN, western KY, eastern IL, and far southwest
> IN. \n \n\n$$\nFor critical after-hours support, the OHRFC cell number is
> 937-725-2222.\njm "],
> "username":["myers"],
> "_version_":1512928767572443136},
> {
> "id":"2011-05-06 12:44:03.0",
> "category":[16],
> "logtext":["\nHydromet Coordination Message\nOhio River Forecast
> Center, Wilmington, OH\n844 AM EDT Friday, May 6, 2011\n\nTO: OHRFC
> HSAs\nFROM: OHRFC \n\n***The Ohio River Forecast Center remains on 24 hour
> operations***\n\nOHRFC QPF Discussion:\n\nMild-mannered system to work its
> way across the Ohio Valley today with\nweak cold front currently pushing
> into western Ohio and middle Kentucky.\nThere will be a brief break after
> this system before the Saturday \nsystem moves into the western portion of
> the basin. This second system\nfeatures slightly deeper moisture, but still
> rainfall amounts should be\nmodest. Utilized HPC's guidance for the QPF
> today, with the exception\nof adjustments to the first period to account
> current rainfall trends.\n\nOHRFC QPF Forecast through 8am EDT
> Sunday:\n\nDay 1: 8am EDT Friday through 8am EDT Saturday:\nRain is
> forecast for most of the Ohio Valley. Amounts will generally be\n0.10 inch
> or less.\n\nDay 2: 8am EDT Saturday through 8am EDT Sunday:\nRain is
> expected basinwide. Most areas will see amounts in the 0.25 \nto 0.50 inch
> range with amounts up to 0.75 inch in southeast Kentucky.\n\nQPF from
> OHRFC, HPC, et al., can be seen at weather.gov/ohrfc/Forecast.php\n$$\nFor
> <http://weather.gov/ohrfc/Forecast.php%5Cn$$%5CnFor> critical after-hours
> support, the OHRFC cell number is 937-725-2222.\nLink Crawford "],
> "username":["crawford"],
> "_version_":1512928767842975744}]
> }}
>
>
>
--
--------------------------
Benedetti Alessandro
Visiting card - http://about.me/alessandro_benedetti
Blog - http://alexbenedetti.blogspot.co.uk
"Tyger, tyger burning bright
In the forests of the night,
What immortal hand or eye
Could frame thy fearful symmetry?"
William Blake - Songs of Experience -1794 England