You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by "Peter A. Kirk" <pk...@alpha-solutions.dk> on 2010/09/14 01:00:42 UTC

Field names

Hi

is it possible to issue a query to solr, to get a list which contains all the field names in the index?

What about to get a list of the freqency of individual words in each field?

thanks,
Peter

Re: solr.DateField: org.apache.solr.common.SolrException: Error while creating field

Posted by Erick Erickson <er...@gmail.com>.
If you're using Javas SimpleDateFormat, try enclosing your Z in the format
string with single quotes, like:
SimpleDateFormat sdf = new SimpleDateFormat("yyyy-MM-dd'T'HH:mm:ss'Z'");

HTH
Erick

On Tue, Sep 14, 2010 at 4:54 PM, h00kpublic@gmail.com <
h00kpublic@googlemail.com> wrote:

>  hi... i am using solr for indexing local files (solrj) and indexing
> crawled nutch-documents...
> i have configured the fieldtype
>
> <fieldType name="date" class="solr.DateField" omitNorms="true"/>
>
> and use this type by field
>
> <field name="metadata_last_modified" type="date" indexed="true"
> stored="true"/>
>
> the pattern for date is how described in DateField.java:
>
>    yyyy-MM-dd'T'HH:mm:ssZ
>
> i need this date for sorting my search results... but i always get the
> exception below...
> can someone tell me whats wrong or give an alternative resolution?
>
> best regards marcel :)
>
> SEVERE: org.apache.solr.common.SolrException: Error while creating field
> 'metadata_last_modified{type=date,properties=indexed,stored,omitNorms}' from
> value '2010-09-14T22:29:24+0200'
>    at org.apache.solr.schema.FieldType.createField(FieldType.java:242)
>    at org.apache.solr.schema.SchemaField.createField(SchemaField.java:94)
>    at
> org.apache.solr.update.DocumentBuilder.addField(DocumentBuilder.java:204)
>    at
> org.apache.solr.update.DocumentBuilder.toDocument(DocumentBuilder.java:277)
>    at
> org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:60)
>    at org.apache.solr.handler.XMLLoader.processUpdate(XMLLoader.java:139)
>    at org.apache.solr.handler.XMLLoader.load(XMLLoader.java:69)
>    at
> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:54)
>    at
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131)
>    at org.apache.solr.core.SolrCore.execute(SolrCore.java:1323)
>    at
> org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:337)
>    at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:240)
>    at
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
>    at
> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:388)
>    at
> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>    at
> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>    at
> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765)
>    at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:418)
>    at
> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>    at
> org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114)
>    at
> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>    at org.mortbay.jetty.Server.handle(Server.java:326)
>    at
> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>    at
> org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:938)
>    at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:842)
>    at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:218)
>    at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>    at
> org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228)
>    at
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> Caused by: org.apache.solr.common.SolrException: Invalid Date
> String:'2010-09-14T22:29:24+0200'
>    at org.apache.solr.schema.DateField.parseMath(DateField.java:166)
>    at org.apache.solr.schema.DateField.toInternal(DateField.java:136)
>    at org.apache.solr.schema.FieldType.createField(FieldType.java:240)
>    ... 28 more
>
>

RE: Re: solr.DateField: org.apache.solr.common.SolrException: Error while creating field

Posted by Dennis Gearon <ge...@sbcglobal.net>.
Can you give us a scencario:
   1/ Like a OOP sequence diagram, "Thishappens, that happens, now that"
   2/ Where you see it useful? 

Isn't it possible to convert before storing/after retrieving?

Couldn't a timezone offset (or .... local timezone designation) be stored as a separate field to filter/sort on?

Dennis Gearon

Signature Warning
----------------
EARTH has a Right To Life,
  otherwise we all die.

Read 'Hot, Flat, and Crowded'
Laugh at http://www.yert.com/film.php


--- On Tue, 9/14/10, Markus Jelsma <ma...@buyways.nl> wrote:

> From: Markus Jelsma <ma...@buyways.nl>
> Subject: RE: Re: solr.DateField: org.apache.solr.common.SolrException: Error while creating field
> To: solr-user@lucene.apache.org
> Date: Tuesday, September 14, 2010, 2:18 PM
> It would be a nice feature if Solr
> supports queries with time zone support on an index where
> all times are UTC. There is some chatter about this in
> SOLR-750 but i haven't found an issue that would add support
> for time zone queries.
> 
>  
> 
> Did i do a lousy search or is the issue missing as of yet?
>  
> -----Original message-----
> From: Yonik Seeley <yo...@lucidimagination.com>
> Sent: Tue 14-09-2010 22:58
> To: solr-user@lucene.apache.org;
> 
> Subject: Re: solr.DateField:
> org.apache.solr.common.SolrException: Error while creating
> field
> 
> On Tue, Sep 14, 2010 at 4:54 PM, h00kpublic@gmail.com
> <h0...@googlemail.com>
> wrote:
> > SEVERE: org.apache.solr.common.SolrException: Error
> while creating field
> >
> 'metadata_last_modified{type=date,properties=indexed,stored,omitNorms}'
> from
> > value '2010-09-14T22:29:24+0200'
> 
> Different timezones are currently not allowed - you must
> UTC (hence
> the "Z" timecode).
> 
> -Yonik
> http://lucenerevolution.org  Lucene/Solr Conference,
> Boston Oct 7-8
> 

RE: Re: solr.DateField: org.apache.solr.common.SolrException: Error while creating field

Posted by Markus Jelsma <ma...@buyways.nl>.
It would be a nice feature if Solr supports queries with time zone support on an index where all times are UTC. There is some chatter about this in SOLR-750 but i haven't found an issue that would add support for time zone queries.

 

Did i do a lousy search or is the issue missing as of yet?
 
-----Original message-----
From: Yonik Seeley <yo...@lucidimagination.com>
Sent: Tue 14-09-2010 22:58
To: solr-user@lucene.apache.org; 
Subject: Re: solr.DateField: org.apache.solr.common.SolrException: Error while creating field

On Tue, Sep 14, 2010 at 4:54 PM, h00kpublic@gmail.com
<h0...@googlemail.com> wrote:
> SEVERE: org.apache.solr.common.SolrException: Error while creating field
> 'metadata_last_modified{type=date,properties=indexed,stored,omitNorms}' from
> value '2010-09-14T22:29:24+0200'

Different timezones are currently not allowed - you must UTC (hence
the "Z" timecode).

-Yonik
http://lucenerevolution.org  Lucene/Solr Conference, Boston Oct 7-8

Re: solr.DateField: org.apache.solr.common.SolrException: Error while creating field

Posted by Yonik Seeley <yo...@lucidimagination.com>.
On Tue, Sep 14, 2010 at 4:54 PM, h00kpublic@gmail.com
<h0...@googlemail.com> wrote:
> SEVERE: org.apache.solr.common.SolrException: Error while creating field
> 'metadata_last_modified{type=date,properties=indexed,stored,omitNorms}' from
> value '2010-09-14T22:29:24+0200'

Different timezones are currently not allowed - you must UTC (hence
the "Z" timecode).

-Yonik
http://lucenerevolution.org  Lucene/Solr Conference, Boston Oct 7-8

solr.DateField: org.apache.solr.common.SolrException: Error while creating field

Posted by "h00kpublic@gmail.com" <h0...@googlemail.com>.
  hi... i am using solr for indexing local files (solrj) and indexing 
crawled nutch-documents...
i have configured the fieldtype

<fieldType name="date" class="solr.DateField" omitNorms="true"/>

and use this type by field

<field name="metadata_last_modified" type="date" indexed="true" 
stored="true"/>

the pattern for date is how described in DateField.java:

     yyyy-MM-dd'T'HH:mm:ssZ

i need this date for sorting my search results... but i always get the 
exception below...
can someone tell me whats wrong or give an alternative resolution?

best regards marcel :)

SEVERE: org.apache.solr.common.SolrException: Error while creating field 
'metadata_last_modified{type=date,properties=indexed,stored,omitNorms}' 
from value '2010-09-14T22:29:24+0200'
     at org.apache.solr.schema.FieldType.createField(FieldType.java:242)
     at org.apache.solr.schema.SchemaField.createField(SchemaField.java:94)
     at 
org.apache.solr.update.DocumentBuilder.addField(DocumentBuilder.java:204)
     at 
org.apache.solr.update.DocumentBuilder.toDocument(DocumentBuilder.java:277)
     at 
org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:60)
     at org.apache.solr.handler.XMLLoader.processUpdate(XMLLoader.java:139)
     at org.apache.solr.handler.XMLLoader.load(XMLLoader.java:69)
     at 
org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:54)
     at 
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131)
     at org.apache.solr.core.SolrCore.execute(SolrCore.java:1323)
     at 
org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:337)
     at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:240)
     at 
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
     at 
org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:388)
     at 
org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
     at 
org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
     at 
org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765)
     at 
org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:418)
     at 
org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
     at 
org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114)
     at 
org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
     at org.mortbay.jetty.Server.handle(Server.java:326)
     at 
org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
     at 
org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:938)
     at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:842)
     at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:218)
     at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
     at 
org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228)
     at 
org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
Caused by: org.apache.solr.common.SolrException: Invalid Date 
String:'2010-09-14T22:29:24+0200'
     at org.apache.solr.schema.DateField.parseMath(DateField.java:166)
     at org.apache.solr.schema.DateField.toInternal(DateField.java:136)
     at org.apache.solr.schema.FieldType.createField(FieldType.java:240)
     ... 28 more


RE: Field names

Posted by "Peter A. Kirk" <pk...@alpha-solutions.dk>.
From: Simon Willnauer [simon.willnauer@googlemail.com]
Sent: Tuesday, 14 September 2010 17:47
To: solr-user@lucene.apache.org
Subject: Re: Field names

>On Tue, Sep 14, 2010 at 1:39 AM, Peter A. Kirk <pk...@alpha-solutions.dk> wrote:
>> <result name="response" numFound="9" start="0">
>>  <doc>
>>
>> So it only finds 9?
>
>Since the "gb" term says 18 occurrences throughout the index I suspect
>you updated you docs once without optimizing or indexing a lot of docs
>so that segments are merged. Try to call optimize if you can afford it
>and see if the doc-freq count goes back to 9

Yes - thanks!

Re: Field names

Posted by Simon Willnauer <si...@googlemail.com>.
On Tue, Sep 14, 2010 at 1:39 AM, Peter A. Kirk <pk...@alpha-solutions.dk> wrote:
> Fantastic - that is exactly what I was looking for!
>
> But here is one thing I don't undertstand:
>
> If I call the url:
> http://localhost:8983/solr/admin/luke?numTerms=10&fl=name
>
> Some of the result looks like:
>
> <lst name="fields">
>  <lst name="name">
>    <lst name="topTerms">
>      <int name="gb">18</int>
>
> Does this mean that the term "gb" occurs 18 times in the name field?
Yes that is the Doc Frequency of the term "gb". Remember that deleted
/ updated documents and their terms contribute to the doc frequency
until they are expunged from the index. That either happens through a
segment merge in the background or due to an explicit call to
optimize.
>
> Because if I issue this search:
> http://localhost:8983/solr/select/?q=name:gb
>
> I get results like:
> <result name="response" numFound="9" start="0">
>  <doc>
>
> So it only finds 9?
Since the "gb" term says 18 occurrences throughout the index I suspect
you updated you docs once without optimizing or indexing a lot of docs
so that segments are merged. Try to call optimize if you can afford it
and see if the doc-freq count goes back to 9

simon
>
> What do the above results actually tell me?
>
> Thanks,
> Peter
>
> ________________________________________
> From: Ryan McKinley [ryantxu@gmail.com]
> Sent: Tuesday, 14 September 2010 11:30
> To: solr-user@lucene.apache.org
> Subject: Re: Field names
>
> check:
> http://wiki.apache.org/solr/LukeRequestHandler
>
>
>
> On Mon, Sep 13, 2010 at 7:00 PM, Peter A. Kirk <pk...@alpha-solutions.dk> wrote:
>> Hi
>>
>> is it possible to issue a query to solr, to get a list which contains all the field names in the index?
>>
>> What about to get a list of the freqency of individual words in each field?
>>
>> thanks,
>> Peter
>>

RE: Field names

Posted by "Peter A. Kirk" <pk...@alpha-solutions.dk>.
Fantastic - that is exactly what I was looking for!

But here is one thing I don't undertstand:

If I call the url:
http://localhost:8983/solr/admin/luke?numTerms=10&fl=name

Some of the result looks like:

<lst name="fields">
  <lst name="name">
    <lst name="topTerms">
      <int name="gb">18</int> 

Does this mean that the term "gb" occurs 18 times in the name field?

Because if I issue this search:
http://localhost:8983/solr/select/?q=name:gb

I get results like:
<result name="response" numFound="9" start="0">
  <doc>

So it only finds 9?

What do the above results actually tell me?

Thanks,
Peter

________________________________________
From: Ryan McKinley [ryantxu@gmail.com]
Sent: Tuesday, 14 September 2010 11:30
To: solr-user@lucene.apache.org
Subject: Re: Field names

check:
http://wiki.apache.org/solr/LukeRequestHandler



On Mon, Sep 13, 2010 at 7:00 PM, Peter A. Kirk <pk...@alpha-solutions.dk> wrote:
> Hi
>
> is it possible to issue a query to solr, to get a list which contains all the field names in the index?
>
> What about to get a list of the freqency of individual words in each field?
>
> thanks,
> Peter
>

Re: Field names

Posted by Ryan McKinley <ry...@gmail.com>.
check:
http://wiki.apache.org/solr/LukeRequestHandler



On Mon, Sep 13, 2010 at 7:00 PM, Peter A. Kirk <pk...@alpha-solutions.dk> wrote:
> Hi
>
> is it possible to issue a query to solr, to get a list which contains all the field names in the index?
>
> What about to get a list of the freqency of individual words in each field?
>
> thanks,
> Peter
>