Populating a custom Solr field with text extracted from document

[Jan <ja...@dsto.defence.gov.au>]

Hi all,

I am a new Solr user, and would like to create a new custom field that is
then populated with text extracted from each document when I crawl my file
system.

For example, I have a bunch of text documents that contain the following
format:
text text text... Received : 04 Jan 2002 17:31:40 ...text text text

I would like to store the value "04 Jan 2002 17:31:40" in my custom field.
The new field has already been created in schema.xml as follows:
<field indexed="true" multiValued="false" name="received" omitNorms="true"
omitPositions="true" omitTermFreqAndPositions="true" stored="true"
termVectors="false" type="text_en" />

I am unsure how to populate this field during the crawl of my data sources.
I looked into analyzers and tokenizers 
http://wiki.apache.org/solr/SolrPlugins#Fields here , and also looked at
SolrCell/ExtractingRequestHandler but from what I understand neither of
these are the correct solution.

Can anyone give me some guidance? Apologies if this has already been
answered elsewhere.

Thank you,
Jan

--
View this message in context: http://lucene.472066.n3.nabble.com/Populating-a-custom-Solr-field-with-text-extracted-from-document-tp3514857p3514857.html
Sent from the Lucene - General mailing list archive at Nabble.com.

Re: Populating a custom Solr field with text extracted from document

[Jan <ja...@dsto.defence.gov.au>]

Thank you for your reply.

: what are you using to do the crawling?

I'm using Solr within LucidWorks Enterprise. As far as I know LucidWorks
provides a default crawler called Aperture so this is what I'm using.

Thank you also for describing a few of the options to tackle the problem. I
did consider writing some custom parsing code, but wanted to explore
existing options first rather than re-inventing the wheel. I've tinkered
with curl a bit and think that POSTing to Solr may be a suitable approach.

--
View this message in context: http://lucene.472066.n3.nabble.com/Populating-a-custom-Solr-field-with-text-extracted-from-document-tp3514857p3541066.html
Sent from the Lucene - General mailing list archive at Nabble.com.

Re: Populating a custom Solr field with text extracted from document

[Chris Hostetter <ho...@fucit.org>]

: I am a new Solr user, and would like to create a new custom field that is
: then populated with text extracted from each document when I crawl my file
: system.

what are you using to do the crawling?

Typically people feed solr structured data -- there are some things in 
Solr (like the ExtractingRequestHandler) that help you pull structure out 
of unstructured or semi-structured files, and there are things like DIH 
that can help you pull data from structure (or semi-structured) sources, 
but those aren't end-all-be-all solutions to all problems -- they aim to 
meet the 80/20 rule of simple common cases.

If you have special requirements about parsing special files...

: text text text... Received : 04 Jan 2002 17:31:40 ...text text text

...you'll need to write your own special code for parsing those files to 
extract the structure you want.

where/how you use your custom code depends on your use cases -- maybe you 
write a custom extractor for Tika nad then use ExtractingRequestHandler, 
maybe you write a custom EntityProcessor and then use DataImportHandler, 
or maybe you just parse the code in the client langauge of your choice and 
POST it to Solr over HTTP ... it all depends on your use case and what you 
are comfortable with.

BTW: Since you definitely seem to interested in using Solr, you should 
consider sending subsequent questions to the solr-user@lucene mailing list 
(general@lucene is generally for discussions about hte overall Lucene 
project, and/or questions when people really have no idea what they want 
to use)

-Hoss