You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by jb <jb...@shi-gmbh.com> on 2011/09/26 15:56:17 UTC
Re: email - DIH
Hi Alonso, Gora,
I run in the same Problem with the MailEntityProcessor.
I have an Email-Folder called "Test". Inside there a "only" two messages.
When I run the DIH everything looks find, except that the two Emails doesn't
get indexed.
Are there any adidtional informations to this problem?
I'm using Solr 3.4.0 (earlier Version the same problem)
Here my config:
<dataConfig>
<document>
<entity name="email" transformer="TemplateTransformer"
processor="MailEntityProcessor" user="shi@zahn-gmbh.de" password="SHI-Test"
host="mail.zahn-gmbh.de" protocol="imap" folders="*"
fetchMailsSince="2000-01-01 00:00:00"
deltaFetch="false" processAttachement="false" batchSize="100"
fetchSize="1024" recurse="true" >
<field column="id" template="email-${email.messageId}"/>
<field column="quelle" template="Email"/>
<field column="title" template="${email.subject}"/>
<field column="author" template="${email.from}"/>
<field column="last_modified" template="${email.sentDate}"
dateTimeFormat="yyyy-MM-dd hh:mm:ss"/>
<field column="text" template="${email.content}"/>
<field column="content_type" template="Email"/>
<field column="quelle" template="Comunigate"/>
<field column="doctype" template="Email"/>
</entity>
</document>
</dataConfig>
And here my response (using the command:
"http://localhost:8080/apache-solr-3.4.0/dataimport-mail?command=full-import&commit=true"):
26.09.2011 15:52:53 org.apache.solr.core.SolrCore execute
INFO: [] webapp=/apache-solr-3.4.0 path=/dataimport-mail
params={commit=true&command=full-import} status=0 QTime=16
26.09.2011 15:52:53 org.apache.solr.handler.dataimport.DataImporter
doFullImport
INFO: Starting Full Import
26.09.2011 15:52:53 org.apache.solr.handler.dataimport.SolrWriter
readIndexerProperties
INFO: Read dataimport-mail.properties
26.09.2011 15:52:53 org.apache.solr.update.DirectUpdateHandler2 deleteAll
INFO: [] REMOVING ALL DOCUMENTS FROM INDEX
26.09.2011 15:52:53 org.apache.solr.core.SolrDeletionPolicy onInit
INFO: SolrDeletionPolicy.onInit: commits:num=1
commit{dir=H:\_Projekt.lfd\zahn\solr_home_34\data\index,segFN=segments_4,version=1317035795833,generation=4,filenames=[segments_4]
26.09.2011 15:52:53 org.apache.solr.core.SolrDeletionPolicy updateCommits
INFO: newest commit = 1317035795833
26.09.2011 15:52:53 org.apache.solr.handler.dataimport.MailEntityProcessor
logConfig
INFO: user : shi@zahn-gmbh.de
pwd : SHI-Test
protocol : imap
host : mail.zahn-gmbh.de
folders : Test
recurse : true
exclude : []
include : []
batchSize : 20
fetchSize : 1024
read timeout : 60000
conection timeout : 30000
custom filter :
fetch mail since : Sat Jan 01 00:00:00 CET 2000
26.09.2011 15:52:54 org.apache.solr.update.DirectUpdateHandler2 commit
INFO: start
commit(optimize=false,waitFlush=true,waitSearcher=true,expungeDeletes=false)
26.09.2011 15:52:54 org.apache.solr.core.SolrDeletionPolicy onCommit
INFO: SolrDeletionPolicy.onCommit: commits:num=2
commit{dir=H:\_Projekt.lfd\zahn\solr_home_34\data\index,segFN=segments_4,version=1317035795833,generation=4,filenames=[segments_4]
commit{dir=H:\_Projekt.lfd\zahn\solr_home_34\data\index,segFN=segments_5,version=1317035795834,generation=5,filenames=[segments_5]
26.09.2011 15:52:54 org.apache.solr.core.SolrDeletionPolicy updateCommits
INFO: newest commit = 1317035795834
26.09.2011 15:52:54 org.apache.solr.search.SolrIndexSearcher <init>
INFO: Opening Searcher@17af46e main
26.09.2011 15:52:54 org.apache.solr.search.SolrIndexSearcher warm
INFO: autowarming Searcher@17af46e main from Searcher@5e8d7d main
fieldValueCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=1,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0,item_doctype={field=doctype,memSize=4224,tindexSize=32,time=0,phase1=0,nTerms=0,bigTerms=0,termInstances=0,uses=2}}
26.09.2011 15:52:54 org.apache.solr.update.DirectUpdateHandler2 commit
INFO: end_commit_flush
26.09.2011 15:52:54 org.apache.solr.search.SolrIndexSearcher warm
INFO: autowarming result for Searcher@17af46e main
fieldValueCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
26.09.2011 15:52:54 org.apache.solr.search.SolrIndexSearcher warm
INFO: autowarming Searcher@17af46e main from Searcher@5e8d7d main
filterCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=2,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
26.09.2011 15:52:54 org.apache.solr.search.SolrIndexSearcher warm
INFO: autowarming result for Searcher@17af46e main
filterCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
26.09.2011 15:52:54 org.apache.solr.search.SolrIndexSearcher warm
INFO: autowarming Searcher@17af46e main from Searcher@5e8d7d main
queryResultCache{lookups=0,hits=0,hitratio=0.00,inserts=2,evictions=0,size=2,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
26.09.2011 15:52:54 org.apache.solr.search.SolrIndexSearcher warm
INFO: autowarming result for Searcher@17af46e main
queryResultCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
26.09.2011 15:52:54 org.apache.solr.search.SolrIndexSearcher warm
INFO: autowarming Searcher@17af46e main from Searcher@5e8d7d main
documentCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
26.09.2011 15:52:54 org.apache.solr.search.SolrIndexSearcher warm
INFO: autowarming result for Searcher@17af46e main
documentCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
26.09.2011 15:52:54 org.apache.solr.core.QuerySenderListener newSearcher
INFO: QuerySenderListener sending requests to Searcher@17af46e main
26.09.2011 15:52:54 org.apache.solr.request.UnInvertedField uninvert
INFO: UnInverted multi-valued field
{field=doctype,memSize=4224,tindexSize=32,time=0,phase1=0,nTerms=0,bigTerms=0,termInstances=0,uses=0}
26.09.2011 15:52:54 org.apache.solr.core.SolrCore execute
INFO: [] webapp=null path=null
params={start=0&event=newSearcher&q=weka&rows=10} hits=0 status=0 QTime=16
26.09.2011 15:52:54 org.apache.solr.core.QuerySenderListener newSearcher
INFO: QuerySenderListener done.
26.09.2011 15:52:54
org.apache.solr.handler.component.SpellCheckComponent$SpellCheckerListener
newSearcher
INFO: Index is not optimized therefore skipping building spell check index
for: default
26.09.2011 15:52:54 org.apache.solr.core.SolrCore registerSearcher
INFO: [] Registered new searcher Searcher@17af46e main
26.09.2011 15:52:54 org.apache.solr.search.SolrIndexSearcher close
INFO: Closing Searcher@5e8d7d main
fieldValueCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=1,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0,item_doctype={field=doctype,memSize=4224,tindexSize=32,time=0,phase1=0,nTerms=0,bigTerms=0,termInstances=0,uses=2}}
filterCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=2,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
queryResultCache{lookups=0,hits=0,hitratio=0.00,inserts=2,evictions=0,size=2,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
documentCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
26.09.2011 15:52:58 org.apache.solr.handler.dataimport.MailEntityProcessor
connectToMailBox
INFO: Connected to mailbox
26.09.2011 15:52:59
org.apache.solr.handler.dataimport.MailEntityProcessor$FolderIterator next
INFO: Opened folder : Test
26.09.2011 15:52:59
org.apache.solr.handler.dataimport.MailEntityProcessor$FolderIterator next
INFO: Added its children to list :
26.09.2011 15:52:59
org.apache.solr.handler.dataimport.MailEntityProcessor$FolderIterator next
INFO: NO children :
26.09.2011 15:53:00
org.apache.solr.handler.dataimport.MailEntityProcessor$MessageIterator
<init>
INFO: Total messages : 2
26.09.2011 15:53:00
org.apache.solr.handler.dataimport.MailEntityProcessor$MessageIterator
<init>
INFO: Search criteria applied. Batching disabled
26.09.2011 15:53:01 org.apache.solr.handler.dataimport.DocBuilder finish
INFO: Import completed successfully
26.09.2011 15:53:01 org.apache.solr.update.DirectUpdateHandler2 commit
INFO: start
commit(optimize=true,waitFlush=false,waitSearcher=true,expungeDeletes=false)
26.09.2011 15:53:01 org.apache.solr.core.SolrDeletionPolicy onInit
INFO: SolrDeletionPolicy.onInit: commits:num=1
commit{dir=H:\_Projekt.lfd\zahn\solr_home_34\data\index,segFN=segments_5,version=1317035795834,generation=5,filenames=[segments_5]
26.09.2011 15:53:01 org.apache.solr.core.SolrDeletionPolicy updateCommits
INFO: newest commit = 1317035795834
26.09.2011 15:53:01 org.apache.solr.search.SolrIndexSearcher <init>
INFO: Opening Searcher@643edd main
26.09.2011 15:53:01 org.apache.solr.update.DirectUpdateHandler2 commit
INFO: end_commit_flush
26.09.2011 15:53:01 org.apache.solr.search.SolrIndexSearcher warm
INFO: autowarming Searcher@643edd main from Searcher@17af46e main
fieldValueCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=1,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0,item_doctype={field=doctype,memSize=4224,tindexSize=32,time=0,phase1=0,nTerms=0,bigTerms=0,termInstances=0,uses=1}}
26.09.2011 15:53:01 org.apache.solr.search.SolrIndexSearcher warm
INFO: autowarming result for Searcher@643edd main
fieldValueCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
26.09.2011 15:53:01 org.apache.solr.search.SolrIndexSearcher warm
INFO: autowarming Searcher@643edd main from Searcher@17af46e main
filterCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=1,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
26.09.2011 15:53:02 org.apache.solr.search.SolrIndexSearcher warm
INFO: autowarming result for Searcher@643edd main
filterCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
26.09.2011 15:53:02 org.apache.solr.search.SolrIndexSearcher warm
INFO: autowarming Searcher@643edd main from Searcher@17af46e main
queryResultCache{lookups=0,hits=0,hitratio=0.00,inserts=1,evictions=0,size=1,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
26.09.2011 15:53:02 org.apache.solr.search.SolrIndexSearcher warm
INFO: autowarming result for Searcher@643edd main
queryResultCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
26.09.2011 15:53:02 org.apache.solr.search.SolrIndexSearcher warm
INFO: autowarming Searcher@643edd main from Searcher@17af46e main
documentCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
26.09.2011 15:53:02 org.apache.solr.search.SolrIndexSearcher warm
INFO: autowarming result for Searcher@643edd main
documentCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
26.09.2011 15:53:02 org.apache.solr.core.QuerySenderListener newSearcher
INFO: QuerySenderListener sending requests to Searcher@643edd main
26.09.2011 15:53:02 org.apache.solr.request.UnInvertedField uninvert
INFO: UnInverted multi-valued field
{field=doctype,memSize=4224,tindexSize=32,time=0,phase1=0,nTerms=0,bigTerms=0,termInstances=0,uses=0}
26.09.2011 15:53:02 org.apache.solr.core.SolrCore execute
INFO: [] webapp=null path=null
params={start=0&event=newSearcher&q=weka&rows=10} hits=0 status=0 QTime=0
26.09.2011 15:53:02 org.apache.solr.core.QuerySenderListener newSearcher
INFO: QuerySenderListener done.
26.09.2011 15:53:02
org.apache.solr.handler.component.SpellCheckComponent$SpellCheckerListener
newSearcher
INFO: Index is not optimized therefore skipping building spell check index
for: default
26.09.2011 15:53:02 org.apache.solr.core.SolrCore registerSearcher
INFO: [] Registered new searcher Searcher@643edd main
26.09.2011 15:53:02 org.apache.solr.search.SolrIndexSearcher close
INFO: Closing Searcher@17af46e main
fieldValueCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=1,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0,item_doctype={field=doctype,memSize=4224,tindexSize=32,time=0,phase1=0,nTerms=0,bigTerms=0,termInstances=0,uses=1}}
filterCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=1,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
queryResultCache{lookups=0,hits=0,hitratio=0.00,inserts=1,evictions=0,size=1,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
documentCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
26.09.2011 15:53:02 org.apache.solr.handler.dataimport.SolrWriter
readIndexerProperties
INFO: Read dataimport-mail.properties
26.09.2011 15:53:02 org.apache.solr.handler.dataimport.SolrWriter persist
INFO: Wrote last indexed time to
H:\_Projekt.lfd\zahn\solr_home_34\.\conf\dataimport-mail.properties
26.09.2011 15:53:02 org.apache.solr.update.processor.LogUpdateProcessor
finish
INFO: {deleteByQuery=*:*,optimize=} 0 16
26.09.2011 15:53:02 org.apache.solr.handler.dataimport.DocBuilder execute
INFO: Time taken = 0:0:8.656
Thanks for help
Johannes
--
View this message in context: http://lucene.472066.n3.nabble.com/email-DIH-tp2711416p3369305.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: email - DIH
Posted by jb <jb...@shi-gmbh.com>.
Nobody to help?
I tried telnet to get informations about the emails. Via telnet with IMAP i
can get any required fields. Is this an implementation issue?
--
View this message in context: http://lucene.472066.n3.nabble.com/email-DIH-tp2711416p3392846.html
Sent from the Solr - User mailing list archive at Nabble.com.