You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by sunnyfr <jo...@gmail.com> on 2008/11/06 17:15:35 UTC
solr 1.3 - its obviously FULL_DUMP /dataimport,
Hi,
I don't get what's happening
I tried to make a full import with a limit inside between to id ...
everything works fine then the time keep running but it looks stuck and
Idon't have any error in my logs:
Like if a process couldn't be executed, I didn't turn on snapshooter or
autocommit is properly setteled I think :
<autoCommit>
<maxDocs>10000</maxDocs>
<maxTime>1000</maxTime>
</autoCommit>
Data-Config:
<str name="config">data-config.xml</str>
</lst>
</lst>
<str name="status">busy</str>
<str name="importResponse">A command is still running...</str>
−
<lst name="statusMessages">
<str name="Time Elapsed">0:16:32.47</str> ... <str name="Time
Elapsed">0:19:18.160</str>
<str name="Total Requests made to DataSource">661141</str>
<str name="Total Rows Fetched">1647265</str>
<str name="Total Documents Processed">110189</str>
<str name="Total Documents Skipped">0</str>
<str name="Full Dump Started">2008-11-06 16:50:17</str>
</lst>
−
<str name="WARNING">
This response format is experimental. It is likely to change in the future.
</str>
</response>
My stats :
name: /dataimport
class: org.apache.solr.handler.dataimport.DataImportHandler
version: 1.0
description: Manage data import from databases to Solr
stats: Status : RUNNING_FULL_DUMP
Documents Processed : 110189
Requests made to DataSource : 661141
Rows Fetched : 1647265
Documents Deleted : 0
Documents Skipped : 0
Total Documents Processed : 0
Total Requests made to DataSource : 0
Total Rows Fetched : 0
Total Documents Deleted : 0
Total Documents Skipped : 0
handlerStart : 1225986600254
requests : 86
errors : 0
timeouts : 0
totalTime : 0
avgTimePerRequest : 0.0
avgRequestsPerSecond : 0.05853414
searcherName : Searcher@7a919b6a main
caching : true
numDocs : 109686
maxDoc : 109686
readerImpl : MultiSegmentReader
readerDir : org.apache.lucene.store.FSDirectory@/data/solr/video/data/index
indexVersion : 1225986600336
openedAt : Thu Nov 06 16:57:37 CET 2008
registeredAt : Thu Nov 06 16:57:37 CET 2008
warmupTime : 0
Logs:
Nov 6 16:57:37 solr-test jsvc.exec[24862]: Nov 6, 2008 4:57:37 PM
org.apache.solr.search.SolrIndexSearcher warm INFO: autowarming result for
Searcher@7a919b6a main
^IfilterCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
Nov 6 16:57:37 solr-test jsvc.exec[24862]: Nov 6, 2008 4:57:37 PM
org.apache.solr.search.SolrIndexSearcher warm INFO: autowarming
Searcher@7a919b6a main from Searcher@6da351a7 main
^IqueryResultCache{lookups=0,hits=0,hitratio=0.00,inserts=5,evictions=0,size=5,warmupTime=50,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
Nov 6 16:57:37 solr-test jsvc.exec[24862]: Nov 6, 2008 4:57:37 PM
org.apache.solr.core.SolrDeletionPolicy onInit INFO:
SolrDeletionPolicy.onInit: commits:num=1
^Icommit{dir=/data/solr/video/data/index,segFN=segments_6g,version=1225986600336,generation=232,filenames=[segments_6g,
_33.fnm, _33.frq, _33.prx, _33.tis, _33.tii, _33.nrm, _33.tvx, _33.tvf,
_33.tvd, _33.fdx, _33.fdt, _66.fnm, _66.frq, _66.prx, _66.tis, _66.tii,
_66.nrm, _66.tvx, _66.tvf, _66.tvd, _66.fdx, _66.fdt, _6g.fnm, _6g.frq,
_6g.prx, _6g.tis, _6g.tii, _6g.nrm, _6g.tvx, _6g.tvf, _6g.tvd, _6g.fdx,
_6g.fdt, _6r.fnm, _6r.frq, _6r.prx, _6r.tis, _6r.tii, _6r.nrm, _6r.tvx,
_6r.tvf, _6r.tvd, _6r.fdx, _6r.fdt, _72.fnm, _72.frq, _72.prx, _72.tis,
_72.tii, _72.nrm, _72.tvx, _72.tvf, _72.tvd, _72.fdx, _72.fdt, _73.fnm,
_73.frq, _73.prx, _73.tis, _73.tii, _73.nrm, _73.tvx, _73.tvf, _73.tvd,
_73.fdx, _73.fdt]
Nov 6 16:57:37 solr-test jsvc.exec[24862]: Nov 6, 2008 4:57:37 PM
org.apache.solr.core.SolrDeletionPolicy updateCommits INFO: last commit =
1225986600336
Nov 6 16:57:37 solr-test jsvc.exec[24862]: Nov 6, 2008 4:57:37 PM
org.apache.solr.search.SolrIndexSearcher warm INFO: autowarming result for
Searcher@7a919b6a main
^IqueryResultCache{lookups=0,hits=0,hitratio=0.00,inserts=5,evictions=0,size=5,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
Nov 6 16:57:37 solr-test jsvc.exec[24862]: Nov 6, 2008 4:57:37 PM
org.apache.solr.search.SolrIndexSearcher warm INFO: autowarming
Searcher@7a919b6a main from Searcher@6da351a7 main
^IdocumentCache{lookups=0,hits=0,hitratio=0.00,inserts=20,evictions=0,size=20,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
Nov 6 16:57:37 solr-test jsvc.exec[24862]: Nov 6, 2008 4:57:37 PM
org.apache.solr.search.SolrIndexSearcher warm INFO: autowarming result for
Searcher@7a919b6a main
^IdocumentCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
Nov 6 16:57:37 solr-test jsvc.exec[24862]: Nov 6, 2008 4:57:37 PM
org.apache.solr.core.QuerySenderListener newSearcher INFO:
QuerySenderListener sending requests to Searcher@7a919b6a main
Nov 6 16:57:37 solr-test jsvc.exec[24862]: Nov 6, 2008 4:57:37 PM
org.apache.solr.core.SolrCore execute INFO: [video] webapp=null path=null
params={start=0&q=solr&rows=10} hits=0 status=0 QTime=0
Nov 6 16:57:37 solr-test jsvc.exec[24862]: Nov 6, 2008 4:57:37 PM
org.apache.solr.core.SolrCore execute INFO: [video] webapp=null path=null
params={start=0&q=rocks&rows=10} hits=960 status=0 QTime=0
Nov 6 16:57:37 solr-test jsvc.exec[24862]: Nov 6, 2008 4:57:37 PM
org.apache.solr.core.SolrCore execute INFO: [video] webapp=null path=null
params={q=static+newSearcher+warming+query+from+solrconfig.xml} hits=1149
status=0 QTime=0
Nov 6 16:57:37 solr-test jsvc.exec[24862]: Nov 6, 2008 4:57:37 PM
org.apache.solr.core.QuerySenderListener newSearcher INFO:
QuerySenderListener done.
Nov 6 16:57:37 solr-test jsvc.exec[24862]: Nov 6, 2008 4:57:37 PM
org.apache.solr.core.SolrCore registerSearcher INFO: [video] Registered new
searcher Searcher@7a919b6a main
Nov 6 16:57:37 solr-test jsvc.exec[24862]: Nov 6, 2008 4:57:37 PM
org.apache.solr.search.SolrIndexSearcher close INFO: Closing
Searcher@6da351a7 main
^IfilterCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
^IqueryResultCache{lookups=0,hits=0,hitratio=0.00,inserts=5,evictions=0,size=5,warmupTime=50,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
^IdocumentCache{lookups=0,hits=0,hitratio=0.00,inserts=20,evictions=0,size=20,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
Nov 6 16:57:59 solr-test jsvc.exec[24862]: Nov 6, 2008 4:57:59 PM
org.apache.solr.core.SolrCore execute INFO: [video] webapp=/solr
path=/dataimport params={} status=0 QTime=0
--
View this message in context: http://www.nabble.com/solr-1.3---its-obviously-FULL_DUMP--dataimport%2C-tp20364250p20364250.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: solr 1.3 - Problem Full Import
Posted by Noble Paul നോബിള് नोब्ळ् <no...@gmail.com>.
I cannot see what is the problem. If you can do a thread dump and
paste it here I may be able to suggest something
On Mon, Nov 10, 2008 at 9:44 PM, sunnyfr <jo...@gmail.com> wrote:
>
> Hi Shalin
>
> I don't get Ive executed :
> http://....com:8180/solr/video/dataimport?command=full-import
> Then when I Checked :
> http://solr-test.adm.video.com:8180/solr/video/dataimport
> It looks like document are created :
> <lst name="initArgs">
> −
> <lst name="defaults">
> <str name="config">data-config.xml</str>
> </lst>
> </lst>
> <str name="status">busy</str>
> <str name="importResponse">A command is still running...</str>
> −
> <lst name="statusMessages">
> <str name="Time Elapsed">0:0:46.331</str>
> <str name="Total Requests made to DataSource">37347</str>
> <str name="Total Rows Fetched">75108</str>
> <str name="Total Documents Processed">6224</str>
> <str name="Total Documents Skipped">0</str>
> <str name="Full Dump Started">2008-11-10 17:02:24</str>
> </lst>
>
> but then after a while, time keep increasing but just time no documents ;..
> and it can be like that for hours.
>
> <str name="status">busy</str>
> <str name="importResponse">A command is still running...</str>
> −
> <lst name="statusMessages">
> <str name="Time Elapsed">0:4:41.295</str>
> <str name="Total Requests made to DataSource">84103</str>
> <str name="Total Rows Fetched">113842</str>
> <str name="Total Documents Processed">14016</str>
> <str name="Total Documents Skipped">0</str>
> <str name="Full Dump Started">2008-11-10 17:02:24</str>
> </lst>
>
> and no exception : http://www.nabble.com/file/p20422919/syslog syslog
>
> Thanks for your help,
>
>
>
> Shalin Shekhar Mangar wrote:
>>
>> Hi,
>>
>> I'm confused by the log you have posted. There is no import command to be
>> seen to /dataimport. Only one request is in the log to /dataimport at the
>> very end. The logs say that Solr started up and some warming queries were
>> executed and that's it.
>>
>> What do you mean by it looks stuck? Does the count of processed documents
>> increase? Any exceptions during import would cause DataImportHandler to
>> abort. The only case that DIH will not catch is when a error like out of
>> memory happened. But still Solr or Tomcat should catch and log that
>> exception (or it may just die).
>>
>> On Mon, Nov 10, 2008 at 8:33 PM, sunnyfr <jo...@gmail.com> wrote:
>>
>>>
>>> Hi,
>>>
>>> I can't get what's going on ??
>>> If somebody could help me it would be very nice
>>> thanks
>>>
>>>
>>> sunnyfr wrote:
>>> >
>>> > Hi,
>>> >
>>> > I don't get what's happening
>>> > I tried to make a full import with a limit inside between to id ...
>>> > everything works fine then the time keep running but it looks stuck and
>>> > Idon't have any error in my logs:
>>> >
>>> > Like if a process couldn't be executed, I didn't turn on snapshooter or
>>> > autocommit is properly setteled I think :
>>> > <autoCommit>
>>> > <maxDocs>10000</maxDocs>
>>> > <maxTime>1000</maxTime>
>>> > </autoCommit>
>>> >
>>> >
>>> > Data-Config:
>>> >
>>> > <str name="config">data-config.xml</str>
>>> > </lst>
>>> > </lst>
>>> > <str name="status">busy</str>
>>> > <str name="importResponse">A command is still running...</str>
>>> > −
>>> > <lst name="statusMessages">
>>> > <str name="Time Elapsed">0:16:32.47</str> ... <str name="Time
>>> > Elapsed">0:19:18.160</str>
>>> > <str name="Total Requests made to DataSource">661141</str>
>>> > <str name="Total Rows Fetched">1647265</str>
>>> > <str name="Total Documents Processed">110189</str>
>>> > <str name="Total Documents Skipped">0</str>
>>> > <str name="Full Dump Started">2008-11-06 16:50:17</str>
>>> > </lst>
>>> > −
>>> > <str name="WARNING">
>>> > This response format is experimental. It is likely to change in the
>>> > future.
>>> > </str>
>>> > </response>
>>> >
>>> > My stats :
>>> > name: /dataimport
>>> > class: org.apache.solr.handler.dataimport.DataImportHandler
>>> > version: 1.0
>>> > description: Manage data import from databases to Solr
>>> > stats: Status : RUNNING_FULL_DUMP
>>> > Documents Processed : 110189
>>> > Requests made to DataSource : 661141
>>> > Rows Fetched : 1647265
>>> > Documents Deleted : 0
>>> > Documents Skipped : 0
>>> > Total Documents Processed : 0
>>> > Total Requests made to DataSource : 0
>>> > Total Rows Fetched : 0
>>> > Total Documents Deleted : 0
>>> > Total Documents Skipped : 0
>>> > handlerStart : 1225986600254
>>> > requests : 86
>>> > errors : 0
>>> > timeouts : 0
>>> > totalTime : 0
>>> > avgTimePerRequest : 0.0
>>> > avgRequestsPerSecond : 0.05853414
>>> >
>>> > searcherName : Searcher@7a919b6a main
>>> > caching : true
>>> > numDocs : 109686
>>> > maxDoc : 109686
>>> > readerImpl : MultiSegmentReader
>>> > readerDir :
>>> > org.apache.lucene.store.FSDirectory@/data/solr/video/data/index
>>> > indexVersion : 1225986600336
>>> > openedAt : Thu Nov 06 16:57:37 CET 2008
>>> > registeredAt : Thu Nov 06 16:57:37 CET 2008
>>> > warmupTime : 0
>>> >
>>> >
>>> > Logs:
>>> > Nov 6 16:57:37 solr-test jsvc.exec[24862]: Nov 6, 2008 4:57:37 PM
>>> > org.apache.solr.search.SolrIndexSearcher warm INFO: autowarming result
>>> for
>>> > Searcher@7a919b6a main
>>> >
>>> ^IfilterCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
>>> > Nov 6 16:57:37 solr-test jsvc.exec[24862]: Nov 6, 2008 4:57:37 PM
>>> > org.apache.solr.search.SolrIndexSearcher warm INFO: autowarming
>>> > Searcher@7a919b6a main from Searcher@6da351a7 main
>>> >
>>> ^IqueryResultCache{lookups=0,hits=0,hitratio=0.00,inserts=5,evictions=0,size=5,warmupTime=50,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
>>> > Nov 6 16:57:37 solr-test jsvc.exec[24862]: Nov 6, 2008 4:57:37 PM
>>> > org.apache.solr.core.SolrDeletionPolicy onInit INFO:
>>> > SolrDeletionPolicy.onInit: commits:num=1
>>> >
>>> ^Icommit{dir=/data/solr/video/data/index,segFN=segments_6g,version=1225986600336,generation=232,filenames=[segments_6g,
>>> > _33.fnm, _33.frq, _33.prx, _33.tis, _33.tii, _33.nrm, _33.tvx, _33.tvf,
>>> > _33.tvd, _33.fdx, _33.fdt, _66.fnm, _66.frq, _66.prx, _66.tis, _66.tii,
>>> > _66.nrm, _66.tvx, _66.tvf, _66.tvd, _66.fdx, _66.fdt, _6g.fnm, _6g.frq,
>>> > _6g.prx, _6g.tis, _6g.tii, _6g.nrm, _6g.tvx, _6g.tvf, _6g.tvd, _6g.fdx,
>>> > _6g.fdt, _6r.fnm, _6r.frq, _6r.prx, _6r.tis, _6r.tii, _6r.nrm, _6r.tvx,
>>> > _6r.tvf, _6r.tvd, _6r.fdx, _6r.fdt, _72.fnm, _72.frq, _72.prx, _72.tis,
>>> > _72.tii, _72.nrm, _72.tvx, _72.tvf, _72.tvd, _72.fdx, _72.fdt, _73.fnm,
>>> > _73.frq, _73.prx, _73.tis, _73.tii, _73.nrm, _73.tvx, _73.tvf, _73.tvd,
>>> > _73.fdx, _73.fdt]
>>> > Nov 6 16:57:37 solr-test jsvc.exec[24862]: Nov 6, 2008 4:57:37 PM
>>> > org.apache.solr.core.SolrDeletionPolicy updateCommits INFO: last commit
>>> =
>>> > 1225986600336
>>> > Nov 6 16:57:37 solr-test jsvc.exec[24862]: Nov 6, 2008 4:57:37 PM
>>> > org.apache.solr.search.SolrIndexSearcher warm INFO: autowarming result
>>> for
>>> > Searcher@7a919b6a main
>>> >
>>> ^IqueryResultCache{lookups=0,hits=0,hitratio=0.00,inserts=5,evictions=0,size=5,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
>>> > Nov 6 16:57:37 solr-test jsvc.exec[24862]: Nov 6, 2008 4:57:37 PM
>>> > org.apache.solr.search.SolrIndexSearcher warm INFO: autowarming
>>> > Searcher@7a919b6a main from Searcher@6da351a7 main
>>> >
>>> ^IdocumentCache{lookups=0,hits=0,hitratio=0.00,inserts=20,evictions=0,size=20,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
>>> > Nov 6 16:57:37 solr-test jsvc.exec[24862]: Nov 6, 2008 4:57:37 PM
>>> > org.apache.solr.search.SolrIndexSearcher warm INFO: autowarming result
>>> for
>>> > Searcher@7a919b6a main
>>> >
>>> ^IdocumentCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
>>> > Nov 6 16:57:37 solr-test jsvc.exec[24862]: Nov 6, 2008 4:57:37 PM
>>> > org.apache.solr.core.QuerySenderListener newSearcher INFO:
>>> > QuerySenderListener sending requests to Searcher@7a919b6a main
>>> > Nov 6 16:57:37 solr-test jsvc.exec[24862]: Nov 6, 2008 4:57:37 PM
>>> > org.apache.solr.core.SolrCore execute INFO: [video] webapp=null
>>> path=null
>>> > params={start=0&q=solr&rows=10} hits=0 status=0 QTime=0
>>> > Nov 6 16:57:37 solr-test jsvc.exec[24862]: Nov 6, 2008 4:57:37 PM
>>> > org.apache.solr.core.SolrCore execute INFO: [video] webapp=null
>>> path=null
>>> > params={start=0&q=rocks&rows=10} hits=960 status=0 QTime=0
>>> > Nov 6 16:57:37 solr-test jsvc.exec[24862]: Nov 6, 2008 4:57:37 PM
>>> > org.apache.solr.core.SolrCore execute INFO: [video] webapp=null
>>> path=null
>>> > params={q=static+newSearcher+warming+query+from+solrconfig.xml}
>>> hits=1149
>>> > status=0 QTime=0
>>> > Nov 6 16:57:37 solr-test jsvc.exec[24862]: Nov 6, 2008 4:57:37 PM
>>> > org.apache.solr.core.QuerySenderListener newSearcher INFO:
>>> > QuerySenderListener done.
>>> > Nov 6 16:57:37 solr-test jsvc.exec[24862]: Nov 6, 2008 4:57:37 PM
>>> > org.apache.solr.core.SolrCore registerSearcher INFO: [video] Registered
>>> > new searcher Searcher@7a919b6a main
>>> > Nov 6 16:57:37 solr-test jsvc.exec[24862]: Nov 6, 2008 4:57:37 PM
>>> > org.apache.solr.search.SolrIndexSearcher close INFO: Closing
>>> > Searcher@6da351a7 main
>>> >
>>> ^IfilterCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
>>> >
>>> ^IqueryResultCache{lookups=0,hits=0,hitratio=0.00,inserts=5,evictions=0,size=5,warmupTime=50,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
>>> >
>>> ^IdocumentCache{lookups=0,hits=0,hitratio=0.00,inserts=20,evictions=0,size=20,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
>>> > Nov 6 16:57:59 solr-test jsvc.exec[24862]: Nov 6, 2008 4:57:59 PM
>>> > org.apache.solr.core.SolrCore execute INFO: [video] webapp=/solr
>>> > path=/dataimport params={} status=0 QTime=0
>>> >
>>> >
>>>
>>> --
>>> View this message in context:
>>> http://www.nabble.com/solr-1.3---Problem-Full-Import-tp20364250p20421643.html
>>> Sent from the Solr - User mailing list archive at Nabble.com.
>>>
>>>
>>
>>
>> --
>> Regards,
>> Shalin Shekhar Mangar.
>>
>>
>
> --
> View this message in context: http://www.nabble.com/solr-1.3---Problem-Full-Import-tp20364250p20422919.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
>
--
--Noble Paul
Re: solr 1.3 - Problem Full Import
Posted by sunnyfr <jo...@gmail.com>.
Hi Shalin
I don't get Ive executed :
http://....com:8180/solr/video/dataimport?command=full-import
Then when I Checked :
http://solr-test.adm.video.com:8180/solr/video/dataimport
It looks like document are created :
<lst name="initArgs">
−
<lst name="defaults">
<str name="config">data-config.xml</str>
</lst>
</lst>
<str name="status">busy</str>
<str name="importResponse">A command is still running...</str>
−
<lst name="statusMessages">
<str name="Time Elapsed">0:0:46.331</str>
<str name="Total Requests made to DataSource">37347</str>
<str name="Total Rows Fetched">75108</str>
<str name="Total Documents Processed">6224</str>
<str name="Total Documents Skipped">0</str>
<str name="Full Dump Started">2008-11-10 17:02:24</str>
</lst>
but then after a while, time keep increasing but just time no documents ;..
and it can be like that for hours.
<str name="status">busy</str>
<str name="importResponse">A command is still running...</str>
−
<lst name="statusMessages">
<str name="Time Elapsed">0:4:41.295</str>
<str name="Total Requests made to DataSource">84103</str>
<str name="Total Rows Fetched">113842</str>
<str name="Total Documents Processed">14016</str>
<str name="Total Documents Skipped">0</str>
<str name="Full Dump Started">2008-11-10 17:02:24</str>
</lst>
and no exception : http://www.nabble.com/file/p20422919/syslog syslog
Thanks for your help,
Shalin Shekhar Mangar wrote:
>
> Hi,
>
> I'm confused by the log you have posted. There is no import command to be
> seen to /dataimport. Only one request is in the log to /dataimport at the
> very end. The logs say that Solr started up and some warming queries were
> executed and that's it.
>
> What do you mean by it looks stuck? Does the count of processed documents
> increase? Any exceptions during import would cause DataImportHandler to
> abort. The only case that DIH will not catch is when a error like out of
> memory happened. But still Solr or Tomcat should catch and log that
> exception (or it may just die).
>
> On Mon, Nov 10, 2008 at 8:33 PM, sunnyfr <jo...@gmail.com> wrote:
>
>>
>> Hi,
>>
>> I can't get what's going on ??
>> If somebody could help me it would be very nice
>> thanks
>>
>>
>> sunnyfr wrote:
>> >
>> > Hi,
>> >
>> > I don't get what's happening
>> > I tried to make a full import with a limit inside between to id ...
>> > everything works fine then the time keep running but it looks stuck and
>> > Idon't have any error in my logs:
>> >
>> > Like if a process couldn't be executed, I didn't turn on snapshooter or
>> > autocommit is properly setteled I think :
>> > <autoCommit>
>> > <maxDocs>10000</maxDocs>
>> > <maxTime>1000</maxTime>
>> > </autoCommit>
>> >
>> >
>> > Data-Config:
>> >
>> > <str name="config">data-config.xml</str>
>> > </lst>
>> > </lst>
>> > <str name="status">busy</str>
>> > <str name="importResponse">A command is still running...</str>
>> > −
>> > <lst name="statusMessages">
>> > <str name="Time Elapsed">0:16:32.47</str> ... <str name="Time
>> > Elapsed">0:19:18.160</str>
>> > <str name="Total Requests made to DataSource">661141</str>
>> > <str name="Total Rows Fetched">1647265</str>
>> > <str name="Total Documents Processed">110189</str>
>> > <str name="Total Documents Skipped">0</str>
>> > <str name="Full Dump Started">2008-11-06 16:50:17</str>
>> > </lst>
>> > −
>> > <str name="WARNING">
>> > This response format is experimental. It is likely to change in the
>> > future.
>> > </str>
>> > </response>
>> >
>> > My stats :
>> > name: /dataimport
>> > class: org.apache.solr.handler.dataimport.DataImportHandler
>> > version: 1.0
>> > description: Manage data import from databases to Solr
>> > stats: Status : RUNNING_FULL_DUMP
>> > Documents Processed : 110189
>> > Requests made to DataSource : 661141
>> > Rows Fetched : 1647265
>> > Documents Deleted : 0
>> > Documents Skipped : 0
>> > Total Documents Processed : 0
>> > Total Requests made to DataSource : 0
>> > Total Rows Fetched : 0
>> > Total Documents Deleted : 0
>> > Total Documents Skipped : 0
>> > handlerStart : 1225986600254
>> > requests : 86
>> > errors : 0
>> > timeouts : 0
>> > totalTime : 0
>> > avgTimePerRequest : 0.0
>> > avgRequestsPerSecond : 0.05853414
>> >
>> > searcherName : Searcher@7a919b6a main
>> > caching : true
>> > numDocs : 109686
>> > maxDoc : 109686
>> > readerImpl : MultiSegmentReader
>> > readerDir :
>> > org.apache.lucene.store.FSDirectory@/data/solr/video/data/index
>> > indexVersion : 1225986600336
>> > openedAt : Thu Nov 06 16:57:37 CET 2008
>> > registeredAt : Thu Nov 06 16:57:37 CET 2008
>> > warmupTime : 0
>> >
>> >
>> > Logs:
>> > Nov 6 16:57:37 solr-test jsvc.exec[24862]: Nov 6, 2008 4:57:37 PM
>> > org.apache.solr.search.SolrIndexSearcher warm INFO: autowarming result
>> for
>> > Searcher@7a919b6a main
>> >
>> ^IfilterCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
>> > Nov 6 16:57:37 solr-test jsvc.exec[24862]: Nov 6, 2008 4:57:37 PM
>> > org.apache.solr.search.SolrIndexSearcher warm INFO: autowarming
>> > Searcher@7a919b6a main from Searcher@6da351a7 main
>> >
>> ^IqueryResultCache{lookups=0,hits=0,hitratio=0.00,inserts=5,evictions=0,size=5,warmupTime=50,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
>> > Nov 6 16:57:37 solr-test jsvc.exec[24862]: Nov 6, 2008 4:57:37 PM
>> > org.apache.solr.core.SolrDeletionPolicy onInit INFO:
>> > SolrDeletionPolicy.onInit: commits:num=1
>> >
>> ^Icommit{dir=/data/solr/video/data/index,segFN=segments_6g,version=1225986600336,generation=232,filenames=[segments_6g,
>> > _33.fnm, _33.frq, _33.prx, _33.tis, _33.tii, _33.nrm, _33.tvx, _33.tvf,
>> > _33.tvd, _33.fdx, _33.fdt, _66.fnm, _66.frq, _66.prx, _66.tis, _66.tii,
>> > _66.nrm, _66.tvx, _66.tvf, _66.tvd, _66.fdx, _66.fdt, _6g.fnm, _6g.frq,
>> > _6g.prx, _6g.tis, _6g.tii, _6g.nrm, _6g.tvx, _6g.tvf, _6g.tvd, _6g.fdx,
>> > _6g.fdt, _6r.fnm, _6r.frq, _6r.prx, _6r.tis, _6r.tii, _6r.nrm, _6r.tvx,
>> > _6r.tvf, _6r.tvd, _6r.fdx, _6r.fdt, _72.fnm, _72.frq, _72.prx, _72.tis,
>> > _72.tii, _72.nrm, _72.tvx, _72.tvf, _72.tvd, _72.fdx, _72.fdt, _73.fnm,
>> > _73.frq, _73.prx, _73.tis, _73.tii, _73.nrm, _73.tvx, _73.tvf, _73.tvd,
>> > _73.fdx, _73.fdt]
>> > Nov 6 16:57:37 solr-test jsvc.exec[24862]: Nov 6, 2008 4:57:37 PM
>> > org.apache.solr.core.SolrDeletionPolicy updateCommits INFO: last commit
>> =
>> > 1225986600336
>> > Nov 6 16:57:37 solr-test jsvc.exec[24862]: Nov 6, 2008 4:57:37 PM
>> > org.apache.solr.search.SolrIndexSearcher warm INFO: autowarming result
>> for
>> > Searcher@7a919b6a main
>> >
>> ^IqueryResultCache{lookups=0,hits=0,hitratio=0.00,inserts=5,evictions=0,size=5,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
>> > Nov 6 16:57:37 solr-test jsvc.exec[24862]: Nov 6, 2008 4:57:37 PM
>> > org.apache.solr.search.SolrIndexSearcher warm INFO: autowarming
>> > Searcher@7a919b6a main from Searcher@6da351a7 main
>> >
>> ^IdocumentCache{lookups=0,hits=0,hitratio=0.00,inserts=20,evictions=0,size=20,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
>> > Nov 6 16:57:37 solr-test jsvc.exec[24862]: Nov 6, 2008 4:57:37 PM
>> > org.apache.solr.search.SolrIndexSearcher warm INFO: autowarming result
>> for
>> > Searcher@7a919b6a main
>> >
>> ^IdocumentCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
>> > Nov 6 16:57:37 solr-test jsvc.exec[24862]: Nov 6, 2008 4:57:37 PM
>> > org.apache.solr.core.QuerySenderListener newSearcher INFO:
>> > QuerySenderListener sending requests to Searcher@7a919b6a main
>> > Nov 6 16:57:37 solr-test jsvc.exec[24862]: Nov 6, 2008 4:57:37 PM
>> > org.apache.solr.core.SolrCore execute INFO: [video] webapp=null
>> path=null
>> > params={start=0&q=solr&rows=10} hits=0 status=0 QTime=0
>> > Nov 6 16:57:37 solr-test jsvc.exec[24862]: Nov 6, 2008 4:57:37 PM
>> > org.apache.solr.core.SolrCore execute INFO: [video] webapp=null
>> path=null
>> > params={start=0&q=rocks&rows=10} hits=960 status=0 QTime=0
>> > Nov 6 16:57:37 solr-test jsvc.exec[24862]: Nov 6, 2008 4:57:37 PM
>> > org.apache.solr.core.SolrCore execute INFO: [video] webapp=null
>> path=null
>> > params={q=static+newSearcher+warming+query+from+solrconfig.xml}
>> hits=1149
>> > status=0 QTime=0
>> > Nov 6 16:57:37 solr-test jsvc.exec[24862]: Nov 6, 2008 4:57:37 PM
>> > org.apache.solr.core.QuerySenderListener newSearcher INFO:
>> > QuerySenderListener done.
>> > Nov 6 16:57:37 solr-test jsvc.exec[24862]: Nov 6, 2008 4:57:37 PM
>> > org.apache.solr.core.SolrCore registerSearcher INFO: [video] Registered
>> > new searcher Searcher@7a919b6a main
>> > Nov 6 16:57:37 solr-test jsvc.exec[24862]: Nov 6, 2008 4:57:37 PM
>> > org.apache.solr.search.SolrIndexSearcher close INFO: Closing
>> > Searcher@6da351a7 main
>> >
>> ^IfilterCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
>> >
>> ^IqueryResultCache{lookups=0,hits=0,hitratio=0.00,inserts=5,evictions=0,size=5,warmupTime=50,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
>> >
>> ^IdocumentCache{lookups=0,hits=0,hitratio=0.00,inserts=20,evictions=0,size=20,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
>> > Nov 6 16:57:59 solr-test jsvc.exec[24862]: Nov 6, 2008 4:57:59 PM
>> > org.apache.solr.core.SolrCore execute INFO: [video] webapp=/solr
>> > path=/dataimport params={} status=0 QTime=0
>> >
>> >
>>
>> --
>> View this message in context:
>> http://www.nabble.com/solr-1.3---Problem-Full-Import-tp20364250p20421643.html
>> Sent from the Solr - User mailing list archive at Nabble.com.
>>
>>
>
>
> --
> Regards,
> Shalin Shekhar Mangar.
>
>
--
View this message in context: http://www.nabble.com/solr-1.3---Problem-Full-Import-tp20364250p20422919.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: solr 1.3 - Problem Full Import
Posted by sunnyfr <jo...@gmail.com>.
Hi Shalin
I don't get Ive executed :
http://....com:8180/solr/video/dataimport?command=full-import
Then when I Checked :
http://solr-test.adm.dailymotion.com:8180/solr/video/dataimport
It looks like document are created :
<lst name="initArgs">
−
<lst name="defaults">
<str name="config">data-config.xml</str>
</lst>
</lst>
<str name="status">busy</str>
<str name="importResponse">A command is still running...</str>
−
<lst name="statusMessages">
<str name="Time Elapsed">0:0:46.331</str>
<str name="Total Requests made to DataSource">37347</str>
<str name="Total Rows Fetched">75108</str>
<str name="Total Documents Processed">6224</str>
<str name="Total Documents Skipped">0</str>
<str name="Full Dump Started">2008-11-10 17:02:24</str>
</lst>
but then after a while, time keep increasing but just time no documents ;..
and it can be like that for hours.
<str name="status">busy</str>
<str name="importResponse">A command is still running...</str>
−
<lst name="statusMessages">
<str name="Time Elapsed">0:4:41.295</str>
<str name="Total Requests made to DataSource">84103</str>
<str name="Total Rows Fetched">113842</str>
<str name="Total Documents Processed">14016</str>
<str name="Total Documents Skipped">0</str>
<str name="Full Dump Started">2008-11-10 17:02:24</str>
</lst>
and no exception : http://www.nabble.com/file/p20422854/syslog syslog
Thanks for your help,
Shalin Shekhar Mangar wrote:
>
> Hi,
>
> I'm confused by the log you have posted. There is no import command to be
> seen to /dataimport. Only one request is in the log to /dataimport at the
> very end. The logs say that Solr started up and some warming queries were
> executed and that's it.
>
> What do you mean by it looks stuck? Does the count of processed documents
> increase? Any exceptions during import would cause DataImportHandler to
> abort. The only case that DIH will not catch is when a error like out of
> memory happened. But still Solr or Tomcat should catch and log that
> exception (or it may just die).
>
> On Mon, Nov 10, 2008 at 8:33 PM, sunnyfr <jo...@gmail.com> wrote:
>
>>
>> Hi,
>>
>> I can't get what's going on ??
>> If somebody could help me it would be very nice
>> thanks
>>
>>
>> sunnyfr wrote:
>> >
>> > Hi,
>> >
>> > I don't get what's happening
>> > I tried to make a full import with a limit inside between to id ...
>> > everything works fine then the time keep running but it looks stuck and
>> > Idon't have any error in my logs:
>> >
>> > Like if a process couldn't be executed, I didn't turn on snapshooter or
>> > autocommit is properly setteled I think :
>> > <autoCommit>
>> > <maxDocs>10000</maxDocs>
>> > <maxTime>1000</maxTime>
>> > </autoCommit>
>> >
>> >
>> > Data-Config:
>> >
>> > <str name="config">data-config.xml</str>
>> > </lst>
>> > </lst>
>> > <str name="status">busy</str>
>> > <str name="importResponse">A command is still running...</str>
>> > −
>> > <lst name="statusMessages">
>> > <str name="Time Elapsed">0:16:32.47</str> ... <str name="Time
>> > Elapsed">0:19:18.160</str>
>> > <str name="Total Requests made to DataSource">661141</str>
>> > <str name="Total Rows Fetched">1647265</str>
>> > <str name="Total Documents Processed">110189</str>
>> > <str name="Total Documents Skipped">0</str>
>> > <str name="Full Dump Started">2008-11-06 16:50:17</str>
>> > </lst>
>> > −
>> > <str name="WARNING">
>> > This response format is experimental. It is likely to change in the
>> > future.
>> > </str>
>> > </response>
>> >
>> > My stats :
>> > name: /dataimport
>> > class: org.apache.solr.handler.dataimport.DataImportHandler
>> > version: 1.0
>> > description: Manage data import from databases to Solr
>> > stats: Status : RUNNING_FULL_DUMP
>> > Documents Processed : 110189
>> > Requests made to DataSource : 661141
>> > Rows Fetched : 1647265
>> > Documents Deleted : 0
>> > Documents Skipped : 0
>> > Total Documents Processed : 0
>> > Total Requests made to DataSource : 0
>> > Total Rows Fetched : 0
>> > Total Documents Deleted : 0
>> > Total Documents Skipped : 0
>> > handlerStart : 1225986600254
>> > requests : 86
>> > errors : 0
>> > timeouts : 0
>> > totalTime : 0
>> > avgTimePerRequest : 0.0
>> > avgRequestsPerSecond : 0.05853414
>> >
>> > searcherName : Searcher@7a919b6a main
>> > caching : true
>> > numDocs : 109686
>> > maxDoc : 109686
>> > readerImpl : MultiSegmentReader
>> > readerDir :
>> > org.apache.lucene.store.FSDirectory@/data/solr/video/data/index
>> > indexVersion : 1225986600336
>> > openedAt : Thu Nov 06 16:57:37 CET 2008
>> > registeredAt : Thu Nov 06 16:57:37 CET 2008
>> > warmupTime : 0
>> >
>> >
>> > Logs:
>> > Nov 6 16:57:37 solr-test jsvc.exec[24862]: Nov 6, 2008 4:57:37 PM
>> > org.apache.solr.search.SolrIndexSearcher warm INFO: autowarming result
>> for
>> > Searcher@7a919b6a main
>> >
>> ^IfilterCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
>> > Nov 6 16:57:37 solr-test jsvc.exec[24862]: Nov 6, 2008 4:57:37 PM
>> > org.apache.solr.search.SolrIndexSearcher warm INFO: autowarming
>> > Searcher@7a919b6a main from Searcher@6da351a7 main
>> >
>> ^IqueryResultCache{lookups=0,hits=0,hitratio=0.00,inserts=5,evictions=0,size=5,warmupTime=50,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
>> > Nov 6 16:57:37 solr-test jsvc.exec[24862]: Nov 6, 2008 4:57:37 PM
>> > org.apache.solr.core.SolrDeletionPolicy onInit INFO:
>> > SolrDeletionPolicy.onInit: commits:num=1
>> >
>> ^Icommit{dir=/data/solr/video/data/index,segFN=segments_6g,version=1225986600336,generation=232,filenames=[segments_6g,
>> > _33.fnm, _33.frq, _33.prx, _33.tis, _33.tii, _33.nrm, _33.tvx, _33.tvf,
>> > _33.tvd, _33.fdx, _33.fdt, _66.fnm, _66.frq, _66.prx, _66.tis, _66.tii,
>> > _66.nrm, _66.tvx, _66.tvf, _66.tvd, _66.fdx, _66.fdt, _6g.fnm, _6g.frq,
>> > _6g.prx, _6g.tis, _6g.tii, _6g.nrm, _6g.tvx, _6g.tvf, _6g.tvd, _6g.fdx,
>> > _6g.fdt, _6r.fnm, _6r.frq, _6r.prx, _6r.tis, _6r.tii, _6r.nrm, _6r.tvx,
>> > _6r.tvf, _6r.tvd, _6r.fdx, _6r.fdt, _72.fnm, _72.frq, _72.prx, _72.tis,
>> > _72.tii, _72.nrm, _72.tvx, _72.tvf, _72.tvd, _72.fdx, _72.fdt, _73.fnm,
>> > _73.frq, _73.prx, _73.tis, _73.tii, _73.nrm, _73.tvx, _73.tvf, _73.tvd,
>> > _73.fdx, _73.fdt]
>> > Nov 6 16:57:37 solr-test jsvc.exec[24862]: Nov 6, 2008 4:57:37 PM
>> > org.apache.solr.core.SolrDeletionPolicy updateCommits INFO: last commit
>> =
>> > 1225986600336
>> > Nov 6 16:57:37 solr-test jsvc.exec[24862]: Nov 6, 2008 4:57:37 PM
>> > org.apache.solr.search.SolrIndexSearcher warm INFO: autowarming result
>> for
>> > Searcher@7a919b6a main
>> >
>> ^IqueryResultCache{lookups=0,hits=0,hitratio=0.00,inserts=5,evictions=0,size=5,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
>> > Nov 6 16:57:37 solr-test jsvc.exec[24862]: Nov 6, 2008 4:57:37 PM
>> > org.apache.solr.search.SolrIndexSearcher warm INFO: autowarming
>> > Searcher@7a919b6a main from Searcher@6da351a7 main
>> >
>> ^IdocumentCache{lookups=0,hits=0,hitratio=0.00,inserts=20,evictions=0,size=20,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
>> > Nov 6 16:57:37 solr-test jsvc.exec[24862]: Nov 6, 2008 4:57:37 PM
>> > org.apache.solr.search.SolrIndexSearcher warm INFO: autowarming result
>> for
>> > Searcher@7a919b6a main
>> >
>> ^IdocumentCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
>> > Nov 6 16:57:37 solr-test jsvc.exec[24862]: Nov 6, 2008 4:57:37 PM
>> > org.apache.solr.core.QuerySenderListener newSearcher INFO:
>> > QuerySenderListener sending requests to Searcher@7a919b6a main
>> > Nov 6 16:57:37 solr-test jsvc.exec[24862]: Nov 6, 2008 4:57:37 PM
>> > org.apache.solr.core.SolrCore execute INFO: [video] webapp=null
>> path=null
>> > params={start=0&q=solr&rows=10} hits=0 status=0 QTime=0
>> > Nov 6 16:57:37 solr-test jsvc.exec[24862]: Nov 6, 2008 4:57:37 PM
>> > org.apache.solr.core.SolrCore execute INFO: [video] webapp=null
>> path=null
>> > params={start=0&q=rocks&rows=10} hits=960 status=0 QTime=0
>> > Nov 6 16:57:37 solr-test jsvc.exec[24862]: Nov 6, 2008 4:57:37 PM
>> > org.apache.solr.core.SolrCore execute INFO: [video] webapp=null
>> path=null
>> > params={q=static+newSearcher+warming+query+from+solrconfig.xml}
>> hits=1149
>> > status=0 QTime=0
>> > Nov 6 16:57:37 solr-test jsvc.exec[24862]: Nov 6, 2008 4:57:37 PM
>> > org.apache.solr.core.QuerySenderListener newSearcher INFO:
>> > QuerySenderListener done.
>> > Nov 6 16:57:37 solr-test jsvc.exec[24862]: Nov 6, 2008 4:57:37 PM
>> > org.apache.solr.core.SolrCore registerSearcher INFO: [video] Registered
>> > new searcher Searcher@7a919b6a main
>> > Nov 6 16:57:37 solr-test jsvc.exec[24862]: Nov 6, 2008 4:57:37 PM
>> > org.apache.solr.search.SolrIndexSearcher close INFO: Closing
>> > Searcher@6da351a7 main
>> >
>> ^IfilterCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
>> >
>> ^IqueryResultCache{lookups=0,hits=0,hitratio=0.00,inserts=5,evictions=0,size=5,warmupTime=50,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
>> >
>> ^IdocumentCache{lookups=0,hits=0,hitratio=0.00,inserts=20,evictions=0,size=20,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
>> > Nov 6 16:57:59 solr-test jsvc.exec[24862]: Nov 6, 2008 4:57:59 PM
>> > org.apache.solr.core.SolrCore execute INFO: [video] webapp=/solr
>> > path=/dataimport params={} status=0 QTime=0
>> >
>> >
>>
>> --
>> View this message in context:
>> http://www.nabble.com/solr-1.3---Problem-Full-Import-tp20364250p20421643.html
>> Sent from the Solr - User mailing list archive at Nabble.com.
>>
>>
>
>
> --
> Regards,
> Shalin Shekhar Mangar.
>
>
--
View this message in context: http://www.nabble.com/solr-1.3---Problem-Full-Import-tp20364250p20422854.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: solr 1.3 - Problem Full Import
Posted by Shalin Shekhar Mangar <sh...@gmail.com>.
Hi,
I'm confused by the log you have posted. There is no import command to be
seen to /dataimport. Only one request is in the log to /dataimport at the
very end. The logs say that Solr started up and some warming queries were
executed and that's it.
What do you mean by it looks stuck? Does the count of processed documents
increase? Any exceptions during import would cause DataImportHandler to
abort. The only case that DIH will not catch is when a error like out of
memory happened. But still Solr or Tomcat should catch and log that
exception (or it may just die).
On Mon, Nov 10, 2008 at 8:33 PM, sunnyfr <jo...@gmail.com> wrote:
>
> Hi,
>
> I can't get what's going on ??
> If somebody could help me it would be very nice
> thanks
>
>
> sunnyfr wrote:
> >
> > Hi,
> >
> > I don't get what's happening
> > I tried to make a full import with a limit inside between to id ...
> > everything works fine then the time keep running but it looks stuck and
> > Idon't have any error in my logs:
> >
> > Like if a process couldn't be executed, I didn't turn on snapshooter or
> > autocommit is properly setteled I think :
> > <autoCommit>
> > <maxDocs>10000</maxDocs>
> > <maxTime>1000</maxTime>
> > </autoCommit>
> >
> >
> > Data-Config:
> >
> > <str name="config">data-config.xml</str>
> > </lst>
> > </lst>
> > <str name="status">busy</str>
> > <str name="importResponse">A command is still running...</str>
> > −
> > <lst name="statusMessages">
> > <str name="Time Elapsed">0:16:32.47</str> ... <str name="Time
> > Elapsed">0:19:18.160</str>
> > <str name="Total Requests made to DataSource">661141</str>
> > <str name="Total Rows Fetched">1647265</str>
> > <str name="Total Documents Processed">110189</str>
> > <str name="Total Documents Skipped">0</str>
> > <str name="Full Dump Started">2008-11-06 16:50:17</str>
> > </lst>
> > −
> > <str name="WARNING">
> > This response format is experimental. It is likely to change in the
> > future.
> > </str>
> > </response>
> >
> > My stats :
> > name: /dataimport
> > class: org.apache.solr.handler.dataimport.DataImportHandler
> > version: 1.0
> > description: Manage data import from databases to Solr
> > stats: Status : RUNNING_FULL_DUMP
> > Documents Processed : 110189
> > Requests made to DataSource : 661141
> > Rows Fetched : 1647265
> > Documents Deleted : 0
> > Documents Skipped : 0
> > Total Documents Processed : 0
> > Total Requests made to DataSource : 0
> > Total Rows Fetched : 0
> > Total Documents Deleted : 0
> > Total Documents Skipped : 0
> > handlerStart : 1225986600254
> > requests : 86
> > errors : 0
> > timeouts : 0
> > totalTime : 0
> > avgTimePerRequest : 0.0
> > avgRequestsPerSecond : 0.05853414
> >
> > searcherName : Searcher@7a919b6a main
> > caching : true
> > numDocs : 109686
> > maxDoc : 109686
> > readerImpl : MultiSegmentReader
> > readerDir :
> > org.apache.lucene.store.FSDirectory@/data/solr/video/data/index
> > indexVersion : 1225986600336
> > openedAt : Thu Nov 06 16:57:37 CET 2008
> > registeredAt : Thu Nov 06 16:57:37 CET 2008
> > warmupTime : 0
> >
> >
> > Logs:
> > Nov 6 16:57:37 solr-test jsvc.exec[24862]: Nov 6, 2008 4:57:37 PM
> > org.apache.solr.search.SolrIndexSearcher warm INFO: autowarming result
> for
> > Searcher@7a919b6a main
> >
> ^IfilterCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
> > Nov 6 16:57:37 solr-test jsvc.exec[24862]: Nov 6, 2008 4:57:37 PM
> > org.apache.solr.search.SolrIndexSearcher warm INFO: autowarming
> > Searcher@7a919b6a main from Searcher@6da351a7 main
> >
> ^IqueryResultCache{lookups=0,hits=0,hitratio=0.00,inserts=5,evictions=0,size=5,warmupTime=50,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
> > Nov 6 16:57:37 solr-test jsvc.exec[24862]: Nov 6, 2008 4:57:37 PM
> > org.apache.solr.core.SolrDeletionPolicy onInit INFO:
> > SolrDeletionPolicy.onInit: commits:num=1
> >
> ^Icommit{dir=/data/solr/video/data/index,segFN=segments_6g,version=1225986600336,generation=232,filenames=[segments_6g,
> > _33.fnm, _33.frq, _33.prx, _33.tis, _33.tii, _33.nrm, _33.tvx, _33.tvf,
> > _33.tvd, _33.fdx, _33.fdt, _66.fnm, _66.frq, _66.prx, _66.tis, _66.tii,
> > _66.nrm, _66.tvx, _66.tvf, _66.tvd, _66.fdx, _66.fdt, _6g.fnm, _6g.frq,
> > _6g.prx, _6g.tis, _6g.tii, _6g.nrm, _6g.tvx, _6g.tvf, _6g.tvd, _6g.fdx,
> > _6g.fdt, _6r.fnm, _6r.frq, _6r.prx, _6r.tis, _6r.tii, _6r.nrm, _6r.tvx,
> > _6r.tvf, _6r.tvd, _6r.fdx, _6r.fdt, _72.fnm, _72.frq, _72.prx, _72.tis,
> > _72.tii, _72.nrm, _72.tvx, _72.tvf, _72.tvd, _72.fdx, _72.fdt, _73.fnm,
> > _73.frq, _73.prx, _73.tis, _73.tii, _73.nrm, _73.tvx, _73.tvf, _73.tvd,
> > _73.fdx, _73.fdt]
> > Nov 6 16:57:37 solr-test jsvc.exec[24862]: Nov 6, 2008 4:57:37 PM
> > org.apache.solr.core.SolrDeletionPolicy updateCommits INFO: last commit =
> > 1225986600336
> > Nov 6 16:57:37 solr-test jsvc.exec[24862]: Nov 6, 2008 4:57:37 PM
> > org.apache.solr.search.SolrIndexSearcher warm INFO: autowarming result
> for
> > Searcher@7a919b6a main
> >
> ^IqueryResultCache{lookups=0,hits=0,hitratio=0.00,inserts=5,evictions=0,size=5,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
> > Nov 6 16:57:37 solr-test jsvc.exec[24862]: Nov 6, 2008 4:57:37 PM
> > org.apache.solr.search.SolrIndexSearcher warm INFO: autowarming
> > Searcher@7a919b6a main from Searcher@6da351a7 main
> >
> ^IdocumentCache{lookups=0,hits=0,hitratio=0.00,inserts=20,evictions=0,size=20,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
> > Nov 6 16:57:37 solr-test jsvc.exec[24862]: Nov 6, 2008 4:57:37 PM
> > org.apache.solr.search.SolrIndexSearcher warm INFO: autowarming result
> for
> > Searcher@7a919b6a main
> >
> ^IdocumentCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
> > Nov 6 16:57:37 solr-test jsvc.exec[24862]: Nov 6, 2008 4:57:37 PM
> > org.apache.solr.core.QuerySenderListener newSearcher INFO:
> > QuerySenderListener sending requests to Searcher@7a919b6a main
> > Nov 6 16:57:37 solr-test jsvc.exec[24862]: Nov 6, 2008 4:57:37 PM
> > org.apache.solr.core.SolrCore execute INFO: [video] webapp=null path=null
> > params={start=0&q=solr&rows=10} hits=0 status=0 QTime=0
> > Nov 6 16:57:37 solr-test jsvc.exec[24862]: Nov 6, 2008 4:57:37 PM
> > org.apache.solr.core.SolrCore execute INFO: [video] webapp=null path=null
> > params={start=0&q=rocks&rows=10} hits=960 status=0 QTime=0
> > Nov 6 16:57:37 solr-test jsvc.exec[24862]: Nov 6, 2008 4:57:37 PM
> > org.apache.solr.core.SolrCore execute INFO: [video] webapp=null path=null
> > params={q=static+newSearcher+warming+query+from+solrconfig.xml} hits=1149
> > status=0 QTime=0
> > Nov 6 16:57:37 solr-test jsvc.exec[24862]: Nov 6, 2008 4:57:37 PM
> > org.apache.solr.core.QuerySenderListener newSearcher INFO:
> > QuerySenderListener done.
> > Nov 6 16:57:37 solr-test jsvc.exec[24862]: Nov 6, 2008 4:57:37 PM
> > org.apache.solr.core.SolrCore registerSearcher INFO: [video] Registered
> > new searcher Searcher@7a919b6a main
> > Nov 6 16:57:37 solr-test jsvc.exec[24862]: Nov 6, 2008 4:57:37 PM
> > org.apache.solr.search.SolrIndexSearcher close INFO: Closing
> > Searcher@6da351a7 main
> >
> ^IfilterCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
> >
> ^IqueryResultCache{lookups=0,hits=0,hitratio=0.00,inserts=5,evictions=0,size=5,warmupTime=50,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
> >
> ^IdocumentCache{lookups=0,hits=0,hitratio=0.00,inserts=20,evictions=0,size=20,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
> > Nov 6 16:57:59 solr-test jsvc.exec[24862]: Nov 6, 2008 4:57:59 PM
> > org.apache.solr.core.SolrCore execute INFO: [video] webapp=/solr
> > path=/dataimport params={} status=0 QTime=0
> >
> >
>
> --
> View this message in context:
> http://www.nabble.com/solr-1.3---Problem-Full-Import-tp20364250p20421643.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
>
--
Regards,
Shalin Shekhar Mangar.
Re: solr 1.3 - Problem Full Import
Posted by sunnyfr <jo...@gmail.com>.
Hi,
I can't get what's going on ??
If somebody could help me it would be very nice
thanks
sunnyfr wrote:
>
> Hi,
>
> I don't get what's happening
> I tried to make a full import with a limit inside between to id ...
> everything works fine then the time keep running but it looks stuck and
> Idon't have any error in my logs:
>
> Like if a process couldn't be executed, I didn't turn on snapshooter or
> autocommit is properly setteled I think :
> <autoCommit>
> <maxDocs>10000</maxDocs>
> <maxTime>1000</maxTime>
> </autoCommit>
>
>
> Data-Config:
>
> <str name="config">data-config.xml</str>
> </lst>
> </lst>
> <str name="status">busy</str>
> <str name="importResponse">A command is still running...</str>
> −
> <lst name="statusMessages">
> <str name="Time Elapsed">0:16:32.47</str> ... <str name="Time
> Elapsed">0:19:18.160</str>
> <str name="Total Requests made to DataSource">661141</str>
> <str name="Total Rows Fetched">1647265</str>
> <str name="Total Documents Processed">110189</str>
> <str name="Total Documents Skipped">0</str>
> <str name="Full Dump Started">2008-11-06 16:50:17</str>
> </lst>
> −
> <str name="WARNING">
> This response format is experimental. It is likely to change in the
> future.
> </str>
> </response>
>
> My stats :
> name: /dataimport
> class: org.apache.solr.handler.dataimport.DataImportHandler
> version: 1.0
> description: Manage data import from databases to Solr
> stats: Status : RUNNING_FULL_DUMP
> Documents Processed : 110189
> Requests made to DataSource : 661141
> Rows Fetched : 1647265
> Documents Deleted : 0
> Documents Skipped : 0
> Total Documents Processed : 0
> Total Requests made to DataSource : 0
> Total Rows Fetched : 0
> Total Documents Deleted : 0
> Total Documents Skipped : 0
> handlerStart : 1225986600254
> requests : 86
> errors : 0
> timeouts : 0
> totalTime : 0
> avgTimePerRequest : 0.0
> avgRequestsPerSecond : 0.05853414
>
> searcherName : Searcher@7a919b6a main
> caching : true
> numDocs : 109686
> maxDoc : 109686
> readerImpl : MultiSegmentReader
> readerDir :
> org.apache.lucene.store.FSDirectory@/data/solr/video/data/index
> indexVersion : 1225986600336
> openedAt : Thu Nov 06 16:57:37 CET 2008
> registeredAt : Thu Nov 06 16:57:37 CET 2008
> warmupTime : 0
>
>
> Logs:
> Nov 6 16:57:37 solr-test jsvc.exec[24862]: Nov 6, 2008 4:57:37 PM
> org.apache.solr.search.SolrIndexSearcher warm INFO: autowarming result for
> Searcher@7a919b6a main
> ^IfilterCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
> Nov 6 16:57:37 solr-test jsvc.exec[24862]: Nov 6, 2008 4:57:37 PM
> org.apache.solr.search.SolrIndexSearcher warm INFO: autowarming
> Searcher@7a919b6a main from Searcher@6da351a7 main
> ^IqueryResultCache{lookups=0,hits=0,hitratio=0.00,inserts=5,evictions=0,size=5,warmupTime=50,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
> Nov 6 16:57:37 solr-test jsvc.exec[24862]: Nov 6, 2008 4:57:37 PM
> org.apache.solr.core.SolrDeletionPolicy onInit INFO:
> SolrDeletionPolicy.onInit: commits:num=1
> ^Icommit{dir=/data/solr/video/data/index,segFN=segments_6g,version=1225986600336,generation=232,filenames=[segments_6g,
> _33.fnm, _33.frq, _33.prx, _33.tis, _33.tii, _33.nrm, _33.tvx, _33.tvf,
> _33.tvd, _33.fdx, _33.fdt, _66.fnm, _66.frq, _66.prx, _66.tis, _66.tii,
> _66.nrm, _66.tvx, _66.tvf, _66.tvd, _66.fdx, _66.fdt, _6g.fnm, _6g.frq,
> _6g.prx, _6g.tis, _6g.tii, _6g.nrm, _6g.tvx, _6g.tvf, _6g.tvd, _6g.fdx,
> _6g.fdt, _6r.fnm, _6r.frq, _6r.prx, _6r.tis, _6r.tii, _6r.nrm, _6r.tvx,
> _6r.tvf, _6r.tvd, _6r.fdx, _6r.fdt, _72.fnm, _72.frq, _72.prx, _72.tis,
> _72.tii, _72.nrm, _72.tvx, _72.tvf, _72.tvd, _72.fdx, _72.fdt, _73.fnm,
> _73.frq, _73.prx, _73.tis, _73.tii, _73.nrm, _73.tvx, _73.tvf, _73.tvd,
> _73.fdx, _73.fdt]
> Nov 6 16:57:37 solr-test jsvc.exec[24862]: Nov 6, 2008 4:57:37 PM
> org.apache.solr.core.SolrDeletionPolicy updateCommits INFO: last commit =
> 1225986600336
> Nov 6 16:57:37 solr-test jsvc.exec[24862]: Nov 6, 2008 4:57:37 PM
> org.apache.solr.search.SolrIndexSearcher warm INFO: autowarming result for
> Searcher@7a919b6a main
> ^IqueryResultCache{lookups=0,hits=0,hitratio=0.00,inserts=5,evictions=0,size=5,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
> Nov 6 16:57:37 solr-test jsvc.exec[24862]: Nov 6, 2008 4:57:37 PM
> org.apache.solr.search.SolrIndexSearcher warm INFO: autowarming
> Searcher@7a919b6a main from Searcher@6da351a7 main
> ^IdocumentCache{lookups=0,hits=0,hitratio=0.00,inserts=20,evictions=0,size=20,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
> Nov 6 16:57:37 solr-test jsvc.exec[24862]: Nov 6, 2008 4:57:37 PM
> org.apache.solr.search.SolrIndexSearcher warm INFO: autowarming result for
> Searcher@7a919b6a main
> ^IdocumentCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
> Nov 6 16:57:37 solr-test jsvc.exec[24862]: Nov 6, 2008 4:57:37 PM
> org.apache.solr.core.QuerySenderListener newSearcher INFO:
> QuerySenderListener sending requests to Searcher@7a919b6a main
> Nov 6 16:57:37 solr-test jsvc.exec[24862]: Nov 6, 2008 4:57:37 PM
> org.apache.solr.core.SolrCore execute INFO: [video] webapp=null path=null
> params={start=0&q=solr&rows=10} hits=0 status=0 QTime=0
> Nov 6 16:57:37 solr-test jsvc.exec[24862]: Nov 6, 2008 4:57:37 PM
> org.apache.solr.core.SolrCore execute INFO: [video] webapp=null path=null
> params={start=0&q=rocks&rows=10} hits=960 status=0 QTime=0
> Nov 6 16:57:37 solr-test jsvc.exec[24862]: Nov 6, 2008 4:57:37 PM
> org.apache.solr.core.SolrCore execute INFO: [video] webapp=null path=null
> params={q=static+newSearcher+warming+query+from+solrconfig.xml} hits=1149
> status=0 QTime=0
> Nov 6 16:57:37 solr-test jsvc.exec[24862]: Nov 6, 2008 4:57:37 PM
> org.apache.solr.core.QuerySenderListener newSearcher INFO:
> QuerySenderListener done.
> Nov 6 16:57:37 solr-test jsvc.exec[24862]: Nov 6, 2008 4:57:37 PM
> org.apache.solr.core.SolrCore registerSearcher INFO: [video] Registered
> new searcher Searcher@7a919b6a main
> Nov 6 16:57:37 solr-test jsvc.exec[24862]: Nov 6, 2008 4:57:37 PM
> org.apache.solr.search.SolrIndexSearcher close INFO: Closing
> Searcher@6da351a7 main
> ^IfilterCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
> ^IqueryResultCache{lookups=0,hits=0,hitratio=0.00,inserts=5,evictions=0,size=5,warmupTime=50,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
> ^IdocumentCache{lookups=0,hits=0,hitratio=0.00,inserts=20,evictions=0,size=20,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
> Nov 6 16:57:59 solr-test jsvc.exec[24862]: Nov 6, 2008 4:57:59 PM
> org.apache.solr.core.SolrCore execute INFO: [video] webapp=/solr
> path=/dataimport params={} status=0 QTime=0
>
>
--
View this message in context: http://www.nabble.com/solr-1.3---Problem-Full-Import-tp20364250p20421643.html
Sent from the Solr - User mailing list archive at Nabble.com.