You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Stephen Boesch <ja...@gmail.com> on 2011/01/01 02:07:52 UTC
solr newbie: Diagnose why DataImportHandler DIH not saving documents
I am asking for a full DataImport via a url. It seems to be partially
happy with the request - with debug=on I can see it saying that 10
documents were indexed. The backend however realizes there are actually 440
records available for the query.
Not sure why only 10 records were selected and then why even those 10
records are not stored.
Here is the obfuscated url used for invoking the DataImport:
<mySolrHost>:8983/solr/core0/dataimport?command=full-import&debug=on<http://knowtate.servehttp.com:8983/solr/core0/dataimport?command=full-import&debug=on>
Here is the output: looks reasonable for the 10 records it does find:
notice it says *added/updated 10 documents*
0360db-data-config.xmlfull-importdebugBrad is testing
thisjava.math.BigDecimal:1java.math.BigDecimal:15000947 Wood Duck
Lanejava.math.BigDecimal:3java.math.BigDecimal:15002Stanford Quad
Sculpturejava.math.BigDecimal:3java.math.BigDecimal:15200Apple Store - Palo
Altojava.math.BigDecimal:3java.math.BigDecimal:15201Fox
Theaterjava.math.BigDecimal:3java.math.BigDecimal:15220java.math.BigDecimal:3java.math.BigDecimal:15222Knowtate
promojava.math.BigDecimal:4welcome to Knowtatejava.math.BigDecimal:16163The
Green Dragon Tavernjava.math.BigDecimal:5java.math.BigDecimal:15020The All
New Infiniti Mjava.math.BigDecimal:5Introjava.math.BigDecimal:15100The All
New Infiniti Mjava.math.BigDecimal:5To hear current
specialsjava.math.BigDecimal:15100idleConfiguration Re-loaded
sucessfully11002010-12-31 16:45:11Indexing completed. *Added/Updated: 10
documents. *Deleted 0 documents.100:0:0.331This response format is
experimental. It is likely to change in the future.
But when I go to the Admin screen, it tells me Documents Processed:
10 *Total
Documents Processed 0*
*
*
So what is difference between Documents and "Total Documents" ?? Note that
there is presently *no *data in the indexes.
<mySolrHost>:8983/solr/core0/admin/stats.jsp<http://knowtate.servehttp.com:8983/solr/core0/admin/>
*name: */dataimport *class:
*org.apache.solr.handler.dataimport.DataImportHandler
*version: *1.0 *description: *Manage data import from databases to Solr *
stats: *Status : IDLE
Documents Processed : 10
Requests made to DataSource : 1
Rows Fetched : 10
Documents Deleted : 0
Documents Skipped : 0
Total Documents Processed : 0
Total Requests made to DataSource : 0
Total Rows Fetched : 0
Total Documents Deleted : 0
Total Documents Skipped : 0
handlerStart : 1293831460260
requests : 2
Re: solr newbie: Diagnose why DataImportHandler DIH not saving documents
Posted by Stephen Boesch <ja...@gmail.com>.
Yes that fixed the problem. interesting.. usually think setting debug just
changes the verbosity level.. in this case caused docs not to be processed.
02db-data-config.xmlfull-importidle144002010-12-31 17:45:03Indexing
completed. Added/Updated: 440 documents. Deleted 0 documents.2010-12-31
17:45:032010-12-31 17:45:034400:0:0.258This response format is experimental.
It is likely to change in the future.
Now I am seeing the full 440 docs being processed.
cool!
*ame: */dataimport *class:
*org.apache.solr.handler.dataimport.DataImportHandler
*version: *1.0 *description: *Manage data import from databases to Solr *
stats: *Status : IDLE
Documents Processed : 440
Requests made to DataSource : 1
Rows Fetched : 440
Documents Deleted : 0
Documents Skipped : 0
Total Documents Processed : 880
Total Requests made to DataSource : 2
Total Rows Fetched : 880
Total Documents Deleted : 0
Total Documents Skipped : 0
handlerStart : 1293831460260
requests : 35
errors : 0
timeouts : 0
totalTime : 3170
avgTimePerRequest : 90.57143
avgRequestsPerSecond : 0.008557899
2010/12/31 Stephen Boesch <ja...@gmail.com>
> sure I'll try that.
>
> 2010/12/31 Ahmet Arslan <io...@yahoo.com>
>
> It seems that with &debug=on there is a hard coded default rows=10.
>>
>>
>> http://knowtate.servehttp.com:8983/solr/core0/dataimport?command=full-import&debug=on&echoParams=all&rows=50
>>
>> returns "Added/Updated: 50 documents. Deleted 0 documents."
>>
>> It seems that debug parameter is related to
>> /solr/core0/admin/dataimport.jsp page.
>>
>> Don't know exact purpose of debug parameter but, can't you just ignore it
>> and use
>>
>>
>> http://knowtate.servehttp.com:8983/solr/core0/dataimport?command=full-import
>>
>>
>> --- On Sat, 1/1/11, Stephen Boesch <ja...@gmail.com> wrote:
>>
>> > From: Stephen Boesch <ja...@gmail.com>
>> > Subject: Re: solr newbie: Diagnose why DataImportHandler DIH not saving
>> documents
>> > To: solr-user@lucene.apache.org
>> > Date: Saturday, January 1, 2011, 3:09 AM
>> > one little extra piece of info: part
>> > of the stats page got omitted - notably
>> > the number of errors was reported as 0.
>> >
>> > errors : 0
>> > timeouts : 0
>> > totalTime : 1963
>> > avgTimePerRequest : 981.5
>> > avgRequestsPerSecond : 0.0011371888
>> >
>> >
>> > 2010/12/31 Stephen Boesch <ja...@gmail.com>
>> >
>> > > I am asking for a full DataImport via a url. It
>> > seems to be partially
>> > > happy with the request - with debug=on I can see
>> > it saying that 10
>> > > documents were indexed. The backend however
>> > realizes there are actually 440
>> > > records available for the query.
>> > >
>> > > Not sure why only 10 records were selected and then
>> > why even those 10
>> > > records are not stored.
>> > >
>> > >
>> > > Here is the obfuscated url used for invoking the
>> > DataImport:
>> > >
>> > >
>> > <mySolrHost>:8983/solr/core0/dataimport?command=full-import&debug=on<
>> http://knowtate.servehttp.com:8983/solr/core0/dataimport?command=full-import&debug=on
>> >
>> > >
>> > >
>> > > Here is the output: looks reasonable for the 10
>> > records it does find:
>> > > notice it says *added/updated 10 documents*
>> > >
>> > > 0360db-data-config.xmlfull-importdebugBrad is testing
>> > >
>> > thisjava.math.BigDecimal:1java.math.BigDecimal:15000947 Wood
>> > Duck
>> > >
>> > Lanejava.math.BigDecimal:3java.math.BigDecimal:15002Stanford
>> > Quad
>> > >
>> > Sculpturejava.math.BigDecimal:3java.math.BigDecimal:15200Apple
>> > Store - Palo
>> > >
>> > Altojava.math.BigDecimal:3java.math.BigDecimal:15201Fox
>> > >
>> >
>> Theaterjava.math.BigDecimal:3java.math.BigDecimal:15220java.math.BigDecimal:3java.math.BigDecimal:15222Knowtate
>> > > promojava.math.BigDecimal:4welcome to
>> > Knowtatejava.math.BigDecimal:16163The
>> > > Green Dragon
>> > Tavernjava.math.BigDecimal:5java.math.BigDecimal:15020The
>> > All
>> > > New Infiniti
>> > Mjava.math.BigDecimal:5Introjava.math.BigDecimal:15100The
>> > All
>> > > New Infiniti Mjava.math.BigDecimal:5To hear current
>> > > specialsjava.math.BigDecimal:15100idleConfiguration
>> > Re-loaded
>> > > sucessfully11002010-12-31 16:45:11Indexing completed.
>> > *Added/Updated: 10
>> > > documents. *Deleted 0 documents.100:0:0.331This
>> > response format is
>> > > experimental. It is likely to change in the future.
>> > >
>> > >
>> > > But when I go to the Admin screen, it tells
>> > me Documents Processed: 10
>> > > *Total Documents Processed 0*
>> > > *
>> > > *
>> > > So what is difference between Documents and "Total
>> > Documents" ?? Note that
>> > > there is presently *no *data in the indexes.
>> > >
>> > >
>> > <mySolrHost>:8983/solr/core0/admin/stats.jsp<
>> http://knowtate.servehttp.com:8983/solr/core0/admin/>
>> > >
>> > > *name: * /dataimport *class: *
>> > org.apache.solr.handler.dataimport.DataImportHandler
>> > > *version: * 1.0 *description: *
>> > Manage data import from databases to
>> > > Solr *stats: * Status : IDLE
>> > > Documents Processed : 10
>> > > Requests made to DataSource : 1
>> > > Rows Fetched : 10
>> > > Documents Deleted : 0
>> > > Documents Skipped : 0
>> > > Total Documents Processed : 0
>> > > Total Requests made to DataSource : 0
>> > > Total Rows Fetched : 0
>> > > Total Documents Deleted : 0
>> > > Total Documents Skipped : 0
>> > > handlerStart : 1293831460260
>> > > requests : 2
>> > >
>> > >
>> > >
>> >
>>
>>
>>
>>
>
Re: solr newbie: Diagnose why DataImportHandler DIH not saving documents
Posted by Stephen Boesch <ja...@gmail.com>.
sure I'll try that.
2010/12/31 Ahmet Arslan <io...@yahoo.com>
> It seems that with &debug=on there is a hard coded default rows=10.
>
>
> http://knowtate.servehttp.com:8983/solr/core0/dataimport?command=full-import&debug=on&echoParams=all&rows=50
>
> returns "Added/Updated: 50 documents. Deleted 0 documents."
>
> It seems that debug parameter is related to
> /solr/core0/admin/dataimport.jsp page.
>
> Don't know exact purpose of debug parameter but, can't you just ignore it
> and use
>
>
> http://knowtate.servehttp.com:8983/solr/core0/dataimport?command=full-import
>
>
> --- On Sat, 1/1/11, Stephen Boesch <ja...@gmail.com> wrote:
>
> > From: Stephen Boesch <ja...@gmail.com>
> > Subject: Re: solr newbie: Diagnose why DataImportHandler DIH not saving
> documents
> > To: solr-user@lucene.apache.org
> > Date: Saturday, January 1, 2011, 3:09 AM
> > one little extra piece of info: part
> > of the stats page got omitted - notably
> > the number of errors was reported as 0.
> >
> > errors : 0
> > timeouts : 0
> > totalTime : 1963
> > avgTimePerRequest : 981.5
> > avgRequestsPerSecond : 0.0011371888
> >
> >
> > 2010/12/31 Stephen Boesch <ja...@gmail.com>
> >
> > > I am asking for a full DataImport via a url. It
> > seems to be partially
> > > happy with the request - with debug=on I can see
> > it saying that 10
> > > documents were indexed. The backend however
> > realizes there are actually 440
> > > records available for the query.
> > >
> > > Not sure why only 10 records were selected and then
> > why even those 10
> > > records are not stored.
> > >
> > >
> > > Here is the obfuscated url used for invoking the
> > DataImport:
> > >
> > >
> > <mySolrHost>:8983/solr/core0/dataimport?command=full-import&debug=on<
> http://knowtate.servehttp.com:8983/solr/core0/dataimport?command=full-import&debug=on
> >
> > >
> > >
> > > Here is the output: looks reasonable for the 10
> > records it does find:
> > > notice it says *added/updated 10 documents*
> > >
> > > 0360db-data-config.xmlfull-importdebugBrad is testing
> > >
> > thisjava.math.BigDecimal:1java.math.BigDecimal:15000947 Wood
> > Duck
> > >
> > Lanejava.math.BigDecimal:3java.math.BigDecimal:15002Stanford
> > Quad
> > >
> > Sculpturejava.math.BigDecimal:3java.math.BigDecimal:15200Apple
> > Store - Palo
> > >
> > Altojava.math.BigDecimal:3java.math.BigDecimal:15201Fox
> > >
> >
> Theaterjava.math.BigDecimal:3java.math.BigDecimal:15220java.math.BigDecimal:3java.math.BigDecimal:15222Knowtate
> > > promojava.math.BigDecimal:4welcome to
> > Knowtatejava.math.BigDecimal:16163The
> > > Green Dragon
> > Tavernjava.math.BigDecimal:5java.math.BigDecimal:15020The
> > All
> > > New Infiniti
> > Mjava.math.BigDecimal:5Introjava.math.BigDecimal:15100The
> > All
> > > New Infiniti Mjava.math.BigDecimal:5To hear current
> > > specialsjava.math.BigDecimal:15100idleConfiguration
> > Re-loaded
> > > sucessfully11002010-12-31 16:45:11Indexing completed.
> > *Added/Updated: 10
> > > documents. *Deleted 0 documents.100:0:0.331This
> > response format is
> > > experimental. It is likely to change in the future.
> > >
> > >
> > > But when I go to the Admin screen, it tells
> > me Documents Processed: 10
> > > *Total Documents Processed 0*
> > > *
> > > *
> > > So what is difference between Documents and "Total
> > Documents" ?? Note that
> > > there is presently *no *data in the indexes.
> > >
> > >
> > <mySolrHost>:8983/solr/core0/admin/stats.jsp<
> http://knowtate.servehttp.com:8983/solr/core0/admin/>
> > >
> > > *name: * /dataimport *class: *
> > org.apache.solr.handler.dataimport.DataImportHandler
> > > *version: * 1.0 *description: *
> > Manage data import from databases to
> > > Solr *stats: * Status : IDLE
> > > Documents Processed : 10
> > > Requests made to DataSource : 1
> > > Rows Fetched : 10
> > > Documents Deleted : 0
> > > Documents Skipped : 0
> > > Total Documents Processed : 0
> > > Total Requests made to DataSource : 0
> > > Total Rows Fetched : 0
> > > Total Documents Deleted : 0
> > > Total Documents Skipped : 0
> > > handlerStart : 1293831460260
> > > requests : 2
> > >
> > >
> > >
> >
>
>
>
>
Re: solr newbie: Diagnose why DataImportHandler DIH not saving documents
Posted by Ahmet Arslan <io...@yahoo.com>.
It seems that with &debug=on there is a hard coded default rows=10.
http://knowtate.servehttp.com:8983/solr/core0/dataimport?command=full-import&debug=on&echoParams=all&rows=50
returns "Added/Updated: 50 documents. Deleted 0 documents."
It seems that debug parameter is related to /solr/core0/admin/dataimport.jsp page.
Don't know exact purpose of debug parameter but, can't you just ignore it and use
http://knowtate.servehttp.com:8983/solr/core0/dataimport?command=full-import
--- On Sat, 1/1/11, Stephen Boesch <ja...@gmail.com> wrote:
> From: Stephen Boesch <ja...@gmail.com>
> Subject: Re: solr newbie: Diagnose why DataImportHandler DIH not saving documents
> To: solr-user@lucene.apache.org
> Date: Saturday, January 1, 2011, 3:09 AM
> one little extra piece of info: part
> of the stats page got omitted - notably
> the number of errors was reported as 0.
>
> errors : 0
> timeouts : 0
> totalTime : 1963
> avgTimePerRequest : 981.5
> avgRequestsPerSecond : 0.0011371888
>
>
> 2010/12/31 Stephen Boesch <ja...@gmail.com>
>
> > I am asking for a full DataImport via a url. It
> seems to be partially
> > happy with the request - with debug=on I can see
> it saying that 10
> > documents were indexed. The backend however
> realizes there are actually 440
> > records available for the query.
> >
> > Not sure why only 10 records were selected and then
> why even those 10
> > records are not stored.
> >
> >
> > Here is the obfuscated url used for invoking the
> DataImport:
> >
> >
> <mySolrHost>:8983/solr/core0/dataimport?command=full-import&debug=on<http://knowtate.servehttp.com:8983/solr/core0/dataimport?command=full-import&debug=on>
> >
> >
> > Here is the output: looks reasonable for the 10
> records it does find:
> > notice it says *added/updated 10 documents*
> >
> > 0360db-data-config.xmlfull-importdebugBrad is testing
> >
> thisjava.math.BigDecimal:1java.math.BigDecimal:15000947 Wood
> Duck
> >
> Lanejava.math.BigDecimal:3java.math.BigDecimal:15002Stanford
> Quad
> >
> Sculpturejava.math.BigDecimal:3java.math.BigDecimal:15200Apple
> Store - Palo
> >
> Altojava.math.BigDecimal:3java.math.BigDecimal:15201Fox
> >
> Theaterjava.math.BigDecimal:3java.math.BigDecimal:15220java.math.BigDecimal:3java.math.BigDecimal:15222Knowtate
> > promojava.math.BigDecimal:4welcome to
> Knowtatejava.math.BigDecimal:16163The
> > Green Dragon
> Tavernjava.math.BigDecimal:5java.math.BigDecimal:15020The
> All
> > New Infiniti
> Mjava.math.BigDecimal:5Introjava.math.BigDecimal:15100The
> All
> > New Infiniti Mjava.math.BigDecimal:5To hear current
> > specialsjava.math.BigDecimal:15100idleConfiguration
> Re-loaded
> > sucessfully11002010-12-31 16:45:11Indexing completed.
> *Added/Updated: 10
> > documents. *Deleted 0 documents.100:0:0.331This
> response format is
> > experimental. It is likely to change in the future.
> >
> >
> > But when I go to the Admin screen, it tells
> me Documents Processed: 10
> > *Total Documents Processed 0*
> > *
> > *
> > So what is difference between Documents and "Total
> Documents" ?? Note that
> > there is presently *no *data in the indexes.
> >
> >
> <mySolrHost>:8983/solr/core0/admin/stats.jsp<http://knowtate.servehttp.com:8983/solr/core0/admin/>
> >
> > *name: * /dataimport *class: *
> org.apache.solr.handler.dataimport.DataImportHandler
> > *version: * 1.0 *description: *
> Manage data import from databases to
> > Solr *stats: * Status : IDLE
> > Documents Processed : 10
> > Requests made to DataSource : 1
> > Rows Fetched : 10
> > Documents Deleted : 0
> > Documents Skipped : 0
> > Total Documents Processed : 0
> > Total Requests made to DataSource : 0
> > Total Rows Fetched : 0
> > Total Documents Deleted : 0
> > Total Documents Skipped : 0
> > handlerStart : 1293831460260
> > requests : 2
> >
> >
> >
>
Re: solr newbie: Diagnose why DataImportHandler DIH not saving documents
Posted by Stephen Boesch <ja...@gmail.com>.
one little extra piece of info: part of the stats page got omitted - notably
the number of errors was reported as 0.
errors : 0
timeouts : 0
totalTime : 1963
avgTimePerRequest : 981.5
avgRequestsPerSecond : 0.0011371888
2010/12/31 Stephen Boesch <ja...@gmail.com>
> I am asking for a full DataImport via a url. It seems to be partially
> happy with the request - with debug=on I can see it saying that 10
> documents were indexed. The backend however realizes there are actually 440
> records available for the query.
>
> Not sure why only 10 records were selected and then why even those 10
> records are not stored.
>
>
> Here is the obfuscated url used for invoking the DataImport:
>
> <mySolrHost>:8983/solr/core0/dataimport?command=full-import&debug=on<http://knowtate.servehttp.com:8983/solr/core0/dataimport?command=full-import&debug=on>
>
>
> Here is the output: looks reasonable for the 10 records it does find:
> notice it says *added/updated 10 documents*
>
> 0360db-data-config.xmlfull-importdebugBrad is testing
> thisjava.math.BigDecimal:1java.math.BigDecimal:15000947 Wood Duck
> Lanejava.math.BigDecimal:3java.math.BigDecimal:15002Stanford Quad
> Sculpturejava.math.BigDecimal:3java.math.BigDecimal:15200Apple Store - Palo
> Altojava.math.BigDecimal:3java.math.BigDecimal:15201Fox
> Theaterjava.math.BigDecimal:3java.math.BigDecimal:15220java.math.BigDecimal:3java.math.BigDecimal:15222Knowtate
> promojava.math.BigDecimal:4welcome to Knowtatejava.math.BigDecimal:16163The
> Green Dragon Tavernjava.math.BigDecimal:5java.math.BigDecimal:15020The All
> New Infiniti Mjava.math.BigDecimal:5Introjava.math.BigDecimal:15100The All
> New Infiniti Mjava.math.BigDecimal:5To hear current
> specialsjava.math.BigDecimal:15100idleConfiguration Re-loaded
> sucessfully11002010-12-31 16:45:11Indexing completed. *Added/Updated: 10
> documents. *Deleted 0 documents.100:0:0.331This response format is
> experimental. It is likely to change in the future.
>
>
> But when I go to the Admin screen, it tells me Documents Processed: 10
> *Total Documents Processed 0*
> *
> *
> So what is difference between Documents and "Total Documents" ?? Note that
> there is presently *no *data in the indexes.
>
> <mySolrHost>:8983/solr/core0/admin/stats.jsp<http://knowtate.servehttp.com:8983/solr/core0/admin/>
>
> *name: * /dataimport *class: * org.apache.solr.handler.dataimport.DataImportHandler
> *version: * 1.0 *description: * Manage data import from databases to
> Solr *stats: * Status : IDLE
> Documents Processed : 10
> Requests made to DataSource : 1
> Rows Fetched : 10
> Documents Deleted : 0
> Documents Skipped : 0
> Total Documents Processed : 0
> Total Requests made to DataSource : 0
> Total Rows Fetched : 0
> Total Documents Deleted : 0
> Total Documents Skipped : 0
> handlerStart : 1293831460260
> requests : 2
>
>
>