You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Giri <gi...@gmail.com> on 2008/11/12 00:23:12 UTC
DataImportHandler not indexing all the records
Hi,
I have about ~ 2 million records in a mySQL database table (about 9 fields
from a single table), and I am trying to load it to the solr using
DataImportHandler using the command=full-import option. it only indexed
about 615360 records out of 2 millions.
here is my db-data-config.xml
<dataConfig>
<dataSource type="JdbcDataSource" driver="com.mysql.jdbc.Driver"
url="jdbc:mysql://localhost:3306/mydb" user="ua" password="pw" batchSize
="-1"/>
<document name="climate">
<entity name="occurence" query="select * from mylargetable">
<field column="id" name="id" />
<field column="title" name="title" />
<field column="url" name="url" />
</entity>
</document>
</dataConfig>
and in my solr schema.xml, i define these fields as:
<field name="id" type="string" indexed="true" stored="true"
multiValued="true"/>
<field name="title" type="text" indexed="true" stored="true"
multiValued="true" required="false"/>
<field name="url" type="text" indexed="true" stored="true"
multiValued="true" required="false"/>
If I try to index just one field (id), then it indexes about 960000 records,
but if I try to index all the above three fields, it indexes only 615360
records.
Any help will be appreciated.
thanks!
Re: DataImportHandler not indexing all the records
Posted by Jon Baer <jo...@gmail.com>.
Ive also had the same issues here but when trying to switch to
HTMLStripWhitespaceTokenizerFactor I found that it only removes the
tags but when it comes to all forms of javascript includes in a
document it keeps it all intact so I ended up w/ scripts in the
document text, is there any easy way to get around that (w/o having to
use the RegEx processor)?
Thanks.
- Jon
On Nov 15, 2008, at 2:21 PM, Shalin Shekhar Mangar wrote:
> I think the problem is that DIH catches Exception but not Error so a
> StackOverFlowError will slip past it. Normally, the
> SolrDispatchFilter will
> log such errors but the import is performed in a new thread, so the
> error is
> not logged anywhere. However, DIH will not commit documents in this
> case
> (and there is no mention of a commit in your DIH status).
>
> We should change the catch clause to catch Throwable so that this is
> not
> repeated. I'll open an issue and give a patch.
>
> Btw, Ahmed, Solr has a Tokenizer which is much better at striping
> html --
> HTMLStripWhitespaceTokenizerFactory which you can use for such tasks.
>
> On Sun, Nov 16, 2008 at 12:30 AM, Ahmed Hammad <ah...@gmail.com>
> wrote:
>
>> I had a similar problem like Giri. I have 17,000 record in one
>> table and
>> DIH
>> can import only 12464.
>>
>> After some investigation, I found my problem.
>>
>> I have a regular expression to strip off html tags form input text,
>> as
>> following:
>>
>> <field sourceColName="content" column="content" regex="<(.|\n)*?
>> >"
>> replaceWith=" "/>
>>
>> The DIH RegEx have stack overflow on the record 17,000 due to error
>> in the
>> content and then DIH exit without any error in the log on in the
>> status
>> command. Here is the status:
>>
>> <lst name="statusMessages">
>> <str name="Time Elapsed">0:0:31.657</str>
>> <str name="Total Requests made to DataSource">1</str>
>> <str name="Total Rows Fetched">12464</str>
>> <str name="Total Documents Processed">12464</str>
>> <str name="Total Documents Skipped">0</str>
>> <str name="Full Dump Started">2008-11-15 20:40:58</str>
>> </lst>
>>
>> I found the error in Eclipse Console window while debugging; it was
>> a stack
>> overflow in the RegEx library.
>>
>> The problem is that, DIH does not show any problem in log file on
>> in status
>> message.
>> What I think is important is to show whatever error happen in the
>> log file.
>>
>> I noticed also that, in case of no error a log message show
>> completness:
>>
>> Nov 15, 2008 8:57:34 PM org.apache.solr.handler.dataimport.DocBuilder
>> execute
>> INFO: Time taken = 0:0:40.656
>>
>> In case of RegEx stack overflow error, this log message does not
>> appear.
>>
>> I am researching on how to catch such error in DIH. Any ideas?
>>
>>
>> Regards,
>> ahmd
>>
>> On Sat, Nov 15, 2008 at 6:32 AM, Noble Paul നോബിള്
>> नोब्ळ् <
>> noble.paul@gmail.com> wrote:
>>
>>> There is no obvious problem
>>>
>>> I can be reasonably sure that
>>> the query
>>>
>>> select * from climatedata.ws_record limit 1000000
>>>
>>> would have fetched only 615360 rows.
>>> This is a very reliable pice of information
>>> <str name="Total Rows Fetched">615360</str>
>>>
>>> On Sat, Nov 15, 2008 at 12:41 AM, Giri <gi...@gmail.com>
>>> wrote:
>>>> Hi Noble,
>>>> thanks for the help, here are the details: the field "id" is
>>>> unique,
>> when
>>> I
>>>> did a select distinct(id), it returned 1 million rows.
>>>>
>>>> -------------------------------------------------------------------
>>>> db-data-config.xml
>>>> note: I limit the resultset to 1 million in the select query
>>>> -------------------------------------------------------------------
>>>> <dataConfig>
>>>> <dataSource type="JdbcDataSource" driver="com.mysql.jdbc.Driver"
>>>> url="jdbc:mysql://localhost:3306/climatedata" user="user"
>>>> password="pw"
>>>> batchSize ="-1"/>
>>>> <document name="climateRecord">
>>>> <entity name="observation" query="select * from
>>>> climatedata.ws_record limit 1000000">
>>>> <field column="id" name="id" />
>>>> <field column="inst_code" name="inst_code" />
>>>> <field column="inst_name" name="inst_name" />
>>>> <field column="meas_name" name="meas_name" />
>>>> <field column="latitude" name="latitude" />
>>>> <field column="longitude" name="longitude" />
>>>> <field column="ob_id" name="ob_id" />
>>>> <field column="in_id" name="in_id" />
>>>> <field column="ob_name" name="ob_name" />
>>>> </entity>
>>>> </document>
>>>> </dataConfig>
>>>>
>>>> -----------------------------------------------------------------
>>>> in the solr Schema.xml:
>>>> ----------------------------------------------------------------
>>>> <fields>
>>>> <field name="id" type="string" indexed="true" stored="true"
>>>> multiValued="false"/>
>>>> <field name="inst_code" type="text" indexed="true" stored="true"
>>>> multiValued="true" required="false"/>
>>>> <field name="inst_name" type="text" indexed="true" stored="true"
>>>> multiValued="true" required="false"/>
>>>> <field name="meas_name" type="text" indexed="true" stored="true"
>>>> multiValued="true" required="false"/>
>>>> <field name="latitude" type="sfloat" class="solr.FloatField"
>>>> indexed="true" stored="true" required="false"/>
>>>> <field name="longitude" type="sfloat" class="solr.FloatField"
>>>> indexed="true" stored="true" required="false"/>
>>>> <field name="ob_id" type="string" indexed="true" stored="true"
>>>> multiValued="true"/>
>>>> <field name="in_id" type="string" indexed="true" stored="true"
>>>> multiValued="true"/>
>>>> <field name="ob_name" type="text" indexed="true" stored="true"
>>>> multiValued="true"/>
>>>>
>>>> <!-- catchall field, containing all other searchable text fields
>>>> (implemented
>>>> via copyField further on in this schema -->
>>>> <field name="text" type="text" indexed="true" stored="false"
>>>> multiValued="true" required="false"/>
>>>>
>>>> <!-- non-tokenized version of manufacturer to make it easier to
>>>> sort
>> or
>>>> group
>>>> results by manufacturer. copied from "manu" via copyField
>>>> -->
>>>> <field name="manu_exact" type="string" indexed="true"
>>>> stored="false"
>>>> required="false"/>
>>>>
>>>>
>>>> <!-- Dynamic field definitions. If a field name is not found,
>>>> dynamicFields
>>>> will be used if the name matches any of the patterns.
>>>> RESTRICTION: the glob-like pattern in the name attribute must
>> have
>>>> a "*" only at the start or the end.
>>>> EXAMPLE: name="*_i" will match any field ending in _i (like
>>> myid_i,
>>>> z_i)
>>>> Longer patterns will be matched first. if equal size
>>>> patterns
>>>> both match, the first appearing in the schema will be
>>>> used. -->
>>>> <dynamicField name="*_i" type="sint" indexed="true"
>>> stored="true"/>
>>>> <dynamicField name="*_s" type="string" indexed="true"
>>> stored="true"/>
>>>> <dynamicField name="*_l" type="slong" indexed="true"
>>> stored="true"/>
>>>> <dynamicField name="*_t" type="text" indexed="true"
>>> stored="true"/>
>>>> <dynamicField name="*_b" type="boolean" indexed="true"
>>> stored="true"/>
>>>> <dynamicField name="*_f" type="sfloat" indexed="true"
>>> stored="true"/>
>>>> <dynamicField name="*_d" type="sdouble" indexed="true"
>>> stored="true"/>
>>>> <dynamicField name="*_dt" type="date" indexed="true"
>>> stored="true"/>
>>>> </fields>
>>>>
>>>> ----------------------------------------------------
>>>> I run the index via firefox browser using
>>>> http://localhost:8080/solr/dataimport?command=full-import
>>>> I checked the status using
>>>> http://localhost:8080/solr/dataimport?command=status
>>>> initially the status increased steadily, but after reaching
>>>> 613071, the
>>>> status stayed for a while (as below), and then it displayed the
>> completed
>>>> message :
>>>> ----------------------------------------------------
>>>> <response>
>>>> -
>>>> <lst name="responseHeader">
>>>> <int name="status">0</int>
>>>> <int name="QTime">1</int>
>>>> </lst>
>>>> -
>>>> <lst name="initArgs">
>>>> -
>>>> <lst name="defaults">
>>>> <str name="config">db-data-config.xml</str>
>>>> </lst>
>>>> </lst>
>>>> <str name="command">status</str>
>>>> <str name="status">busy</str>
>>>> <str name="importResponse">A command is still running...</str>
>>>> -
>>>> <lst name="statusMessages">
>>>> <str name="Time Elapsed">0:3:24.266</str>
>>>> <str name="Total Requests made to DataSource">1</str>
>>>> <str name="Total Rows Fetched">613071</str>
>>>> <str name="Total Documents Processed">613070</str>
>>>> <str name="Total Documents Skipped">0</str>
>>>> <str name="Full Dump Started">2008-11-14 12:12:16</str>
>>>> </lst>
>>>> -
>>>> <str name="WARNING">
>>>> This response format is experimental. It is likely to change in
>>>> the
>>> future.
>>>> </str>
>>>> </response>
>>>>
>>>> -----------------------------------------------------------
>>>>
>>>>>> NOTE: this is the status result after it completed
>>>> -----------------------------------------------------------
>>>>
>>>> <response>
>>>> -
>>>> <lst name="responseHeader">
>>>> <int name="status">0</int>
>>>> <int name="QTime">1</int>
>>>> </lst>
>>>> -
>>>> <lst name="initArgs">
>>>> -
>>>> <lst name="defaults">
>>>> <str name="config">db-data-config.xml</str>
>>>> </lst>
>>>> </lst>
>>>> <str name="command">status</str>
>>>> <str name="status">idle</str>
>>>> <str name="importResponse"/>
>>>> -
>>>> <lst name="statusMessages">
>>>> <str name="Total Requests made to DataSource">1</str>
>>>> <str name="Total Rows Fetched">615360</str>
>>>> <str name="Total Documents Skipped">0</str>
>>>> <str name="Full Dump Started">2008-11-14 12:12:16</str>
>>>> -
>>>> <str name="">
>>>> Indexing completed. Added/Updated: 615360 documents. Deleted 0
>> documents.
>>>> </str>
>>>> <str name="Committed">2008-11-14 12:16:32</str>
>>>> <str name="Optimized">2008-11-14 12:16:32</str>
>>>> <str name="Time taken ">0:4:16.154</str>
>>>> </lst>
>>>> -
>>>> <str name="WARNING">
>>>> This response format is experimental. It is likely to change in
>>>> the
>>> future.
>>>> </str>
>>>> </response>
>>>>
>>>> -----------------------------------------------------
>>>>
>>>> here is the full solr scehma.xml content:
>>>> ----------------------------------------------------
>>>> <?xml version="1.0" ?>
>>>> <!-- The Solr schema file. This file should be named "schema.xml"
>>>> and
>>>> should be in the conf directory under the solr home
>>>> (i.e. ./solr/conf/schema.xml by default)
>>>> or located where the classloader for the Solr webapp can find it.
>>>>
>>>> For more information, on how to customize this file, please see...
>>>> http://wiki.apache.org/solr/SchemaXml
>>>> -->
>>>>
>>>> <schema name="example" version="1.1">
>>>> <types>
>>>> <!-- field type definitions. The "name" attribute is
>>>> just a label to be used by field definitions. The "class"
>>>> attribute and any other attributes determine the real
>>>> behavior of the fieldtype. -->
>>>>
>>>> <!-- The StringField type is not analyzed, but indexed/stored
>> verbatim
>>>> -->
>>>> <fieldtype name="string" class="solr.StrField"
>>> sortMissingLast="true"/>
>>>>
>>>> <!-- boolean type: "true" or "false" -->
>>>> <fieldtype name="boolean" class="solr.BoolField"
>>>> sortMissingLast="true"/>
>>>>
>>>> <!-- The optional sortMissingLast and sortMissingFirst attributes
>> are
>>>> currently supported on types that are sorted internally as a
>>>> strings.
>>>> - If sortMissingLast="true" then a sort on this field will
>>>> cause
>>>> documents
>>>> without the field to come after documents with the field,
>>>> regardless of the requested sort order (asc or desc).
>>>> - If sortMissingFirst="true" then a sort on this field will
>>>> cause
>>>> documents
>>>> without the field to come before documents with the field,
>>>> regardless of the requested sort order.
>>>> - If sortMissingLast="false" and sortMissingFirst="false" (the
>>>> default),
>>>> then default lucene sorting will be used which places docs
>> without
>>>> the field
>>>> first in an ascending sort and last in a descending sort.
>>>> -->
>>>>
>>>> <!-- numeric field types that store and index the text
>>>> value verbatim (and hence don't support range queries
>>>> since the
>>>> lexicographic ordering isn't equal to the numeric
>>>> ordering) -->
>>>> <fieldtype name="integer" class="solr.IntField"/>
>>>> <fieldtype name="long" class="solr.LongField"/>
>>>> <fieldtype name="float" class="solr.FloatField"/>
>>>> <fieldtype name="double" class="solr.DoubleField"/>
>>>>
>>>>
>>>> <!-- Numeric field types that manipulate the value into
>>>> a string value that isn't human readable in it's internal
>>>> form,
>>>> but with a lexicographic ordering the same as the numeric
>>> ordering
>>>> so that range queries correctly work. -->
>>>> <fieldtype name="sint" class="solr.SortableIntField"
>>>> sortMissingLast="true"/>
>>>> <fieldtype name="slong" class="solr.SortableLongField"
>>>> sortMissingLast="true"/>
>>>> <fieldtype name="sfloat" class="solr.SortableFloatField"
>>>> sortMissingLast="true"/>
>>>> <fieldtype name="sdouble" class="solr.SortableDoubleField"
>>>> sortMissingLast="true"/>
>>>>
>>>>
>>>> <!-- The format for this date field is of the form
>>> 1995-12-31T23:59:59Z,
>>>> and
>>>> is a more restricted form of the canonical representation of
>>>> dateTime
>>>> http://www.w3.org/TR/xmlschema-2/#dateTime
>>>> The trailing "Z" designates UTC time and is mandatory.
>>>> Optional fractional seconds are allowed:
>> 1995-12-31T23:59:59.999Z
>>>> All other components are mandatory. -->
>>>> <fieldtype name="date" class="solr.DateField"
>> sortMissingLast="true"/>
>>>>
>>>> <!-- solr.TextField allows the specification of custom text
>> analyzers
>>>> specified as a tokenizer and a list of token filters.
>>>> Different
>>>> analyzers may be specified for indexing and querying.
>>>>
>>>> The optional positionIncrementGap puts space between
>>>> multiple
>>>> fields of
>>>> this type on the same document, with the purpose of
>>>> preventing
>>>> false phrase
>>>> matching across fields.
>>>>
>>>> For more info on customizing your analyzer chain, please
>>>> see...
>>>> http://wiki.apache.org/solr/AnalyzersTokenizersTokenFilters
>>>>
>>>> -->
>>>>
>>>> <!-- Standard analyzer commonly used by Lucene developers
>>>> -->
>>>> <!-- Standard analyzer commonly used by Lucene developers -->
>>>> <fieldtype name="text_lu" class="solr.TextField"
>>>> positionIncrementGap="100">
>>>> <analyzer>
>>>> <tokenizer class="solr.StandardTokenizerFactory"/>
>>>> <filter class="solr.StandardFilterFactory"/>
>>>> <filter class="solr.LowerCaseFilterFactory"/>
>>>> <filter class="solr.StopFilterFactory"/>
>>>> <filter class="solr.EnglishPorterFilterFactory"/>
>>>> </analyzer>
>>>> </fieldtype>
>>>> <!-- One could also specify an existing Analyzer implementation
>>>> in
>>> Java
>>>> via the class attribute on the analyzer element:
>>>> <fieldtype name="text_lu" class="solr.TextField">
>>>> <analyzer
>>>> class="org.apache.lucene.analysis.snowball.SnowballAnalyzer"/>
>>>> </fieldType>
>>>> -->
>>>>
>>>> <!-- A text field that only splits on whitespace for more exact
>>> matching
>>>> -->
>>>> <fieldtype name="text_ws" class="solr.TextField"
>>>> positionIncrementGap="100">
>>>> <analyzer>
>>>> <tokenizer class="solr.WhitespaceTokenizerFactory"/>
>>>> </analyzer>
>>>> </fieldtype>
>>>>
>>>> <!-- A text field that uses WordDelimiterFilter to enable
>>>> splitting
>>> and
>>>> matching of
>>>> words on case-change, alpha numeric boundaries, and
>>> non-alphanumeric
>>>> chars
>>>> so that a query of "wifi" or "wi fi" could match a document
>>>> containing "Wi-Fi".
>>>> Synonyms and stopwords are customized by external files, and
>>>> stemming is enabled -->
>>>> <fieldtype name="text" class="solr.TextField"
>>>> positionIncrementGap="100">
>>>> <analyzer type="index">
>>>> <tokenizer class="solr.WhitespaceTokenizerFactory"/>
>>>> <!-- in this example, we will only use synonyms at query
>>>> time
>>>> <filter class="solr.SynonymFilterFactory"
>>>> synonyms="index_synonyms.txt" ignoreCase="true" expand="false"/>
>>>> -->
>>>> <!--<filter class="solr.WordDelimiterFilterFactory"
>>>> generateWordParts="1"/>-->
>>>> <filter class="solr.StopFilterFactory" ignoreCase="true"/>
>>>> <filter class="solr.LowerCaseFilterFactory"/>
>>>> </analyzer>
>>>> <analyzer type="query">
>>>> <tokenizer class="solr.WhitespaceTokenizerFactory"/>
>>>> <filter class="solr.StopFilterFactory" ignoreCase="true"/>
>>>> <filter class="solr.LowerCaseFilterFactory"/>
>>>> </analyzer>
>>>> </fieldtype>
>>>>
>>>> <!-- Less flexible matching, but less false matches. Probably
>>>> not
>>> ideal
>>>> for product names
>>>> but may be good for SKUs. Can insert dashes in the wrong
>>>> place
>>> and
>>>> still match. -->
>>>> <fieldtype name="textTight" class="solr.TextField"
>>>> positionIncrementGap="100" >
>>>> <analyzer>
>>>> <tokenizer class="solr.WhitespaceTokenizerFactory"/>
>>>> <filter class="solr.SynonymFilterFactory"
>> synonyms="synonyms.txt"
>>>> ignoreCase="true" expand="false"/>
>>>> <filter class="solr.StopFilterFactory" ignoreCase="true"/>
>>>> <filter class="solr.WordDelimiterFilterFactory"
>>>> generateWordParts="0" generateNumberParts="0" catenateWords="1"
>>>> catenateNumbers="1" catenateAll="0"/>
>>>> <filter class="solr.LowerCaseFilterFactory"/>
>>>> <filter class="solr.EnglishPorterFilterFactory"
>>>> protected="protwords.txt"/>
>>>> </analyzer>
>>>> </fieldtype>
>>>> </types>
>>>> <fields>
>>>> <!-- Valid attributes for fields:
>>>> name: mandatory - the name for the field
>>>> type: mandatory - the name of a previously defined type from
>>>> the
>>>> <types> section
>>>> indexed: true if this field should be indexed (searchable)
>>>> stored: true if this field should be retrievable
>>>> multiValued: true if this field may contain multiple values
>>>> per
>>>> document
>>>> omitNorms: (expert) set to true to omit the norms associated
>>>> with
>>>> this field
>>>> (this disables length normalization and index-time
>>>> boosting for the field)
>>>> -->
>>>> <field name="id" type="string" indexed="true" stored="true"
>>>> multiValued="false"/>
>>>> <field name="inst_code" type="text" indexed="true" stored="true"
>>>> multiValued="true" required="false"/>
>>>> <field name="inst_name" type="text" indexed="true" stored="true"
>>>> multiValued="true" required="false"/>
>>>> <field name="meas_name" type="text" indexed="true" stored="true"
>>>> multiValued="true" required="false"/>
>>>> <field name="latitude" type="sfloat" class="solr.FloatField"
>>>> indexed="true" stored="true" required="false"/>
>>>> <field name="longitude" type="sfloat" class="solr.FloatField"
>>>> indexed="true" stored="true" required="false"/>
>>>> <field name="ob_id" type="string" indexed="true" stored="true"
>>>> multiValued="true"/>
>>>> <field name="in_id" type="string" indexed="true" stored="true"
>>>> multiValued="true"/>
>>>> <field name="ob_name" type="text" indexed="true" stored="true"
>>>> multiValued="true"/>
>>>>
>>>> <!-- catchall field, containing all other searchable text fields
>>>> (implemented
>>>> via copyField further on in this schema -->
>>>> <field name="text" type="text" indexed="true" stored="false"
>>>> multiValued="true" required="false"/>
>>>>
>>>>
>>>> <!-- non-tokenized version of manufacturer to make it easier to
>>>> sort
>> or
>>>> group
>>>> results by manufacturer. copied from "manu" via copyField
>>>> -->
>>>> <field name="manu_exact" type="string" indexed="true"
>>>> stored="false"
>>>> required="false"/>
>>>>
>>>>
>>>> <!-- Dynamic field definitions. If a field name is not found,
>>>> dynamicFields
>>>> will be used if the name matches any of the patterns.
>>>> RESTRICTION: the glob-like pattern in the name attribute must
>> have
>>>> a "*" only at the start or the end.
>>>> EXAMPLE: name="*_i" will match any field ending in _i (like
>>> myid_i,
>>>> z_i)
>>>> Longer patterns will be matched first. if equal size
>>>> patterns
>>>> both match, the first appearing in the schema will be
>>>> used. -->
>>>> <dynamicField name="*_i" type="sint" indexed="true"
>>> stored="true"/>
>>>> <dynamicField name="*_s" type="string" indexed="true"
>>> stored="true"/>
>>>> <dynamicField name="*_l" type="slong" indexed="true"
>>> stored="true"/>
>>>> <dynamicField name="*_t" type="text" indexed="true"
>>> stored="true"/>
>>>> <dynamicField name="*_b" type="boolean" indexed="true"
>>> stored="true"/>
>>>> <dynamicField name="*_f" type="sfloat" indexed="true"
>>> stored="true"/>
>>>> <dynamicField name="*_d" type="sdouble" indexed="true"
>>> stored="true"/>
>>>> <dynamicField name="*_dt" type="date" indexed="true"
>>> stored="true"/>
>>>> </fields>
>>>>
>>>> <!-- field to use to determine and enforce document uniqueness. -->
>>>> <uniqueKey>id</uniqueKey>
>>>>
>>>> <!-- field for the QueryParser to use when an explicit fieldname is
>>> absent
>>>> -->
>>>> <defaultSearchField>text</defaultSearchField>
>>>>
>>>> <!-- SolrQueryParser configuration: defaultOperator="AND|OR" -->
>>>> <solrQueryParser defaultOperator="AND"/>
>>>>
>>>> <!-- copyField commands copy one field to another at the time a
>> document
>>>> is added to the index. It's used either to index the same
>>>> field
>>>> different
>>>> ways, or to add multiple fields to the same field for
>>> easier/faster
>>>> searching. -->
>>>>
>>>>
>>>>
>>>> <!-- Similarity is the scoring routine for each document vs a
>>>> query.
>>>> A custom similarity may be specified here, but the default is
>>>> fine
>>>> for most applications. -->
>>>> <!-- <similarity
>>>> class="org.apache.lucene.search.DefaultSimilarity"/>
>>> -->
>>>>
>>>> </schema>
>>>>
>>>
>> -------------------------------------------------------------------------------------------------------------------------------------------------------------
>>>>
>>>>
>>>> On Wed, Nov 12, 2008 at 11:01 PM, Noble Paul നോബിള്
>>>> नोब्ळ् <
>>>> noble.paul@gmail.com> wrote:
>>>>
>>>>> the fact that it got committed in the end suggests there was no
>>>>> error
>> in
>>>>> between
>>>>>
>>>>> look at the status url and see the no:of rows returned etc.
>>>>>
>>>>> It gives a clue as to what would have really happened. or you can
>>>>> paste your dataconfig and status xmls and we may be able to
>>>>> suggest
>>>>> something
>>>>>
>>>>> On Thu, Nov 13, 2008 at 9:26 AM, Giri <gi...@gmail.com>
>>>>> wrote:
>>>>>> Hi Noble,
>>>>>>
>>>>>> thanks for reply, my comments are below
>>>>>>
>>>>>>>> why is the id field multivalued?
>>>>>> I was just trying various options, yes, this ID is unique, and I
>> check
>>>>> for
>>>>>> duplicates, when I did a distinct (id) query to the MySQL
>>>>>> database,
>> it
>>>>>> returned almost 2 million.
>>>>>>
>>>>>>>> look at the status host:post/dataimport gives you the status
>>>>>> I constantly checked the status using the dataimport URL, the
>>> status
>>>>> was
>>>>>> increased upto 600K records, then it stopped increasing, then
>>>>>> took
>> few
>>>>>> minutes to commit the indexed data.
>>>>>>
>>>>>>
>>>>>> On Tue, Nov 11, 2008 at 11:35 PM, Noble Paul
>>>>>> നോബിള് नोब्ळ् <
>>>>>> noble.paul@gmail.com> wrote:
>>>>>>
>>>>>>> why is the id field multivalued? is there a uniqueKey in the
>>>>>>> schema
>> ?
>>>>>>> Are you sure there are no duplicates?
>>>>>>>
>>>>>>> look at the status host:post/dataimport gives you the status
>>>>>>> it can give you some clue
>>>>>>>
>>>>>>> --Noble
>>>>>>>
>>>>>>>
>>>>>>> On Wed, Nov 12, 2008 at 4:53 AM, Giri <gi...@gmail.com>
>> wrote:
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> I have about ~ 2 million records in a mySQL database table
>>>>>>>> (about
>> 9
>>>>>>> fields
>>>>>>>> from a single table), and I am trying to load it to the solr
>> using
>>>>>>>> DataImportHandler using the command=full-import option. it only
>>>>> indexed
>>>>>>>> about 615360 records out of 2 millions.
>>>>>>>>
>>>>>>>> here is my db-data-config.xml
>>>>>>>> <dataConfig>
>>>>>>>> <dataSource type="JdbcDataSource"
>> driver="com.mysql.jdbc.Driver"
>>>>>>>> url="jdbc:mysql://localhost:3306/mydb" user="ua" password="pw"
>>>>> batchSize
>>>>>>>> ="-1"/>
>>>>>>>> <document name="climate">
>>>>>>>> <entity name="occurence" query="select * from
>> mylargetable">
>>>>>>>> <field column="id" name="id" />
>>>>>>>> <field column="title" name="title" />
>>>>>>>> <field column="url" name="url" />
>>>>>>>> </entity>
>>>>>>>> </document>
>>>>>>>> </dataConfig>
>>>>>>>>
>>>>>>>> and in my solr schema.xml, i define these fields as:
>>>>>>>>
>>>>>>>> <field name="id" type="string" indexed="true" stored="true"
>>>>>>>> multiValued="true"/>
>>>>>>>> <field name="title" type="text" indexed="true" stored="true"
>>>>>>>> multiValued="true" required="false"/>
>>>>>>>> <field name="url" type="text" indexed="true" stored="true"
>>>>>>>> multiValued="true" required="false"/>
>>>>>>>>
>>>>>>>>
>>>>>>>> If I try to index just one field (id), then it indexes about
>> 960000
>>>>>>> records,
>>>>>>>> but if I try to index all the above three fields, it indexes
>>>>>>>> only
>>>>> 615360
>>>>>>>> records.
>>>>>>>>
>>>>>>>> Any help will be appreciated.
>>>>>>>>
>>>>>>>> thanks!
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> --Noble Paul
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> --Noble Paul
>>>>>
>>>>
>>>
>>>
>>>
>>> --
>>> --Noble Paul
>>>
>>
>
>
>
> --
> Regards,
> Shalin Shekhar Mangar.
Re: DataImportHandler not indexing all the records
Posted by Ahmed Hammad <ah...@gmail.com>.
Thanks Shalin,
I have added a new field type in my schema as following:
<!-- A text field that only splits on whitespace ans strip off HTML tags
-->
<fieldType name="text_html" class="solr.TextField"
positionIncrementGap="100">
<analyzer>
<tokenizer class="solr.HTMLStripWhitespaceTokenizerFactory" />
</analyzer>
</fieldType>
and added my field
<field name="content" type="text_html" indexed="true" stored="true"
required="false" />
After restarting Solr, and making full-import, everything work just fine.
Many thanks.
Regards,
Ahmed
My best wishes,
Regards,
Ahmed Hammad
On Sat, Nov 15, 2008 at 9:21 PM, Shalin Shekhar Mangar <
shalinmangar@gmail.com> wrote:
> I think the problem is that DIH catches Exception but not Error so a
> StackOverFlowError will slip past it. Normally, the SolrDispatchFilter will
> log such errors but the import is performed in a new thread, so the error
> is
> not logged anywhere. However, DIH will not commit documents in this case
> (and there is no mention of a commit in your DIH status).
>
> We should change the catch clause to catch Throwable so that this is not
> repeated. I'll open an issue and give a patch.
>
> Btw, Ahmed, Solr has a Tokenizer which is much better at striping html --
> HTMLStripWhitespaceTokenizerFactory which you can use for such tasks.
>
> On Sun, Nov 16, 2008 at 12:30 AM, Ahmed Hammad <ah...@gmail.com> wrote:
>
> > I had a similar problem like Giri. I have 17,000 record in one table and
> > DIH
> > can import only 12464.
> >
> > After some investigation, I found my problem.
> >
> > I have a regular expression to strip off html tags form input text, as
> > following:
> >
> > <field sourceColName="content" column="content" regex="<(.|\n)*?>"
> > replaceWith=" "/>
> >
> > The DIH RegEx have stack overflow on the record 17,000 due to error in
> the
> > content and then DIH exit without any error in the log on in the status
> > command. Here is the status:
> >
> > <lst name="statusMessages">
> > <str name="Time Elapsed">0:0:31.657</str>
> > <str name="Total Requests made to DataSource">1</str>
> > <str name="Total Rows Fetched">12464</str>
> > <str name="Total Documents Processed">12464</str>
> > <str name="Total Documents Skipped">0</str>
> > <str name="Full Dump Started">2008-11-15 20:40:58</str>
> > </lst>
> >
> > I found the error in Eclipse Console window while debugging; it was a
> stack
> > overflow in the RegEx library.
> >
> > The problem is that, DIH does not show any problem in log file on in
> status
> > message.
> > What I think is important is to show whatever error happen in the log
> file.
> >
> > I noticed also that, in case of no error a log message show completness:
> >
> > Nov 15, 2008 8:57:34 PM org.apache.solr.handler.dataimport.DocBuilder
> > execute
> > INFO: Time taken = 0:0:40.656
> >
> > In case of RegEx stack overflow error, this log message does not appear.
> >
> > I am researching on how to catch such error in DIH. Any ideas?
> >
> >
> > Regards,
> > ahmd
> >
> > On Sat, Nov 15, 2008 at 6:32 AM, Noble Paul നോബിള് नोब्ळ् <
> > noble.paul@gmail.com> wrote:
> >
> > > There is no obvious problem
> > >
> > > I can be reasonably sure that
> > > the query
> > >
> > > select * from climatedata.ws_record limit 1000000
> > >
> > > would have fetched only 615360 rows.
> > > This is a very reliable pice of information
> > > <str name="Total Rows Fetched">615360</str>
> > >
> > > On Sat, Nov 15, 2008 at 12:41 AM, Giri <gi...@gmail.com> wrote:
> > > > Hi Noble,
> > > > thanks for the help, here are the details: the field "id" is unique,
> > when
> > > I
> > > > did a select distinct(id), it returned 1 million rows.
> > > >
> > > > -------------------------------------------------------------------
> > > > db-data-config.xml
> > > > note: I limit the resultset to 1 million in the select query
> > > > -------------------------------------------------------------------
> > > > <dataConfig>
> > > > <dataSource type="JdbcDataSource" driver="com.mysql.jdbc.Driver"
> > > > url="jdbc:mysql://localhost:3306/climatedata" user="user"
> password="pw"
> > > > batchSize ="-1"/>
> > > > <document name="climateRecord">
> > > > <entity name="observation" query="select * from
> > > > climatedata.ws_record limit 1000000">
> > > > <field column="id" name="id" />
> > > > <field column="inst_code" name="inst_code" />
> > > > <field column="inst_name" name="inst_name" />
> > > > <field column="meas_name" name="meas_name" />
> > > > <field column="latitude" name="latitude" />
> > > > <field column="longitude" name="longitude" />
> > > > <field column="ob_id" name="ob_id" />
> > > > <field column="in_id" name="in_id" />
> > > > <field column="ob_name" name="ob_name" />
> > > > </entity>
> > > > </document>
> > > > </dataConfig>
> > > >
> > > > -----------------------------------------------------------------
> > > > in the solr Schema.xml:
> > > > ----------------------------------------------------------------
> > > > <fields>
> > > > <field name="id" type="string" indexed="true" stored="true"
> > > > multiValued="false"/>
> > > > <field name="inst_code" type="text" indexed="true" stored="true"
> > > > multiValued="true" required="false"/>
> > > > <field name="inst_name" type="text" indexed="true" stored="true"
> > > > multiValued="true" required="false"/>
> > > > <field name="meas_name" type="text" indexed="true" stored="true"
> > > > multiValued="true" required="false"/>
> > > > <field name="latitude" type="sfloat" class="solr.FloatField"
> > > > indexed="true" stored="true" required="false"/>
> > > > <field name="longitude" type="sfloat" class="solr.FloatField"
> > > > indexed="true" stored="true" required="false"/>
> > > > <field name="ob_id" type="string" indexed="true" stored="true"
> > > > multiValued="true"/>
> > > > <field name="in_id" type="string" indexed="true" stored="true"
> > > > multiValued="true"/>
> > > > <field name="ob_name" type="text" indexed="true" stored="true"
> > > > multiValued="true"/>
> > > >
> > > > <!-- catchall field, containing all other searchable text fields
> > > > (implemented
> > > > via copyField further on in this schema -->
> > > > <field name="text" type="text" indexed="true" stored="false"
> > > > multiValued="true" required="false"/>
> > > >
> > > > <!-- non-tokenized version of manufacturer to make it easier to
> sort
> > or
> > > > group
> > > > results by manufacturer. copied from "manu" via copyField -->
> > > > <field name="manu_exact" type="string" indexed="true"
> stored="false"
> > > > required="false"/>
> > > >
> > > >
> > > > <!-- Dynamic field definitions. If a field name is not found,
> > > > dynamicFields
> > > > will be used if the name matches any of the patterns.
> > > > RESTRICTION: the glob-like pattern in the name attribute must
> > have
> > > > a "*" only at the start or the end.
> > > > EXAMPLE: name="*_i" will match any field ending in _i (like
> > > myid_i,
> > > > z_i)
> > > > Longer patterns will be matched first. if equal size patterns
> > > > both match, the first appearing in the schema will be used.
> -->
> > > > <dynamicField name="*_i" type="sint" indexed="true"
> > > stored="true"/>
> > > > <dynamicField name="*_s" type="string" indexed="true"
> > > stored="true"/>
> > > > <dynamicField name="*_l" type="slong" indexed="true"
> > > stored="true"/>
> > > > <dynamicField name="*_t" type="text" indexed="true"
> > > stored="true"/>
> > > > <dynamicField name="*_b" type="boolean" indexed="true"
> > > stored="true"/>
> > > > <dynamicField name="*_f" type="sfloat" indexed="true"
> > > stored="true"/>
> > > > <dynamicField name="*_d" type="sdouble" indexed="true"
> > > stored="true"/>
> > > > <dynamicField name="*_dt" type="date" indexed="true"
> > > stored="true"/>
> > > > </fields>
> > > >
> > > > ----------------------------------------------------
> > > > I run the index via firefox browser using
> > > > http://localhost:8080/solr/dataimport?command=full-import
> > > > I checked the status using
> > > > http://localhost:8080/solr/dataimport?command=status
> > > > initially the status increased steadily, but after reaching 613071,
> the
> > > > status stayed for a while (as below), and then it displayed the
> > completed
> > > > message :
> > > > ----------------------------------------------------
> > > > <response>
> > > > -
> > > > <lst name="responseHeader">
> > > > <int name="status">0</int>
> > > > <int name="QTime">1</int>
> > > > </lst>
> > > > -
> > > > <lst name="initArgs">
> > > > -
> > > > <lst name="defaults">
> > > > <str name="config">db-data-config.xml</str>
> > > > </lst>
> > > > </lst>
> > > > <str name="command">status</str>
> > > > <str name="status">busy</str>
> > > > <str name="importResponse">A command is still running...</str>
> > > > -
> > > > <lst name="statusMessages">
> > > > <str name="Time Elapsed">0:3:24.266</str>
> > > > <str name="Total Requests made to DataSource">1</str>
> > > > <str name="Total Rows Fetched">613071</str>
> > > > <str name="Total Documents Processed">613070</str>
> > > > <str name="Total Documents Skipped">0</str>
> > > > <str name="Full Dump Started">2008-11-14 12:12:16</str>
> > > > </lst>
> > > > -
> > > > <str name="WARNING">
> > > > This response format is experimental. It is likely to change in the
> > > future.
> > > > </str>
> > > > </response>
> > > >
> > > > -----------------------------------------------------------
> > > >
> > > >>>NOTE: this is the status result after it completed
> > > > -----------------------------------------------------------
> > > >
> > > > <response>
> > > > -
> > > > <lst name="responseHeader">
> > > > <int name="status">0</int>
> > > > <int name="QTime">1</int>
> > > > </lst>
> > > > -
> > > > <lst name="initArgs">
> > > > -
> > > > <lst name="defaults">
> > > > <str name="config">db-data-config.xml</str>
> > > > </lst>
> > > > </lst>
> > > > <str name="command">status</str>
> > > > <str name="status">idle</str>
> > > > <str name="importResponse"/>
> > > > -
> > > > <lst name="statusMessages">
> > > > <str name="Total Requests made to DataSource">1</str>
> > > > <str name="Total Rows Fetched">615360</str>
> > > > <str name="Total Documents Skipped">0</str>
> > > > <str name="Full Dump Started">2008-11-14 12:12:16</str>
> > > > -
> > > > <str name="">
> > > > Indexing completed. Added/Updated: 615360 documents. Deleted 0
> > documents.
> > > > </str>
> > > > <str name="Committed">2008-11-14 12:16:32</str>
> > > > <str name="Optimized">2008-11-14 12:16:32</str>
> > > > <str name="Time taken ">0:4:16.154</str>
> > > > </lst>
> > > > -
> > > > <str name="WARNING">
> > > > This response format is experimental. It is likely to change in the
> > > future.
> > > > </str>
> > > > </response>
> > > >
> > > > -----------------------------------------------------
> > > >
> > > > here is the full solr scehma.xml content:
> > > > ----------------------------------------------------
> > > > <?xml version="1.0" ?>
> > > > <!-- The Solr schema file. This file should be named "schema.xml" and
> > > > should be in the conf directory under the solr home
> > > > (i.e. ./solr/conf/schema.xml by default)
> > > > or located where the classloader for the Solr webapp can find it.
> > > >
> > > > For more information, on how to customize this file, please see...
> > > > http://wiki.apache.org/solr/SchemaXml
> > > > -->
> > > >
> > > > <schema name="example" version="1.1">
> > > > <types>
> > > > <!-- field type definitions. The "name" attribute is
> > > > just a label to be used by field definitions. The "class"
> > > > attribute and any other attributes determine the real
> > > > behavior of the fieldtype. -->
> > > >
> > > > <!-- The StringField type is not analyzed, but indexed/stored
> > verbatim
> > > > -->
> > > > <fieldtype name="string" class="solr.StrField"
> > > sortMissingLast="true"/>
> > > >
> > > > <!-- boolean type: "true" or "false" -->
> > > > <fieldtype name="boolean" class="solr.BoolField"
> > > > sortMissingLast="true"/>
> > > >
> > > > <!-- The optional sortMissingLast and sortMissingFirst attributes
> > are
> > > > currently supported on types that are sorted internally as a
> > > > strings.
> > > > - If sortMissingLast="true" then a sort on this field will
> cause
> > > > documents
> > > > without the field to come after documents with the field,
> > > > regardless of the requested sort order (asc or desc).
> > > > - If sortMissingFirst="true" then a sort on this field will
> cause
> > > > documents
> > > > without the field to come before documents with the field,
> > > > regardless of the requested sort order.
> > > > - If sortMissingLast="false" and sortMissingFirst="false" (the
> > > > default),
> > > > then default lucene sorting will be used which places docs
> > without
> > > > the field
> > > > first in an ascending sort and last in a descending sort.
> > > > -->
> > > >
> > > > <!-- numeric field types that store and index the text
> > > > value verbatim (and hence don't support range queries since
> the
> > > > lexicographic ordering isn't equal to the numeric ordering)
> -->
> > > > <fieldtype name="integer" class="solr.IntField"/>
> > > > <fieldtype name="long" class="solr.LongField"/>
> > > > <fieldtype name="float" class="solr.FloatField"/>
> > > > <fieldtype name="double" class="solr.DoubleField"/>
> > > >
> > > >
> > > > <!-- Numeric field types that manipulate the value into
> > > > a string value that isn't human readable in it's internal
> form,
> > > > but with a lexicographic ordering the same as the numeric
> > > ordering
> > > > so that range queries correctly work. -->
> > > > <fieldtype name="sint" class="solr.SortableIntField"
> > > > sortMissingLast="true"/>
> > > > <fieldtype name="slong" class="solr.SortableLongField"
> > > > sortMissingLast="true"/>
> > > > <fieldtype name="sfloat" class="solr.SortableFloatField"
> > > > sortMissingLast="true"/>
> > > > <fieldtype name="sdouble" class="solr.SortableDoubleField"
> > > > sortMissingLast="true"/>
> > > >
> > > >
> > > > <!-- The format for this date field is of the form
> > > 1995-12-31T23:59:59Z,
> > > > and
> > > > is a more restricted form of the canonical representation of
> > > > dateTime
> > > > http://www.w3.org/TR/xmlschema-2/#dateTime
> > > > The trailing "Z" designates UTC time and is mandatory.
> > > > Optional fractional seconds are allowed:
> > 1995-12-31T23:59:59.999Z
> > > > All other components are mandatory. -->
> > > > <fieldtype name="date" class="solr.DateField"
> > sortMissingLast="true"/>
> > > >
> > > > <!-- solr.TextField allows the specification of custom text
> > analyzers
> > > > specified as a tokenizer and a list of token filters.
> Different
> > > > analyzers may be specified for indexing and querying.
> > > >
> > > > The optional positionIncrementGap puts space between multiple
> > > > fields of
> > > > this type on the same document, with the purpose of
> preventing
> > > > false phrase
> > > > matching across fields.
> > > >
> > > > For more info on customizing your analyzer chain, please
> see...
> > > > http://wiki.apache.org/solr/AnalyzersTokenizersTokenFilters
> > > >
> > > > -->
> > > >
> > > > <!-- Standard analyzer commonly used by Lucene developers
> > > > -->
> > > > <!-- Standard analyzer commonly used by Lucene developers -->
> > > > <fieldtype name="text_lu" class="solr.TextField"
> > > > positionIncrementGap="100">
> > > > <analyzer>
> > > > <tokenizer class="solr.StandardTokenizerFactory"/>
> > > > <filter class="solr.StandardFilterFactory"/>
> > > > <filter class="solr.LowerCaseFilterFactory"/>
> > > > <filter class="solr.StopFilterFactory"/>
> > > > <filter class="solr.EnglishPorterFilterFactory"/>
> > > > </analyzer>
> > > > </fieldtype>
> > > > <!-- One could also specify an existing Analyzer implementation in
> > > Java
> > > > via the class attribute on the analyzer element:
> > > > <fieldtype name="text_lu" class="solr.TextField">
> > > > <analyzer
> > > > class="org.apache.lucene.analysis.snowball.SnowballAnalyzer"/>
> > > > </fieldType>
> > > > -->
> > > >
> > > > <!-- A text field that only splits on whitespace for more exact
> > > matching
> > > > -->
> > > > <fieldtype name="text_ws" class="solr.TextField"
> > > > positionIncrementGap="100">
> > > > <analyzer>
> > > > <tokenizer class="solr.WhitespaceTokenizerFactory"/>
> > > > </analyzer>
> > > > </fieldtype>
> > > >
> > > > <!-- A text field that uses WordDelimiterFilter to enable
> splitting
> > > and
> > > > matching of
> > > > words on case-change, alpha numeric boundaries, and
> > > non-alphanumeric
> > > > chars
> > > > so that a query of "wifi" or "wi fi" could match a document
> > > > containing "Wi-Fi".
> > > > Synonyms and stopwords are customized by external files, and
> > > > stemming is enabled -->
> > > > <fieldtype name="text" class="solr.TextField"
> > > > positionIncrementGap="100">
> > > > <analyzer type="index">
> > > > <tokenizer class="solr.WhitespaceTokenizerFactory"/>
> > > > <!-- in this example, we will only use synonyms at query
> time
> > > > <filter class="solr.SynonymFilterFactory"
> > > > synonyms="index_synonyms.txt" ignoreCase="true" expand="false"/>
> > > > -->
> > > > <!--<filter class="solr.WordDelimiterFilterFactory"
> > > > generateWordParts="1"/>-->
> > > > <filter class="solr.StopFilterFactory" ignoreCase="true"/>
> > > > <filter class="solr.LowerCaseFilterFactory"/>
> > > > </analyzer>
> > > > <analyzer type="query">
> > > > <tokenizer class="solr.WhitespaceTokenizerFactory"/>
> > > > <filter class="solr.StopFilterFactory" ignoreCase="true"/>
> > > > <filter class="solr.LowerCaseFilterFactory"/>
> > > > </analyzer>
> > > > </fieldtype>
> > > >
> > > > <!-- Less flexible matching, but less false matches. Probably not
> > > ideal
> > > > for product names
> > > > but may be good for SKUs. Can insert dashes in the wrong
> place
> > > and
> > > > still match. -->
> > > > <fieldtype name="textTight" class="solr.TextField"
> > > > positionIncrementGap="100" >
> > > > <analyzer>
> > > > <tokenizer class="solr.WhitespaceTokenizerFactory"/>
> > > > <filter class="solr.SynonymFilterFactory"
> > synonyms="synonyms.txt"
> > > > ignoreCase="true" expand="false"/>
> > > > <filter class="solr.StopFilterFactory" ignoreCase="true"/>
> > > > <filter class="solr.WordDelimiterFilterFactory"
> > > > generateWordParts="0" generateNumberParts="0" catenateWords="1"
> > > > catenateNumbers="1" catenateAll="0"/>
> > > > <filter class="solr.LowerCaseFilterFactory"/>
> > > > <filter class="solr.EnglishPorterFilterFactory"
> > > > protected="protwords.txt"/>
> > > > </analyzer>
> > > > </fieldtype>
> > > > </types>
> > > > <fields>
> > > > <!-- Valid attributes for fields:
> > > > name: mandatory - the name for the field
> > > > type: mandatory - the name of a previously defined type from
> the
> > > > <types> section
> > > > indexed: true if this field should be indexed (searchable)
> > > > stored: true if this field should be retrievable
> > > > multiValued: true if this field may contain multiple values per
> > > > document
> > > > omitNorms: (expert) set to true to omit the norms associated
> with
> > > > this field
> > > > (this disables length normalization and index-time
> > > > boosting for the field)
> > > > -->
> > > > <field name="id" type="string" indexed="true" stored="true"
> > > > multiValued="false"/>
> > > > <field name="inst_code" type="text" indexed="true" stored="true"
> > > > multiValued="true" required="false"/>
> > > > <field name="inst_name" type="text" indexed="true" stored="true"
> > > > multiValued="true" required="false"/>
> > > > <field name="meas_name" type="text" indexed="true" stored="true"
> > > > multiValued="true" required="false"/>
> > > > <field name="latitude" type="sfloat" class="solr.FloatField"
> > > > indexed="true" stored="true" required="false"/>
> > > > <field name="longitude" type="sfloat" class="solr.FloatField"
> > > > indexed="true" stored="true" required="false"/>
> > > > <field name="ob_id" type="string" indexed="true" stored="true"
> > > > multiValued="true"/>
> > > > <field name="in_id" type="string" indexed="true" stored="true"
> > > > multiValued="true"/>
> > > > <field name="ob_name" type="text" indexed="true" stored="true"
> > > > multiValued="true"/>
> > > >
> > > > <!-- catchall field, containing all other searchable text fields
> > > > (implemented
> > > > via copyField further on in this schema -->
> > > > <field name="text" type="text" indexed="true" stored="false"
> > > > multiValued="true" required="false"/>
> > > >
> > > >
> > > > <!-- non-tokenized version of manufacturer to make it easier to
> sort
> > or
> > > > group
> > > > results by manufacturer. copied from "manu" via copyField -->
> > > > <field name="manu_exact" type="string" indexed="true"
> stored="false"
> > > > required="false"/>
> > > >
> > > >
> > > > <!-- Dynamic field definitions. If a field name is not found,
> > > > dynamicFields
> > > > will be used if the name matches any of the patterns.
> > > > RESTRICTION: the glob-like pattern in the name attribute must
> > have
> > > > a "*" only at the start or the end.
> > > > EXAMPLE: name="*_i" will match any field ending in _i (like
> > > myid_i,
> > > > z_i)
> > > > Longer patterns will be matched first. if equal size patterns
> > > > both match, the first appearing in the schema will be used.
> -->
> > > > <dynamicField name="*_i" type="sint" indexed="true"
> > > stored="true"/>
> > > > <dynamicField name="*_s" type="string" indexed="true"
> > > stored="true"/>
> > > > <dynamicField name="*_l" type="slong" indexed="true"
> > > stored="true"/>
> > > > <dynamicField name="*_t" type="text" indexed="true"
> > > stored="true"/>
> > > > <dynamicField name="*_b" type="boolean" indexed="true"
> > > stored="true"/>
> > > > <dynamicField name="*_f" type="sfloat" indexed="true"
> > > stored="true"/>
> > > > <dynamicField name="*_d" type="sdouble" indexed="true"
> > > stored="true"/>
> > > > <dynamicField name="*_dt" type="date" indexed="true"
> > > stored="true"/>
> > > > </fields>
> > > >
> > > > <!-- field to use to determine and enforce document uniqueness. -->
> > > > <uniqueKey>id</uniqueKey>
> > > >
> > > > <!-- field for the QueryParser to use when an explicit fieldname is
> > > absent
> > > > -->
> > > > <defaultSearchField>text</defaultSearchField>
> > > >
> > > > <!-- SolrQueryParser configuration: defaultOperator="AND|OR" -->
> > > > <solrQueryParser defaultOperator="AND"/>
> > > >
> > > > <!-- copyField commands copy one field to another at the time a
> > document
> > > > is added to the index. It's used either to index the same
> field
> > > > different
> > > > ways, or to add multiple fields to the same field for
> > > easier/faster
> > > > searching. -->
> > > >
> > > >
> > > >
> > > > <!-- Similarity is the scoring routine for each document vs a query.
> > > > A custom similarity may be specified here, but the default is
> fine
> > > > for most applications. -->
> > > > <!-- <similarity
> class="org.apache.lucene.search.DefaultSimilarity"/>
> > > -->
> > > >
> > > > </schema>
> > > >
> > >
> >
> -------------------------------------------------------------------------------------------------------------------------------------------------------------
> > > >
> > > >
> > > > On Wed, Nov 12, 2008 at 11:01 PM, Noble Paul നോബിള് नोब्ळ् <
> > > > noble.paul@gmail.com> wrote:
> > > >
> > > >> the fact that it got committed in the end suggests there was no
> error
> > in
> > > >> between
> > > >>
> > > >> look at the status url and see the no:of rows returned etc.
> > > >>
> > > >> It gives a clue as to what would have really happened. or you can
> > > >> paste your dataconfig and status xmls and we may be able to suggest
> > > >> something
> > > >>
> > > >> On Thu, Nov 13, 2008 at 9:26 AM, Giri <gi...@gmail.com>
> wrote:
> > > >> > Hi Noble,
> > > >> >
> > > >> > thanks for reply, my comments are below
> > > >> >
> > > >> >>>why is the id field multivalued?
> > > >> > I was just trying various options, yes, this ID is unique, and I
> > check
> > > >> for
> > > >> > duplicates, when I did a distinct (id) query to the MySQL
> database,
> > it
> > > >> > returned almost 2 million.
> > > >> >
> > > >> >>> look at the status host:post/dataimport gives you the status
> > > >> > I constantly checked the status using the dataimport URL, the
> > > status
> > > >> was
> > > >> > increased upto 600K records, then it stopped increasing, then took
> > few
> > > >> > minutes to commit the indexed data.
> > > >> >
> > > >> >
> > > >> > On Tue, Nov 11, 2008 at 11:35 PM, Noble Paul നോബിള് नोब्ळ् <
> > > >> > noble.paul@gmail.com> wrote:
> > > >> >
> > > >> >> why is the id field multivalued? is there a uniqueKey in the
> schema
> > ?
> > > >> >> Are you sure there are no duplicates?
> > > >> >>
> > > >> >> look at the status host:post/dataimport gives you the status
> > > >> >> it can give you some clue
> > > >> >>
> > > >> >> --Noble
> > > >> >>
> > > >> >>
> > > >> >> On Wed, Nov 12, 2008 at 4:53 AM, Giri <gi...@gmail.com>
> > wrote:
> > > >> >> > Hi,
> > > >> >> >
> > > >> >> > I have about ~ 2 million records in a mySQL database table
> (about
> > 9
> > > >> >> fields
> > > >> >> > from a single table), and I am trying to load it to the solr
> > using
> > > >> >> > DataImportHandler using the command=full-import option. it only
> > > >> indexed
> > > >> >> > about 615360 records out of 2 millions.
> > > >> >> >
> > > >> >> > here is my db-data-config.xml
> > > >> >> > <dataConfig>
> > > >> >> > <dataSource type="JdbcDataSource"
> > driver="com.mysql.jdbc.Driver"
> > > >> >> > url="jdbc:mysql://localhost:3306/mydb" user="ua" password="pw"
> > > >> batchSize
> > > >> >> > ="-1"/>
> > > >> >> > <document name="climate">
> > > >> >> > <entity name="occurence" query="select * from
> > mylargetable">
> > > >> >> > <field column="id" name="id" />
> > > >> >> > <field column="title" name="title" />
> > > >> >> > <field column="url" name="url" />
> > > >> >> > </entity>
> > > >> >> > </document>
> > > >> >> > </dataConfig>
> > > >> >> >
> > > >> >> > and in my solr schema.xml, i define these fields as:
> > > >> >> >
> > > >> >> > <field name="id" type="string" indexed="true" stored="true"
> > > >> >> > multiValued="true"/>
> > > >> >> > <field name="title" type="text" indexed="true" stored="true"
> > > >> >> > multiValued="true" required="false"/>
> > > >> >> > <field name="url" type="text" indexed="true" stored="true"
> > > >> >> > multiValued="true" required="false"/>
> > > >> >> >
> > > >> >> >
> > > >> >> > If I try to index just one field (id), then it indexes about
> > 960000
> > > >> >> records,
> > > >> >> > but if I try to index all the above three fields, it indexes
> only
> > > >> 615360
> > > >> >> > records.
> > > >> >> >
> > > >> >> > Any help will be appreciated.
> > > >> >> >
> > > >> >> > thanks!
> > > >> >> >
> > > >> >>
> > > >> >>
> > > >> >>
> > > >> >> --
> > > >> >> --Noble Paul
> > > >> >>
> > > >> >
> > > >>
> > > >>
> > > >>
> > > >> --
> > > >> --Noble Paul
> > > >>
> > > >
> > >
> > >
> > >
> > > --
> > > --Noble Paul
> > >
> >
>
>
>
> --
> Regards,
> Shalin Shekhar Mangar.
>
Re: DataImportHandler not indexing all the records
Posted by Shalin Shekhar Mangar <sh...@gmail.com>.
I think the problem is that DIH catches Exception but not Error so a
StackOverFlowError will slip past it. Normally, the SolrDispatchFilter will
log such errors but the import is performed in a new thread, so the error is
not logged anywhere. However, DIH will not commit documents in this case
(and there is no mention of a commit in your DIH status).
We should change the catch clause to catch Throwable so that this is not
repeated. I'll open an issue and give a patch.
Btw, Ahmed, Solr has a Tokenizer which is much better at striping html --
HTMLStripWhitespaceTokenizerFactory which you can use for such tasks.
On Sun, Nov 16, 2008 at 12:30 AM, Ahmed Hammad <ah...@gmail.com> wrote:
> I had a similar problem like Giri. I have 17,000 record in one table and
> DIH
> can import only 12464.
>
> After some investigation, I found my problem.
>
> I have a regular expression to strip off html tags form input text, as
> following:
>
> <field sourceColName="content" column="content" regex="<(.|\n)*?>"
> replaceWith=" "/>
>
> The DIH RegEx have stack overflow on the record 17,000 due to error in the
> content and then DIH exit without any error in the log on in the status
> command. Here is the status:
>
> <lst name="statusMessages">
> <str name="Time Elapsed">0:0:31.657</str>
> <str name="Total Requests made to DataSource">1</str>
> <str name="Total Rows Fetched">12464</str>
> <str name="Total Documents Processed">12464</str>
> <str name="Total Documents Skipped">0</str>
> <str name="Full Dump Started">2008-11-15 20:40:58</str>
> </lst>
>
> I found the error in Eclipse Console window while debugging; it was a stack
> overflow in the RegEx library.
>
> The problem is that, DIH does not show any problem in log file on in status
> message.
> What I think is important is to show whatever error happen in the log file.
>
> I noticed also that, in case of no error a log message show completness:
>
> Nov 15, 2008 8:57:34 PM org.apache.solr.handler.dataimport.DocBuilder
> execute
> INFO: Time taken = 0:0:40.656
>
> In case of RegEx stack overflow error, this log message does not appear.
>
> I am researching on how to catch such error in DIH. Any ideas?
>
>
> Regards,
> ahmd
>
> On Sat, Nov 15, 2008 at 6:32 AM, Noble Paul നോബിള് नोब्ळ् <
> noble.paul@gmail.com> wrote:
>
> > There is no obvious problem
> >
> > I can be reasonably sure that
> > the query
> >
> > select * from climatedata.ws_record limit 1000000
> >
> > would have fetched only 615360 rows.
> > This is a very reliable pice of information
> > <str name="Total Rows Fetched">615360</str>
> >
> > On Sat, Nov 15, 2008 at 12:41 AM, Giri <gi...@gmail.com> wrote:
> > > Hi Noble,
> > > thanks for the help, here are the details: the field "id" is unique,
> when
> > I
> > > did a select distinct(id), it returned 1 million rows.
> > >
> > > -------------------------------------------------------------------
> > > db-data-config.xml
> > > note: I limit the resultset to 1 million in the select query
> > > -------------------------------------------------------------------
> > > <dataConfig>
> > > <dataSource type="JdbcDataSource" driver="com.mysql.jdbc.Driver"
> > > url="jdbc:mysql://localhost:3306/climatedata" user="user" password="pw"
> > > batchSize ="-1"/>
> > > <document name="climateRecord">
> > > <entity name="observation" query="select * from
> > > climatedata.ws_record limit 1000000">
> > > <field column="id" name="id" />
> > > <field column="inst_code" name="inst_code" />
> > > <field column="inst_name" name="inst_name" />
> > > <field column="meas_name" name="meas_name" />
> > > <field column="latitude" name="latitude" />
> > > <field column="longitude" name="longitude" />
> > > <field column="ob_id" name="ob_id" />
> > > <field column="in_id" name="in_id" />
> > > <field column="ob_name" name="ob_name" />
> > > </entity>
> > > </document>
> > > </dataConfig>
> > >
> > > -----------------------------------------------------------------
> > > in the solr Schema.xml:
> > > ----------------------------------------------------------------
> > > <fields>
> > > <field name="id" type="string" indexed="true" stored="true"
> > > multiValued="false"/>
> > > <field name="inst_code" type="text" indexed="true" stored="true"
> > > multiValued="true" required="false"/>
> > > <field name="inst_name" type="text" indexed="true" stored="true"
> > > multiValued="true" required="false"/>
> > > <field name="meas_name" type="text" indexed="true" stored="true"
> > > multiValued="true" required="false"/>
> > > <field name="latitude" type="sfloat" class="solr.FloatField"
> > > indexed="true" stored="true" required="false"/>
> > > <field name="longitude" type="sfloat" class="solr.FloatField"
> > > indexed="true" stored="true" required="false"/>
> > > <field name="ob_id" type="string" indexed="true" stored="true"
> > > multiValued="true"/>
> > > <field name="in_id" type="string" indexed="true" stored="true"
> > > multiValued="true"/>
> > > <field name="ob_name" type="text" indexed="true" stored="true"
> > > multiValued="true"/>
> > >
> > > <!-- catchall field, containing all other searchable text fields
> > > (implemented
> > > via copyField further on in this schema -->
> > > <field name="text" type="text" indexed="true" stored="false"
> > > multiValued="true" required="false"/>
> > >
> > > <!-- non-tokenized version of manufacturer to make it easier to sort
> or
> > > group
> > > results by manufacturer. copied from "manu" via copyField -->
> > > <field name="manu_exact" type="string" indexed="true" stored="false"
> > > required="false"/>
> > >
> > >
> > > <!-- Dynamic field definitions. If a field name is not found,
> > > dynamicFields
> > > will be used if the name matches any of the patterns.
> > > RESTRICTION: the glob-like pattern in the name attribute must
> have
> > > a "*" only at the start or the end.
> > > EXAMPLE: name="*_i" will match any field ending in _i (like
> > myid_i,
> > > z_i)
> > > Longer patterns will be matched first. if equal size patterns
> > > both match, the first appearing in the schema will be used. -->
> > > <dynamicField name="*_i" type="sint" indexed="true"
> > stored="true"/>
> > > <dynamicField name="*_s" type="string" indexed="true"
> > stored="true"/>
> > > <dynamicField name="*_l" type="slong" indexed="true"
> > stored="true"/>
> > > <dynamicField name="*_t" type="text" indexed="true"
> > stored="true"/>
> > > <dynamicField name="*_b" type="boolean" indexed="true"
> > stored="true"/>
> > > <dynamicField name="*_f" type="sfloat" indexed="true"
> > stored="true"/>
> > > <dynamicField name="*_d" type="sdouble" indexed="true"
> > stored="true"/>
> > > <dynamicField name="*_dt" type="date" indexed="true"
> > stored="true"/>
> > > </fields>
> > >
> > > ----------------------------------------------------
> > > I run the index via firefox browser using
> > > http://localhost:8080/solr/dataimport?command=full-import
> > > I checked the status using
> > > http://localhost:8080/solr/dataimport?command=status
> > > initially the status increased steadily, but after reaching 613071, the
> > > status stayed for a while (as below), and then it displayed the
> completed
> > > message :
> > > ----------------------------------------------------
> > > <response>
> > > -
> > > <lst name="responseHeader">
> > > <int name="status">0</int>
> > > <int name="QTime">1</int>
> > > </lst>
> > > -
> > > <lst name="initArgs">
> > > -
> > > <lst name="defaults">
> > > <str name="config">db-data-config.xml</str>
> > > </lst>
> > > </lst>
> > > <str name="command">status</str>
> > > <str name="status">busy</str>
> > > <str name="importResponse">A command is still running...</str>
> > > -
> > > <lst name="statusMessages">
> > > <str name="Time Elapsed">0:3:24.266</str>
> > > <str name="Total Requests made to DataSource">1</str>
> > > <str name="Total Rows Fetched">613071</str>
> > > <str name="Total Documents Processed">613070</str>
> > > <str name="Total Documents Skipped">0</str>
> > > <str name="Full Dump Started">2008-11-14 12:12:16</str>
> > > </lst>
> > > -
> > > <str name="WARNING">
> > > This response format is experimental. It is likely to change in the
> > future.
> > > </str>
> > > </response>
> > >
> > > -----------------------------------------------------------
> > >
> > >>>NOTE: this is the status result after it completed
> > > -----------------------------------------------------------
> > >
> > > <response>
> > > -
> > > <lst name="responseHeader">
> > > <int name="status">0</int>
> > > <int name="QTime">1</int>
> > > </lst>
> > > -
> > > <lst name="initArgs">
> > > -
> > > <lst name="defaults">
> > > <str name="config">db-data-config.xml</str>
> > > </lst>
> > > </lst>
> > > <str name="command">status</str>
> > > <str name="status">idle</str>
> > > <str name="importResponse"/>
> > > -
> > > <lst name="statusMessages">
> > > <str name="Total Requests made to DataSource">1</str>
> > > <str name="Total Rows Fetched">615360</str>
> > > <str name="Total Documents Skipped">0</str>
> > > <str name="Full Dump Started">2008-11-14 12:12:16</str>
> > > -
> > > <str name="">
> > > Indexing completed. Added/Updated: 615360 documents. Deleted 0
> documents.
> > > </str>
> > > <str name="Committed">2008-11-14 12:16:32</str>
> > > <str name="Optimized">2008-11-14 12:16:32</str>
> > > <str name="Time taken ">0:4:16.154</str>
> > > </lst>
> > > -
> > > <str name="WARNING">
> > > This response format is experimental. It is likely to change in the
> > future.
> > > </str>
> > > </response>
> > >
> > > -----------------------------------------------------
> > >
> > > here is the full solr scehma.xml content:
> > > ----------------------------------------------------
> > > <?xml version="1.0" ?>
> > > <!-- The Solr schema file. This file should be named "schema.xml" and
> > > should be in the conf directory under the solr home
> > > (i.e. ./solr/conf/schema.xml by default)
> > > or located where the classloader for the Solr webapp can find it.
> > >
> > > For more information, on how to customize this file, please see...
> > > http://wiki.apache.org/solr/SchemaXml
> > > -->
> > >
> > > <schema name="example" version="1.1">
> > > <types>
> > > <!-- field type definitions. The "name" attribute is
> > > just a label to be used by field definitions. The "class"
> > > attribute and any other attributes determine the real
> > > behavior of the fieldtype. -->
> > >
> > > <!-- The StringField type is not analyzed, but indexed/stored
> verbatim
> > > -->
> > > <fieldtype name="string" class="solr.StrField"
> > sortMissingLast="true"/>
> > >
> > > <!-- boolean type: "true" or "false" -->
> > > <fieldtype name="boolean" class="solr.BoolField"
> > > sortMissingLast="true"/>
> > >
> > > <!-- The optional sortMissingLast and sortMissingFirst attributes
> are
> > > currently supported on types that are sorted internally as a
> > > strings.
> > > - If sortMissingLast="true" then a sort on this field will cause
> > > documents
> > > without the field to come after documents with the field,
> > > regardless of the requested sort order (asc or desc).
> > > - If sortMissingFirst="true" then a sort on this field will cause
> > > documents
> > > without the field to come before documents with the field,
> > > regardless of the requested sort order.
> > > - If sortMissingLast="false" and sortMissingFirst="false" (the
> > > default),
> > > then default lucene sorting will be used which places docs
> without
> > > the field
> > > first in an ascending sort and last in a descending sort.
> > > -->
> > >
> > > <!-- numeric field types that store and index the text
> > > value verbatim (and hence don't support range queries since the
> > > lexicographic ordering isn't equal to the numeric ordering) -->
> > > <fieldtype name="integer" class="solr.IntField"/>
> > > <fieldtype name="long" class="solr.LongField"/>
> > > <fieldtype name="float" class="solr.FloatField"/>
> > > <fieldtype name="double" class="solr.DoubleField"/>
> > >
> > >
> > > <!-- Numeric field types that manipulate the value into
> > > a string value that isn't human readable in it's internal form,
> > > but with a lexicographic ordering the same as the numeric
> > ordering
> > > so that range queries correctly work. -->
> > > <fieldtype name="sint" class="solr.SortableIntField"
> > > sortMissingLast="true"/>
> > > <fieldtype name="slong" class="solr.SortableLongField"
> > > sortMissingLast="true"/>
> > > <fieldtype name="sfloat" class="solr.SortableFloatField"
> > > sortMissingLast="true"/>
> > > <fieldtype name="sdouble" class="solr.SortableDoubleField"
> > > sortMissingLast="true"/>
> > >
> > >
> > > <!-- The format for this date field is of the form
> > 1995-12-31T23:59:59Z,
> > > and
> > > is a more restricted form of the canonical representation of
> > > dateTime
> > > http://www.w3.org/TR/xmlschema-2/#dateTime
> > > The trailing "Z" designates UTC time and is mandatory.
> > > Optional fractional seconds are allowed:
> 1995-12-31T23:59:59.999Z
> > > All other components are mandatory. -->
> > > <fieldtype name="date" class="solr.DateField"
> sortMissingLast="true"/>
> > >
> > > <!-- solr.TextField allows the specification of custom text
> analyzers
> > > specified as a tokenizer and a list of token filters. Different
> > > analyzers may be specified for indexing and querying.
> > >
> > > The optional positionIncrementGap puts space between multiple
> > > fields of
> > > this type on the same document, with the purpose of preventing
> > > false phrase
> > > matching across fields.
> > >
> > > For more info on customizing your analyzer chain, please see...
> > > http://wiki.apache.org/solr/AnalyzersTokenizersTokenFilters
> > >
> > > -->
> > >
> > > <!-- Standard analyzer commonly used by Lucene developers
> > > -->
> > > <!-- Standard analyzer commonly used by Lucene developers -->
> > > <fieldtype name="text_lu" class="solr.TextField"
> > > positionIncrementGap="100">
> > > <analyzer>
> > > <tokenizer class="solr.StandardTokenizerFactory"/>
> > > <filter class="solr.StandardFilterFactory"/>
> > > <filter class="solr.LowerCaseFilterFactory"/>
> > > <filter class="solr.StopFilterFactory"/>
> > > <filter class="solr.EnglishPorterFilterFactory"/>
> > > </analyzer>
> > > </fieldtype>
> > > <!-- One could also specify an existing Analyzer implementation in
> > Java
> > > via the class attribute on the analyzer element:
> > > <fieldtype name="text_lu" class="solr.TextField">
> > > <analyzer
> > > class="org.apache.lucene.analysis.snowball.SnowballAnalyzer"/>
> > > </fieldType>
> > > -->
> > >
> > > <!-- A text field that only splits on whitespace for more exact
> > matching
> > > -->
> > > <fieldtype name="text_ws" class="solr.TextField"
> > > positionIncrementGap="100">
> > > <analyzer>
> > > <tokenizer class="solr.WhitespaceTokenizerFactory"/>
> > > </analyzer>
> > > </fieldtype>
> > >
> > > <!-- A text field that uses WordDelimiterFilter to enable splitting
> > and
> > > matching of
> > > words on case-change, alpha numeric boundaries, and
> > non-alphanumeric
> > > chars
> > > so that a query of "wifi" or "wi fi" could match a document
> > > containing "Wi-Fi".
> > > Synonyms and stopwords are customized by external files, and
> > > stemming is enabled -->
> > > <fieldtype name="text" class="solr.TextField"
> > > positionIncrementGap="100">
> > > <analyzer type="index">
> > > <tokenizer class="solr.WhitespaceTokenizerFactory"/>
> > > <!-- in this example, we will only use synonyms at query time
> > > <filter class="solr.SynonymFilterFactory"
> > > synonyms="index_synonyms.txt" ignoreCase="true" expand="false"/>
> > > -->
> > > <!--<filter class="solr.WordDelimiterFilterFactory"
> > > generateWordParts="1"/>-->
> > > <filter class="solr.StopFilterFactory" ignoreCase="true"/>
> > > <filter class="solr.LowerCaseFilterFactory"/>
> > > </analyzer>
> > > <analyzer type="query">
> > > <tokenizer class="solr.WhitespaceTokenizerFactory"/>
> > > <filter class="solr.StopFilterFactory" ignoreCase="true"/>
> > > <filter class="solr.LowerCaseFilterFactory"/>
> > > </analyzer>
> > > </fieldtype>
> > >
> > > <!-- Less flexible matching, but less false matches. Probably not
> > ideal
> > > for product names
> > > but may be good for SKUs. Can insert dashes in the wrong place
> > and
> > > still match. -->
> > > <fieldtype name="textTight" class="solr.TextField"
> > > positionIncrementGap="100" >
> > > <analyzer>
> > > <tokenizer class="solr.WhitespaceTokenizerFactory"/>
> > > <filter class="solr.SynonymFilterFactory"
> synonyms="synonyms.txt"
> > > ignoreCase="true" expand="false"/>
> > > <filter class="solr.StopFilterFactory" ignoreCase="true"/>
> > > <filter class="solr.WordDelimiterFilterFactory"
> > > generateWordParts="0" generateNumberParts="0" catenateWords="1"
> > > catenateNumbers="1" catenateAll="0"/>
> > > <filter class="solr.LowerCaseFilterFactory"/>
> > > <filter class="solr.EnglishPorterFilterFactory"
> > > protected="protwords.txt"/>
> > > </analyzer>
> > > </fieldtype>
> > > </types>
> > > <fields>
> > > <!-- Valid attributes for fields:
> > > name: mandatory - the name for the field
> > > type: mandatory - the name of a previously defined type from the
> > > <types> section
> > > indexed: true if this field should be indexed (searchable)
> > > stored: true if this field should be retrievable
> > > multiValued: true if this field may contain multiple values per
> > > document
> > > omitNorms: (expert) set to true to omit the norms associated with
> > > this field
> > > (this disables length normalization and index-time
> > > boosting for the field)
> > > -->
> > > <field name="id" type="string" indexed="true" stored="true"
> > > multiValued="false"/>
> > > <field name="inst_code" type="text" indexed="true" stored="true"
> > > multiValued="true" required="false"/>
> > > <field name="inst_name" type="text" indexed="true" stored="true"
> > > multiValued="true" required="false"/>
> > > <field name="meas_name" type="text" indexed="true" stored="true"
> > > multiValued="true" required="false"/>
> > > <field name="latitude" type="sfloat" class="solr.FloatField"
> > > indexed="true" stored="true" required="false"/>
> > > <field name="longitude" type="sfloat" class="solr.FloatField"
> > > indexed="true" stored="true" required="false"/>
> > > <field name="ob_id" type="string" indexed="true" stored="true"
> > > multiValued="true"/>
> > > <field name="in_id" type="string" indexed="true" stored="true"
> > > multiValued="true"/>
> > > <field name="ob_name" type="text" indexed="true" stored="true"
> > > multiValued="true"/>
> > >
> > > <!-- catchall field, containing all other searchable text fields
> > > (implemented
> > > via copyField further on in this schema -->
> > > <field name="text" type="text" indexed="true" stored="false"
> > > multiValued="true" required="false"/>
> > >
> > >
> > > <!-- non-tokenized version of manufacturer to make it easier to sort
> or
> > > group
> > > results by manufacturer. copied from "manu" via copyField -->
> > > <field name="manu_exact" type="string" indexed="true" stored="false"
> > > required="false"/>
> > >
> > >
> > > <!-- Dynamic field definitions. If a field name is not found,
> > > dynamicFields
> > > will be used if the name matches any of the patterns.
> > > RESTRICTION: the glob-like pattern in the name attribute must
> have
> > > a "*" only at the start or the end.
> > > EXAMPLE: name="*_i" will match any field ending in _i (like
> > myid_i,
> > > z_i)
> > > Longer patterns will be matched first. if equal size patterns
> > > both match, the first appearing in the schema will be used. -->
> > > <dynamicField name="*_i" type="sint" indexed="true"
> > stored="true"/>
> > > <dynamicField name="*_s" type="string" indexed="true"
> > stored="true"/>
> > > <dynamicField name="*_l" type="slong" indexed="true"
> > stored="true"/>
> > > <dynamicField name="*_t" type="text" indexed="true"
> > stored="true"/>
> > > <dynamicField name="*_b" type="boolean" indexed="true"
> > stored="true"/>
> > > <dynamicField name="*_f" type="sfloat" indexed="true"
> > stored="true"/>
> > > <dynamicField name="*_d" type="sdouble" indexed="true"
> > stored="true"/>
> > > <dynamicField name="*_dt" type="date" indexed="true"
> > stored="true"/>
> > > </fields>
> > >
> > > <!-- field to use to determine and enforce document uniqueness. -->
> > > <uniqueKey>id</uniqueKey>
> > >
> > > <!-- field for the QueryParser to use when an explicit fieldname is
> > absent
> > > -->
> > > <defaultSearchField>text</defaultSearchField>
> > >
> > > <!-- SolrQueryParser configuration: defaultOperator="AND|OR" -->
> > > <solrQueryParser defaultOperator="AND"/>
> > >
> > > <!-- copyField commands copy one field to another at the time a
> document
> > > is added to the index. It's used either to index the same field
> > > different
> > > ways, or to add multiple fields to the same field for
> > easier/faster
> > > searching. -->
> > >
> > >
> > >
> > > <!-- Similarity is the scoring routine for each document vs a query.
> > > A custom similarity may be specified here, but the default is fine
> > > for most applications. -->
> > > <!-- <similarity class="org.apache.lucene.search.DefaultSimilarity"/>
> > -->
> > >
> > > </schema>
> > >
> >
> -------------------------------------------------------------------------------------------------------------------------------------------------------------
> > >
> > >
> > > On Wed, Nov 12, 2008 at 11:01 PM, Noble Paul നോബിള് नोब्ळ् <
> > > noble.paul@gmail.com> wrote:
> > >
> > >> the fact that it got committed in the end suggests there was no error
> in
> > >> between
> > >>
> > >> look at the status url and see the no:of rows returned etc.
> > >>
> > >> It gives a clue as to what would have really happened. or you can
> > >> paste your dataconfig and status xmls and we may be able to suggest
> > >> something
> > >>
> > >> On Thu, Nov 13, 2008 at 9:26 AM, Giri <gi...@gmail.com> wrote:
> > >> > Hi Noble,
> > >> >
> > >> > thanks for reply, my comments are below
> > >> >
> > >> >>>why is the id field multivalued?
> > >> > I was just trying various options, yes, this ID is unique, and I
> check
> > >> for
> > >> > duplicates, when I did a distinct (id) query to the MySQL database,
> it
> > >> > returned almost 2 million.
> > >> >
> > >> >>> look at the status host:post/dataimport gives you the status
> > >> > I constantly checked the status using the dataimport URL, the
> > status
> > >> was
> > >> > increased upto 600K records, then it stopped increasing, then took
> few
> > >> > minutes to commit the indexed data.
> > >> >
> > >> >
> > >> > On Tue, Nov 11, 2008 at 11:35 PM, Noble Paul നോബിള് नोब्ळ् <
> > >> > noble.paul@gmail.com> wrote:
> > >> >
> > >> >> why is the id field multivalued? is there a uniqueKey in the schema
> ?
> > >> >> Are you sure there are no duplicates?
> > >> >>
> > >> >> look at the status host:post/dataimport gives you the status
> > >> >> it can give you some clue
> > >> >>
> > >> >> --Noble
> > >> >>
> > >> >>
> > >> >> On Wed, Nov 12, 2008 at 4:53 AM, Giri <gi...@gmail.com>
> wrote:
> > >> >> > Hi,
> > >> >> >
> > >> >> > I have about ~ 2 million records in a mySQL database table (about
> 9
> > >> >> fields
> > >> >> > from a single table), and I am trying to load it to the solr
> using
> > >> >> > DataImportHandler using the command=full-import option. it only
> > >> indexed
> > >> >> > about 615360 records out of 2 millions.
> > >> >> >
> > >> >> > here is my db-data-config.xml
> > >> >> > <dataConfig>
> > >> >> > <dataSource type="JdbcDataSource"
> driver="com.mysql.jdbc.Driver"
> > >> >> > url="jdbc:mysql://localhost:3306/mydb" user="ua" password="pw"
> > >> batchSize
> > >> >> > ="-1"/>
> > >> >> > <document name="climate">
> > >> >> > <entity name="occurence" query="select * from
> mylargetable">
> > >> >> > <field column="id" name="id" />
> > >> >> > <field column="title" name="title" />
> > >> >> > <field column="url" name="url" />
> > >> >> > </entity>
> > >> >> > </document>
> > >> >> > </dataConfig>
> > >> >> >
> > >> >> > and in my solr schema.xml, i define these fields as:
> > >> >> >
> > >> >> > <field name="id" type="string" indexed="true" stored="true"
> > >> >> > multiValued="true"/>
> > >> >> > <field name="title" type="text" indexed="true" stored="true"
> > >> >> > multiValued="true" required="false"/>
> > >> >> > <field name="url" type="text" indexed="true" stored="true"
> > >> >> > multiValued="true" required="false"/>
> > >> >> >
> > >> >> >
> > >> >> > If I try to index just one field (id), then it indexes about
> 960000
> > >> >> records,
> > >> >> > but if I try to index all the above three fields, it indexes only
> > >> 615360
> > >> >> > records.
> > >> >> >
> > >> >> > Any help will be appreciated.
> > >> >> >
> > >> >> > thanks!
> > >> >> >
> > >> >>
> > >> >>
> > >> >>
> > >> >> --
> > >> >> --Noble Paul
> > >> >>
> > >> >
> > >>
> > >>
> > >>
> > >> --
> > >> --Noble Paul
> > >>
> > >
> >
> >
> >
> > --
> > --Noble Paul
> >
>
--
Regards,
Shalin Shekhar Mangar.
Re: DataImportHandler not indexing all the records
Posted by Ahmed Hammad <ah...@gmail.com>.
I had a similar problem like Giri. I have 17,000 record in one table and DIH
can import only 12464.
After some investigation, I found my problem.
I have a regular expression to strip off html tags form input text, as
following:
<field sourceColName="content" column="content" regex="<(.|\n)*?>"
replaceWith=" "/>
The DIH RegEx have stack overflow on the record 17,000 due to error in the
content and then DIH exit without any error in the log on in the status
command. Here is the status:
<lst name="statusMessages">
<str name="Time Elapsed">0:0:31.657</str>
<str name="Total Requests made to DataSource">1</str>
<str name="Total Rows Fetched">12464</str>
<str name="Total Documents Processed">12464</str>
<str name="Total Documents Skipped">0</str>
<str name="Full Dump Started">2008-11-15 20:40:58</str>
</lst>
I found the error in Eclipse Console window while debugging; it was a stack
overflow in the RegEx library.
The problem is that, DIH does not show any problem in log file on in status
message.
What I think is important is to show whatever error happen in the log file.
I noticed also that, in case of no error a log message show completness:
Nov 15, 2008 8:57:34 PM org.apache.solr.handler.dataimport.DocBuilder
execute
INFO: Time taken = 0:0:40.656
In case of RegEx stack overflow error, this log message does not appear.
I am researching on how to catch such error in DIH. Any ideas?
Regards,
ahmd
On Sat, Nov 15, 2008 at 6:32 AM, Noble Paul നോബിള് नोब्ळ् <
noble.paul@gmail.com> wrote:
> There is no obvious problem
>
> I can be reasonably sure that
> the query
>
> select * from climatedata.ws_record limit 1000000
>
> would have fetched only 615360 rows.
> This is a very reliable pice of information
> <str name="Total Rows Fetched">615360</str>
>
> On Sat, Nov 15, 2008 at 12:41 AM, Giri <gi...@gmail.com> wrote:
> > Hi Noble,
> > thanks for the help, here are the details: the field "id" is unique, when
> I
> > did a select distinct(id), it returned 1 million rows.
> >
> > -------------------------------------------------------------------
> > db-data-config.xml
> > note: I limit the resultset to 1 million in the select query
> > -------------------------------------------------------------------
> > <dataConfig>
> > <dataSource type="JdbcDataSource" driver="com.mysql.jdbc.Driver"
> > url="jdbc:mysql://localhost:3306/climatedata" user="user" password="pw"
> > batchSize ="-1"/>
> > <document name="climateRecord">
> > <entity name="observation" query="select * from
> > climatedata.ws_record limit 1000000">
> > <field column="id" name="id" />
> > <field column="inst_code" name="inst_code" />
> > <field column="inst_name" name="inst_name" />
> > <field column="meas_name" name="meas_name" />
> > <field column="latitude" name="latitude" />
> > <field column="longitude" name="longitude" />
> > <field column="ob_id" name="ob_id" />
> > <field column="in_id" name="in_id" />
> > <field column="ob_name" name="ob_name" />
> > </entity>
> > </document>
> > </dataConfig>
> >
> > -----------------------------------------------------------------
> > in the solr Schema.xml:
> > ----------------------------------------------------------------
> > <fields>
> > <field name="id" type="string" indexed="true" stored="true"
> > multiValued="false"/>
> > <field name="inst_code" type="text" indexed="true" stored="true"
> > multiValued="true" required="false"/>
> > <field name="inst_name" type="text" indexed="true" stored="true"
> > multiValued="true" required="false"/>
> > <field name="meas_name" type="text" indexed="true" stored="true"
> > multiValued="true" required="false"/>
> > <field name="latitude" type="sfloat" class="solr.FloatField"
> > indexed="true" stored="true" required="false"/>
> > <field name="longitude" type="sfloat" class="solr.FloatField"
> > indexed="true" stored="true" required="false"/>
> > <field name="ob_id" type="string" indexed="true" stored="true"
> > multiValued="true"/>
> > <field name="in_id" type="string" indexed="true" stored="true"
> > multiValued="true"/>
> > <field name="ob_name" type="text" indexed="true" stored="true"
> > multiValued="true"/>
> >
> > <!-- catchall field, containing all other searchable text fields
> > (implemented
> > via copyField further on in this schema -->
> > <field name="text" type="text" indexed="true" stored="false"
> > multiValued="true" required="false"/>
> >
> > <!-- non-tokenized version of manufacturer to make it easier to sort or
> > group
> > results by manufacturer. copied from "manu" via copyField -->
> > <field name="manu_exact" type="string" indexed="true" stored="false"
> > required="false"/>
> >
> >
> > <!-- Dynamic field definitions. If a field name is not found,
> > dynamicFields
> > will be used if the name matches any of the patterns.
> > RESTRICTION: the glob-like pattern in the name attribute must have
> > a "*" only at the start or the end.
> > EXAMPLE: name="*_i" will match any field ending in _i (like
> myid_i,
> > z_i)
> > Longer patterns will be matched first. if equal size patterns
> > both match, the first appearing in the schema will be used. -->
> > <dynamicField name="*_i" type="sint" indexed="true"
> stored="true"/>
> > <dynamicField name="*_s" type="string" indexed="true"
> stored="true"/>
> > <dynamicField name="*_l" type="slong" indexed="true"
> stored="true"/>
> > <dynamicField name="*_t" type="text" indexed="true"
> stored="true"/>
> > <dynamicField name="*_b" type="boolean" indexed="true"
> stored="true"/>
> > <dynamicField name="*_f" type="sfloat" indexed="true"
> stored="true"/>
> > <dynamicField name="*_d" type="sdouble" indexed="true"
> stored="true"/>
> > <dynamicField name="*_dt" type="date" indexed="true"
> stored="true"/>
> > </fields>
> >
> > ----------------------------------------------------
> > I run the index via firefox browser using
> > http://localhost:8080/solr/dataimport?command=full-import
> > I checked the status using
> > http://localhost:8080/solr/dataimport?command=status
> > initially the status increased steadily, but after reaching 613071, the
> > status stayed for a while (as below), and then it displayed the completed
> > message :
> > ----------------------------------------------------
> > <response>
> > -
> > <lst name="responseHeader">
> > <int name="status">0</int>
> > <int name="QTime">1</int>
> > </lst>
> > -
> > <lst name="initArgs">
> > -
> > <lst name="defaults">
> > <str name="config">db-data-config.xml</str>
> > </lst>
> > </lst>
> > <str name="command">status</str>
> > <str name="status">busy</str>
> > <str name="importResponse">A command is still running...</str>
> > -
> > <lst name="statusMessages">
> > <str name="Time Elapsed">0:3:24.266</str>
> > <str name="Total Requests made to DataSource">1</str>
> > <str name="Total Rows Fetched">613071</str>
> > <str name="Total Documents Processed">613070</str>
> > <str name="Total Documents Skipped">0</str>
> > <str name="Full Dump Started">2008-11-14 12:12:16</str>
> > </lst>
> > -
> > <str name="WARNING">
> > This response format is experimental. It is likely to change in the
> future.
> > </str>
> > </response>
> >
> > -----------------------------------------------------------
> >
> >>>NOTE: this is the status result after it completed
> > -----------------------------------------------------------
> >
> > <response>
> > -
> > <lst name="responseHeader">
> > <int name="status">0</int>
> > <int name="QTime">1</int>
> > </lst>
> > -
> > <lst name="initArgs">
> > -
> > <lst name="defaults">
> > <str name="config">db-data-config.xml</str>
> > </lst>
> > </lst>
> > <str name="command">status</str>
> > <str name="status">idle</str>
> > <str name="importResponse"/>
> > -
> > <lst name="statusMessages">
> > <str name="Total Requests made to DataSource">1</str>
> > <str name="Total Rows Fetched">615360</str>
> > <str name="Total Documents Skipped">0</str>
> > <str name="Full Dump Started">2008-11-14 12:12:16</str>
> > -
> > <str name="">
> > Indexing completed. Added/Updated: 615360 documents. Deleted 0 documents.
> > </str>
> > <str name="Committed">2008-11-14 12:16:32</str>
> > <str name="Optimized">2008-11-14 12:16:32</str>
> > <str name="Time taken ">0:4:16.154</str>
> > </lst>
> > -
> > <str name="WARNING">
> > This response format is experimental. It is likely to change in the
> future.
> > </str>
> > </response>
> >
> > -----------------------------------------------------
> >
> > here is the full solr scehma.xml content:
> > ----------------------------------------------------
> > <?xml version="1.0" ?>
> > <!-- The Solr schema file. This file should be named "schema.xml" and
> > should be in the conf directory under the solr home
> > (i.e. ./solr/conf/schema.xml by default)
> > or located where the classloader for the Solr webapp can find it.
> >
> > For more information, on how to customize this file, please see...
> > http://wiki.apache.org/solr/SchemaXml
> > -->
> >
> > <schema name="example" version="1.1">
> > <types>
> > <!-- field type definitions. The "name" attribute is
> > just a label to be used by field definitions. The "class"
> > attribute and any other attributes determine the real
> > behavior of the fieldtype. -->
> >
> > <!-- The StringField type is not analyzed, but indexed/stored verbatim
> > -->
> > <fieldtype name="string" class="solr.StrField"
> sortMissingLast="true"/>
> >
> > <!-- boolean type: "true" or "false" -->
> > <fieldtype name="boolean" class="solr.BoolField"
> > sortMissingLast="true"/>
> >
> > <!-- The optional sortMissingLast and sortMissingFirst attributes are
> > currently supported on types that are sorted internally as a
> > strings.
> > - If sortMissingLast="true" then a sort on this field will cause
> > documents
> > without the field to come after documents with the field,
> > regardless of the requested sort order (asc or desc).
> > - If sortMissingFirst="true" then a sort on this field will cause
> > documents
> > without the field to come before documents with the field,
> > regardless of the requested sort order.
> > - If sortMissingLast="false" and sortMissingFirst="false" (the
> > default),
> > then default lucene sorting will be used which places docs without
> > the field
> > first in an ascending sort and last in a descending sort.
> > -->
> >
> > <!-- numeric field types that store and index the text
> > value verbatim (and hence don't support range queries since the
> > lexicographic ordering isn't equal to the numeric ordering) -->
> > <fieldtype name="integer" class="solr.IntField"/>
> > <fieldtype name="long" class="solr.LongField"/>
> > <fieldtype name="float" class="solr.FloatField"/>
> > <fieldtype name="double" class="solr.DoubleField"/>
> >
> >
> > <!-- Numeric field types that manipulate the value into
> > a string value that isn't human readable in it's internal form,
> > but with a lexicographic ordering the same as the numeric
> ordering
> > so that range queries correctly work. -->
> > <fieldtype name="sint" class="solr.SortableIntField"
> > sortMissingLast="true"/>
> > <fieldtype name="slong" class="solr.SortableLongField"
> > sortMissingLast="true"/>
> > <fieldtype name="sfloat" class="solr.SortableFloatField"
> > sortMissingLast="true"/>
> > <fieldtype name="sdouble" class="solr.SortableDoubleField"
> > sortMissingLast="true"/>
> >
> >
> > <!-- The format for this date field is of the form
> 1995-12-31T23:59:59Z,
> > and
> > is a more restricted form of the canonical representation of
> > dateTime
> > http://www.w3.org/TR/xmlschema-2/#dateTime
> > The trailing "Z" designates UTC time and is mandatory.
> > Optional fractional seconds are allowed: 1995-12-31T23:59:59.999Z
> > All other components are mandatory. -->
> > <fieldtype name="date" class="solr.DateField" sortMissingLast="true"/>
> >
> > <!-- solr.TextField allows the specification of custom text analyzers
> > specified as a tokenizer and a list of token filters. Different
> > analyzers may be specified for indexing and querying.
> >
> > The optional positionIncrementGap puts space between multiple
> > fields of
> > this type on the same document, with the purpose of preventing
> > false phrase
> > matching across fields.
> >
> > For more info on customizing your analyzer chain, please see...
> > http://wiki.apache.org/solr/AnalyzersTokenizersTokenFilters
> >
> > -->
> >
> > <!-- Standard analyzer commonly used by Lucene developers
> > -->
> > <!-- Standard analyzer commonly used by Lucene developers -->
> > <fieldtype name="text_lu" class="solr.TextField"
> > positionIncrementGap="100">
> > <analyzer>
> > <tokenizer class="solr.StandardTokenizerFactory"/>
> > <filter class="solr.StandardFilterFactory"/>
> > <filter class="solr.LowerCaseFilterFactory"/>
> > <filter class="solr.StopFilterFactory"/>
> > <filter class="solr.EnglishPorterFilterFactory"/>
> > </analyzer>
> > </fieldtype>
> > <!-- One could also specify an existing Analyzer implementation in
> Java
> > via the class attribute on the analyzer element:
> > <fieldtype name="text_lu" class="solr.TextField">
> > <analyzer
> > class="org.apache.lucene.analysis.snowball.SnowballAnalyzer"/>
> > </fieldType>
> > -->
> >
> > <!-- A text field that only splits on whitespace for more exact
> matching
> > -->
> > <fieldtype name="text_ws" class="solr.TextField"
> > positionIncrementGap="100">
> > <analyzer>
> > <tokenizer class="solr.WhitespaceTokenizerFactory"/>
> > </analyzer>
> > </fieldtype>
> >
> > <!-- A text field that uses WordDelimiterFilter to enable splitting
> and
> > matching of
> > words on case-change, alpha numeric boundaries, and
> non-alphanumeric
> > chars
> > so that a query of "wifi" or "wi fi" could match a document
> > containing "Wi-Fi".
> > Synonyms and stopwords are customized by external files, and
> > stemming is enabled -->
> > <fieldtype name="text" class="solr.TextField"
> > positionIncrementGap="100">
> > <analyzer type="index">
> > <tokenizer class="solr.WhitespaceTokenizerFactory"/>
> > <!-- in this example, we will only use synonyms at query time
> > <filter class="solr.SynonymFilterFactory"
> > synonyms="index_synonyms.txt" ignoreCase="true" expand="false"/>
> > -->
> > <!--<filter class="solr.WordDelimiterFilterFactory"
> > generateWordParts="1"/>-->
> > <filter class="solr.StopFilterFactory" ignoreCase="true"/>
> > <filter class="solr.LowerCaseFilterFactory"/>
> > </analyzer>
> > <analyzer type="query">
> > <tokenizer class="solr.WhitespaceTokenizerFactory"/>
> > <filter class="solr.StopFilterFactory" ignoreCase="true"/>
> > <filter class="solr.LowerCaseFilterFactory"/>
> > </analyzer>
> > </fieldtype>
> >
> > <!-- Less flexible matching, but less false matches. Probably not
> ideal
> > for product names
> > but may be good for SKUs. Can insert dashes in the wrong place
> and
> > still match. -->
> > <fieldtype name="textTight" class="solr.TextField"
> > positionIncrementGap="100" >
> > <analyzer>
> > <tokenizer class="solr.WhitespaceTokenizerFactory"/>
> > <filter class="solr.SynonymFilterFactory" synonyms="synonyms.txt"
> > ignoreCase="true" expand="false"/>
> > <filter class="solr.StopFilterFactory" ignoreCase="true"/>
> > <filter class="solr.WordDelimiterFilterFactory"
> > generateWordParts="0" generateNumberParts="0" catenateWords="1"
> > catenateNumbers="1" catenateAll="0"/>
> > <filter class="solr.LowerCaseFilterFactory"/>
> > <filter class="solr.EnglishPorterFilterFactory"
> > protected="protwords.txt"/>
> > </analyzer>
> > </fieldtype>
> > </types>
> > <fields>
> > <!-- Valid attributes for fields:
> > name: mandatory - the name for the field
> > type: mandatory - the name of a previously defined type from the
> > <types> section
> > indexed: true if this field should be indexed (searchable)
> > stored: true if this field should be retrievable
> > multiValued: true if this field may contain multiple values per
> > document
> > omitNorms: (expert) set to true to omit the norms associated with
> > this field
> > (this disables length normalization and index-time
> > boosting for the field)
> > -->
> > <field name="id" type="string" indexed="true" stored="true"
> > multiValued="false"/>
> > <field name="inst_code" type="text" indexed="true" stored="true"
> > multiValued="true" required="false"/>
> > <field name="inst_name" type="text" indexed="true" stored="true"
> > multiValued="true" required="false"/>
> > <field name="meas_name" type="text" indexed="true" stored="true"
> > multiValued="true" required="false"/>
> > <field name="latitude" type="sfloat" class="solr.FloatField"
> > indexed="true" stored="true" required="false"/>
> > <field name="longitude" type="sfloat" class="solr.FloatField"
> > indexed="true" stored="true" required="false"/>
> > <field name="ob_id" type="string" indexed="true" stored="true"
> > multiValued="true"/>
> > <field name="in_id" type="string" indexed="true" stored="true"
> > multiValued="true"/>
> > <field name="ob_name" type="text" indexed="true" stored="true"
> > multiValued="true"/>
> >
> > <!-- catchall field, containing all other searchable text fields
> > (implemented
> > via copyField further on in this schema -->
> > <field name="text" type="text" indexed="true" stored="false"
> > multiValued="true" required="false"/>
> >
> >
> > <!-- non-tokenized version of manufacturer to make it easier to sort or
> > group
> > results by manufacturer. copied from "manu" via copyField -->
> > <field name="manu_exact" type="string" indexed="true" stored="false"
> > required="false"/>
> >
> >
> > <!-- Dynamic field definitions. If a field name is not found,
> > dynamicFields
> > will be used if the name matches any of the patterns.
> > RESTRICTION: the glob-like pattern in the name attribute must have
> > a "*" only at the start or the end.
> > EXAMPLE: name="*_i" will match any field ending in _i (like
> myid_i,
> > z_i)
> > Longer patterns will be matched first. if equal size patterns
> > both match, the first appearing in the schema will be used. -->
> > <dynamicField name="*_i" type="sint" indexed="true"
> stored="true"/>
> > <dynamicField name="*_s" type="string" indexed="true"
> stored="true"/>
> > <dynamicField name="*_l" type="slong" indexed="true"
> stored="true"/>
> > <dynamicField name="*_t" type="text" indexed="true"
> stored="true"/>
> > <dynamicField name="*_b" type="boolean" indexed="true"
> stored="true"/>
> > <dynamicField name="*_f" type="sfloat" indexed="true"
> stored="true"/>
> > <dynamicField name="*_d" type="sdouble" indexed="true"
> stored="true"/>
> > <dynamicField name="*_dt" type="date" indexed="true"
> stored="true"/>
> > </fields>
> >
> > <!-- field to use to determine and enforce document uniqueness. -->
> > <uniqueKey>id</uniqueKey>
> >
> > <!-- field for the QueryParser to use when an explicit fieldname is
> absent
> > -->
> > <defaultSearchField>text</defaultSearchField>
> >
> > <!-- SolrQueryParser configuration: defaultOperator="AND|OR" -->
> > <solrQueryParser defaultOperator="AND"/>
> >
> > <!-- copyField commands copy one field to another at the time a document
> > is added to the index. It's used either to index the same field
> > different
> > ways, or to add multiple fields to the same field for
> easier/faster
> > searching. -->
> >
> >
> >
> > <!-- Similarity is the scoring routine for each document vs a query.
> > A custom similarity may be specified here, but the default is fine
> > for most applications. -->
> > <!-- <similarity class="org.apache.lucene.search.DefaultSimilarity"/>
> -->
> >
> > </schema>
> >
> -------------------------------------------------------------------------------------------------------------------------------------------------------------
> >
> >
> > On Wed, Nov 12, 2008 at 11:01 PM, Noble Paul നോബിള് नोब्ळ् <
> > noble.paul@gmail.com> wrote:
> >
> >> the fact that it got committed in the end suggests there was no error in
> >> between
> >>
> >> look at the status url and see the no:of rows returned etc.
> >>
> >> It gives a clue as to what would have really happened. or you can
> >> paste your dataconfig and status xmls and we may be able to suggest
> >> something
> >>
> >> On Thu, Nov 13, 2008 at 9:26 AM, Giri <gi...@gmail.com> wrote:
> >> > Hi Noble,
> >> >
> >> > thanks for reply, my comments are below
> >> >
> >> >>>why is the id field multivalued?
> >> > I was just trying various options, yes, this ID is unique, and I check
> >> for
> >> > duplicates, when I did a distinct (id) query to the MySQL database, it
> >> > returned almost 2 million.
> >> >
> >> >>> look at the status host:post/dataimport gives you the status
> >> > I constantly checked the status using the dataimport URL, the
> status
> >> was
> >> > increased upto 600K records, then it stopped increasing, then took few
> >> > minutes to commit the indexed data.
> >> >
> >> >
> >> > On Tue, Nov 11, 2008 at 11:35 PM, Noble Paul നോബിള് नोब्ळ् <
> >> > noble.paul@gmail.com> wrote:
> >> >
> >> >> why is the id field multivalued? is there a uniqueKey in the schema ?
> >> >> Are you sure there are no duplicates?
> >> >>
> >> >> look at the status host:post/dataimport gives you the status
> >> >> it can give you some clue
> >> >>
> >> >> --Noble
> >> >>
> >> >>
> >> >> On Wed, Nov 12, 2008 at 4:53 AM, Giri <gi...@gmail.com> wrote:
> >> >> > Hi,
> >> >> >
> >> >> > I have about ~ 2 million records in a mySQL database table (about 9
> >> >> fields
> >> >> > from a single table), and I am trying to load it to the solr using
> >> >> > DataImportHandler using the command=full-import option. it only
> >> indexed
> >> >> > about 615360 records out of 2 millions.
> >> >> >
> >> >> > here is my db-data-config.xml
> >> >> > <dataConfig>
> >> >> > <dataSource type="JdbcDataSource" driver="com.mysql.jdbc.Driver"
> >> >> > url="jdbc:mysql://localhost:3306/mydb" user="ua" password="pw"
> >> batchSize
> >> >> > ="-1"/>
> >> >> > <document name="climate">
> >> >> > <entity name="occurence" query="select * from mylargetable">
> >> >> > <field column="id" name="id" />
> >> >> > <field column="title" name="title" />
> >> >> > <field column="url" name="url" />
> >> >> > </entity>
> >> >> > </document>
> >> >> > </dataConfig>
> >> >> >
> >> >> > and in my solr schema.xml, i define these fields as:
> >> >> >
> >> >> > <field name="id" type="string" indexed="true" stored="true"
> >> >> > multiValued="true"/>
> >> >> > <field name="title" type="text" indexed="true" stored="true"
> >> >> > multiValued="true" required="false"/>
> >> >> > <field name="url" type="text" indexed="true" stored="true"
> >> >> > multiValued="true" required="false"/>
> >> >> >
> >> >> >
> >> >> > If I try to index just one field (id), then it indexes about 960000
> >> >> records,
> >> >> > but if I try to index all the above three fields, it indexes only
> >> 615360
> >> >> > records.
> >> >> >
> >> >> > Any help will be appreciated.
> >> >> >
> >> >> > thanks!
> >> >> >
> >> >>
> >> >>
> >> >>
> >> >> --
> >> >> --Noble Paul
> >> >>
> >> >
> >>
> >>
> >>
> >> --
> >> --Noble Paul
> >>
> >
>
>
>
> --
> --Noble Paul
>
Re: DataImportHandler not indexing all the records
Posted by Noble Paul നോബിള് नोब्ळ् <no...@gmail.com>.
There is no obvious problem
I can be reasonably sure that
the query
select * from climatedata.ws_record limit 1000000
would have fetched only 615360 rows.
This is a very reliable pice of information
<str name="Total Rows Fetched">615360</str>
On Sat, Nov 15, 2008 at 12:41 AM, Giri <gi...@gmail.com> wrote:
> Hi Noble,
> thanks for the help, here are the details: the field "id" is unique, when I
> did a select distinct(id), it returned 1 million rows.
>
> -------------------------------------------------------------------
> db-data-config.xml
> note: I limit the resultset to 1 million in the select query
> -------------------------------------------------------------------
> <dataConfig>
> <dataSource type="JdbcDataSource" driver="com.mysql.jdbc.Driver"
> url="jdbc:mysql://localhost:3306/climatedata" user="user" password="pw"
> batchSize ="-1"/>
> <document name="climateRecord">
> <entity name="observation" query="select * from
> climatedata.ws_record limit 1000000">
> <field column="id" name="id" />
> <field column="inst_code" name="inst_code" />
> <field column="inst_name" name="inst_name" />
> <field column="meas_name" name="meas_name" />
> <field column="latitude" name="latitude" />
> <field column="longitude" name="longitude" />
> <field column="ob_id" name="ob_id" />
> <field column="in_id" name="in_id" />
> <field column="ob_name" name="ob_name" />
> </entity>
> </document>
> </dataConfig>
>
> -----------------------------------------------------------------
> in the solr Schema.xml:
> ----------------------------------------------------------------
> <fields>
> <field name="id" type="string" indexed="true" stored="true"
> multiValued="false"/>
> <field name="inst_code" type="text" indexed="true" stored="true"
> multiValued="true" required="false"/>
> <field name="inst_name" type="text" indexed="true" stored="true"
> multiValued="true" required="false"/>
> <field name="meas_name" type="text" indexed="true" stored="true"
> multiValued="true" required="false"/>
> <field name="latitude" type="sfloat" class="solr.FloatField"
> indexed="true" stored="true" required="false"/>
> <field name="longitude" type="sfloat" class="solr.FloatField"
> indexed="true" stored="true" required="false"/>
> <field name="ob_id" type="string" indexed="true" stored="true"
> multiValued="true"/>
> <field name="in_id" type="string" indexed="true" stored="true"
> multiValued="true"/>
> <field name="ob_name" type="text" indexed="true" stored="true"
> multiValued="true"/>
>
> <!-- catchall field, containing all other searchable text fields
> (implemented
> via copyField further on in this schema -->
> <field name="text" type="text" indexed="true" stored="false"
> multiValued="true" required="false"/>
>
> <!-- non-tokenized version of manufacturer to make it easier to sort or
> group
> results by manufacturer. copied from "manu" via copyField -->
> <field name="manu_exact" type="string" indexed="true" stored="false"
> required="false"/>
>
>
> <!-- Dynamic field definitions. If a field name is not found,
> dynamicFields
> will be used if the name matches any of the patterns.
> RESTRICTION: the glob-like pattern in the name attribute must have
> a "*" only at the start or the end.
> EXAMPLE: name="*_i" will match any field ending in _i (like myid_i,
> z_i)
> Longer patterns will be matched first. if equal size patterns
> both match, the first appearing in the schema will be used. -->
> <dynamicField name="*_i" type="sint" indexed="true" stored="true"/>
> <dynamicField name="*_s" type="string" indexed="true" stored="true"/>
> <dynamicField name="*_l" type="slong" indexed="true" stored="true"/>
> <dynamicField name="*_t" type="text" indexed="true" stored="true"/>
> <dynamicField name="*_b" type="boolean" indexed="true" stored="true"/>
> <dynamicField name="*_f" type="sfloat" indexed="true" stored="true"/>
> <dynamicField name="*_d" type="sdouble" indexed="true" stored="true"/>
> <dynamicField name="*_dt" type="date" indexed="true" stored="true"/>
> </fields>
>
> ----------------------------------------------------
> I run the index via firefox browser using
> http://localhost:8080/solr/dataimport?command=full-import
> I checked the status using
> http://localhost:8080/solr/dataimport?command=status
> initially the status increased steadily, but after reaching 613071, the
> status stayed for a while (as below), and then it displayed the completed
> message :
> ----------------------------------------------------
> <response>
> -
> <lst name="responseHeader">
> <int name="status">0</int>
> <int name="QTime">1</int>
> </lst>
> -
> <lst name="initArgs">
> -
> <lst name="defaults">
> <str name="config">db-data-config.xml</str>
> </lst>
> </lst>
> <str name="command">status</str>
> <str name="status">busy</str>
> <str name="importResponse">A command is still running...</str>
> -
> <lst name="statusMessages">
> <str name="Time Elapsed">0:3:24.266</str>
> <str name="Total Requests made to DataSource">1</str>
> <str name="Total Rows Fetched">613071</str>
> <str name="Total Documents Processed">613070</str>
> <str name="Total Documents Skipped">0</str>
> <str name="Full Dump Started">2008-11-14 12:12:16</str>
> </lst>
> -
> <str name="WARNING">
> This response format is experimental. It is likely to change in the future.
> </str>
> </response>
>
> -----------------------------------------------------------
>
>>>NOTE: this is the status result after it completed
> -----------------------------------------------------------
>
> <response>
> -
> <lst name="responseHeader">
> <int name="status">0</int>
> <int name="QTime">1</int>
> </lst>
> -
> <lst name="initArgs">
> -
> <lst name="defaults">
> <str name="config">db-data-config.xml</str>
> </lst>
> </lst>
> <str name="command">status</str>
> <str name="status">idle</str>
> <str name="importResponse"/>
> -
> <lst name="statusMessages">
> <str name="Total Requests made to DataSource">1</str>
> <str name="Total Rows Fetched">615360</str>
> <str name="Total Documents Skipped">0</str>
> <str name="Full Dump Started">2008-11-14 12:12:16</str>
> -
> <str name="">
> Indexing completed. Added/Updated: 615360 documents. Deleted 0 documents.
> </str>
> <str name="Committed">2008-11-14 12:16:32</str>
> <str name="Optimized">2008-11-14 12:16:32</str>
> <str name="Time taken ">0:4:16.154</str>
> </lst>
> -
> <str name="WARNING">
> This response format is experimental. It is likely to change in the future.
> </str>
> </response>
>
> -----------------------------------------------------
>
> here is the full solr scehma.xml content:
> ----------------------------------------------------
> <?xml version="1.0" ?>
> <!-- The Solr schema file. This file should be named "schema.xml" and
> should be in the conf directory under the solr home
> (i.e. ./solr/conf/schema.xml by default)
> or located where the classloader for the Solr webapp can find it.
>
> For more information, on how to customize this file, please see...
> http://wiki.apache.org/solr/SchemaXml
> -->
>
> <schema name="example" version="1.1">
> <types>
> <!-- field type definitions. The "name" attribute is
> just a label to be used by field definitions. The "class"
> attribute and any other attributes determine the real
> behavior of the fieldtype. -->
>
> <!-- The StringField type is not analyzed, but indexed/stored verbatim
> -->
> <fieldtype name="string" class="solr.StrField" sortMissingLast="true"/>
>
> <!-- boolean type: "true" or "false" -->
> <fieldtype name="boolean" class="solr.BoolField"
> sortMissingLast="true"/>
>
> <!-- The optional sortMissingLast and sortMissingFirst attributes are
> currently supported on types that are sorted internally as a
> strings.
> - If sortMissingLast="true" then a sort on this field will cause
> documents
> without the field to come after documents with the field,
> regardless of the requested sort order (asc or desc).
> - If sortMissingFirst="true" then a sort on this field will cause
> documents
> without the field to come before documents with the field,
> regardless of the requested sort order.
> - If sortMissingLast="false" and sortMissingFirst="false" (the
> default),
> then default lucene sorting will be used which places docs without
> the field
> first in an ascending sort and last in a descending sort.
> -->
>
> <!-- numeric field types that store and index the text
> value verbatim (and hence don't support range queries since the
> lexicographic ordering isn't equal to the numeric ordering) -->
> <fieldtype name="integer" class="solr.IntField"/>
> <fieldtype name="long" class="solr.LongField"/>
> <fieldtype name="float" class="solr.FloatField"/>
> <fieldtype name="double" class="solr.DoubleField"/>
>
>
> <!-- Numeric field types that manipulate the value into
> a string value that isn't human readable in it's internal form,
> but with a lexicographic ordering the same as the numeric ordering
> so that range queries correctly work. -->
> <fieldtype name="sint" class="solr.SortableIntField"
> sortMissingLast="true"/>
> <fieldtype name="slong" class="solr.SortableLongField"
> sortMissingLast="true"/>
> <fieldtype name="sfloat" class="solr.SortableFloatField"
> sortMissingLast="true"/>
> <fieldtype name="sdouble" class="solr.SortableDoubleField"
> sortMissingLast="true"/>
>
>
> <!-- The format for this date field is of the form 1995-12-31T23:59:59Z,
> and
> is a more restricted form of the canonical representation of
> dateTime
> http://www.w3.org/TR/xmlschema-2/#dateTime
> The trailing "Z" designates UTC time and is mandatory.
> Optional fractional seconds are allowed: 1995-12-31T23:59:59.999Z
> All other components are mandatory. -->
> <fieldtype name="date" class="solr.DateField" sortMissingLast="true"/>
>
> <!-- solr.TextField allows the specification of custom text analyzers
> specified as a tokenizer and a list of token filters. Different
> analyzers may be specified for indexing and querying.
>
> The optional positionIncrementGap puts space between multiple
> fields of
> this type on the same document, with the purpose of preventing
> false phrase
> matching across fields.
>
> For more info on customizing your analyzer chain, please see...
> http://wiki.apache.org/solr/AnalyzersTokenizersTokenFilters
>
> -->
>
> <!-- Standard analyzer commonly used by Lucene developers
> -->
> <!-- Standard analyzer commonly used by Lucene developers -->
> <fieldtype name="text_lu" class="solr.TextField"
> positionIncrementGap="100">
> <analyzer>
> <tokenizer class="solr.StandardTokenizerFactory"/>
> <filter class="solr.StandardFilterFactory"/>
> <filter class="solr.LowerCaseFilterFactory"/>
> <filter class="solr.StopFilterFactory"/>
> <filter class="solr.EnglishPorterFilterFactory"/>
> </analyzer>
> </fieldtype>
> <!-- One could also specify an existing Analyzer implementation in Java
> via the class attribute on the analyzer element:
> <fieldtype name="text_lu" class="solr.TextField">
> <analyzer
> class="org.apache.lucene.analysis.snowball.SnowballAnalyzer"/>
> </fieldType>
> -->
>
> <!-- A text field that only splits on whitespace for more exact matching
> -->
> <fieldtype name="text_ws" class="solr.TextField"
> positionIncrementGap="100">
> <analyzer>
> <tokenizer class="solr.WhitespaceTokenizerFactory"/>
> </analyzer>
> </fieldtype>
>
> <!-- A text field that uses WordDelimiterFilter to enable splitting and
> matching of
> words on case-change, alpha numeric boundaries, and non-alphanumeric
> chars
> so that a query of "wifi" or "wi fi" could match a document
> containing "Wi-Fi".
> Synonyms and stopwords are customized by external files, and
> stemming is enabled -->
> <fieldtype name="text" class="solr.TextField"
> positionIncrementGap="100">
> <analyzer type="index">
> <tokenizer class="solr.WhitespaceTokenizerFactory"/>
> <!-- in this example, we will only use synonyms at query time
> <filter class="solr.SynonymFilterFactory"
> synonyms="index_synonyms.txt" ignoreCase="true" expand="false"/>
> -->
> <!--<filter class="solr.WordDelimiterFilterFactory"
> generateWordParts="1"/>-->
> <filter class="solr.StopFilterFactory" ignoreCase="true"/>
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> <analyzer type="query">
> <tokenizer class="solr.WhitespaceTokenizerFactory"/>
> <filter class="solr.StopFilterFactory" ignoreCase="true"/>
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> </fieldtype>
>
> <!-- Less flexible matching, but less false matches. Probably not ideal
> for product names
> but may be good for SKUs. Can insert dashes in the wrong place and
> still match. -->
> <fieldtype name="textTight" class="solr.TextField"
> positionIncrementGap="100" >
> <analyzer>
> <tokenizer class="solr.WhitespaceTokenizerFactory"/>
> <filter class="solr.SynonymFilterFactory" synonyms="synonyms.txt"
> ignoreCase="true" expand="false"/>
> <filter class="solr.StopFilterFactory" ignoreCase="true"/>
> <filter class="solr.WordDelimiterFilterFactory"
> generateWordParts="0" generateNumberParts="0" catenateWords="1"
> catenateNumbers="1" catenateAll="0"/>
> <filter class="solr.LowerCaseFilterFactory"/>
> <filter class="solr.EnglishPorterFilterFactory"
> protected="protwords.txt"/>
> </analyzer>
> </fieldtype>
> </types>
> <fields>
> <!-- Valid attributes for fields:
> name: mandatory - the name for the field
> type: mandatory - the name of a previously defined type from the
> <types> section
> indexed: true if this field should be indexed (searchable)
> stored: true if this field should be retrievable
> multiValued: true if this field may contain multiple values per
> document
> omitNorms: (expert) set to true to omit the norms associated with
> this field
> (this disables length normalization and index-time
> boosting for the field)
> -->
> <field name="id" type="string" indexed="true" stored="true"
> multiValued="false"/>
> <field name="inst_code" type="text" indexed="true" stored="true"
> multiValued="true" required="false"/>
> <field name="inst_name" type="text" indexed="true" stored="true"
> multiValued="true" required="false"/>
> <field name="meas_name" type="text" indexed="true" stored="true"
> multiValued="true" required="false"/>
> <field name="latitude" type="sfloat" class="solr.FloatField"
> indexed="true" stored="true" required="false"/>
> <field name="longitude" type="sfloat" class="solr.FloatField"
> indexed="true" stored="true" required="false"/>
> <field name="ob_id" type="string" indexed="true" stored="true"
> multiValued="true"/>
> <field name="in_id" type="string" indexed="true" stored="true"
> multiValued="true"/>
> <field name="ob_name" type="text" indexed="true" stored="true"
> multiValued="true"/>
>
> <!-- catchall field, containing all other searchable text fields
> (implemented
> via copyField further on in this schema -->
> <field name="text" type="text" indexed="true" stored="false"
> multiValued="true" required="false"/>
>
>
> <!-- non-tokenized version of manufacturer to make it easier to sort or
> group
> results by manufacturer. copied from "manu" via copyField -->
> <field name="manu_exact" type="string" indexed="true" stored="false"
> required="false"/>
>
>
> <!-- Dynamic field definitions. If a field name is not found,
> dynamicFields
> will be used if the name matches any of the patterns.
> RESTRICTION: the glob-like pattern in the name attribute must have
> a "*" only at the start or the end.
> EXAMPLE: name="*_i" will match any field ending in _i (like myid_i,
> z_i)
> Longer patterns will be matched first. if equal size patterns
> both match, the first appearing in the schema will be used. -->
> <dynamicField name="*_i" type="sint" indexed="true" stored="true"/>
> <dynamicField name="*_s" type="string" indexed="true" stored="true"/>
> <dynamicField name="*_l" type="slong" indexed="true" stored="true"/>
> <dynamicField name="*_t" type="text" indexed="true" stored="true"/>
> <dynamicField name="*_b" type="boolean" indexed="true" stored="true"/>
> <dynamicField name="*_f" type="sfloat" indexed="true" stored="true"/>
> <dynamicField name="*_d" type="sdouble" indexed="true" stored="true"/>
> <dynamicField name="*_dt" type="date" indexed="true" stored="true"/>
> </fields>
>
> <!-- field to use to determine and enforce document uniqueness. -->
> <uniqueKey>id</uniqueKey>
>
> <!-- field for the QueryParser to use when an explicit fieldname is absent
> -->
> <defaultSearchField>text</defaultSearchField>
>
> <!-- SolrQueryParser configuration: defaultOperator="AND|OR" -->
> <solrQueryParser defaultOperator="AND"/>
>
> <!-- copyField commands copy one field to another at the time a document
> is added to the index. It's used either to index the same field
> different
> ways, or to add multiple fields to the same field for easier/faster
> searching. -->
>
>
>
> <!-- Similarity is the scoring routine for each document vs a query.
> A custom similarity may be specified here, but the default is fine
> for most applications. -->
> <!-- <similarity class="org.apache.lucene.search.DefaultSimilarity"/> -->
>
> </schema>
> -------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>
> On Wed, Nov 12, 2008 at 11:01 PM, Noble Paul നോബിള് नोब्ळ् <
> noble.paul@gmail.com> wrote:
>
>> the fact that it got committed in the end suggests there was no error in
>> between
>>
>> look at the status url and see the no:of rows returned etc.
>>
>> It gives a clue as to what would have really happened. or you can
>> paste your dataconfig and status xmls and we may be able to suggest
>> something
>>
>> On Thu, Nov 13, 2008 at 9:26 AM, Giri <gi...@gmail.com> wrote:
>> > Hi Noble,
>> >
>> > thanks for reply, my comments are below
>> >
>> >>>why is the id field multivalued?
>> > I was just trying various options, yes, this ID is unique, and I check
>> for
>> > duplicates, when I did a distinct (id) query to the MySQL database, it
>> > returned almost 2 million.
>> >
>> >>> look at the status host:post/dataimport gives you the status
>> > I constantly checked the status using the dataimport URL, the status
>> was
>> > increased upto 600K records, then it stopped increasing, then took few
>> > minutes to commit the indexed data.
>> >
>> >
>> > On Tue, Nov 11, 2008 at 11:35 PM, Noble Paul നോബിള് नोब्ळ् <
>> > noble.paul@gmail.com> wrote:
>> >
>> >> why is the id field multivalued? is there a uniqueKey in the schema ?
>> >> Are you sure there are no duplicates?
>> >>
>> >> look at the status host:post/dataimport gives you the status
>> >> it can give you some clue
>> >>
>> >> --Noble
>> >>
>> >>
>> >> On Wed, Nov 12, 2008 at 4:53 AM, Giri <gi...@gmail.com> wrote:
>> >> > Hi,
>> >> >
>> >> > I have about ~ 2 million records in a mySQL database table (about 9
>> >> fields
>> >> > from a single table), and I am trying to load it to the solr using
>> >> > DataImportHandler using the command=full-import option. it only
>> indexed
>> >> > about 615360 records out of 2 millions.
>> >> >
>> >> > here is my db-data-config.xml
>> >> > <dataConfig>
>> >> > <dataSource type="JdbcDataSource" driver="com.mysql.jdbc.Driver"
>> >> > url="jdbc:mysql://localhost:3306/mydb" user="ua" password="pw"
>> batchSize
>> >> > ="-1"/>
>> >> > <document name="climate">
>> >> > <entity name="occurence" query="select * from mylargetable">
>> >> > <field column="id" name="id" />
>> >> > <field column="title" name="title" />
>> >> > <field column="url" name="url" />
>> >> > </entity>
>> >> > </document>
>> >> > </dataConfig>
>> >> >
>> >> > and in my solr schema.xml, i define these fields as:
>> >> >
>> >> > <field name="id" type="string" indexed="true" stored="true"
>> >> > multiValued="true"/>
>> >> > <field name="title" type="text" indexed="true" stored="true"
>> >> > multiValued="true" required="false"/>
>> >> > <field name="url" type="text" indexed="true" stored="true"
>> >> > multiValued="true" required="false"/>
>> >> >
>> >> >
>> >> > If I try to index just one field (id), then it indexes about 960000
>> >> records,
>> >> > but if I try to index all the above three fields, it indexes only
>> 615360
>> >> > records.
>> >> >
>> >> > Any help will be appreciated.
>> >> >
>> >> > thanks!
>> >> >
>> >>
>> >>
>> >>
>> >> --
>> >> --Noble Paul
>> >>
>> >
>>
>>
>>
>> --
>> --Noble Paul
>>
>
--
--Noble Paul
Re: DataImportHandler not indexing all the records
Posted by Giri <gi...@gmail.com>.
Hi Noble,
thanks for the help, here are the details: the field "id" is unique, when I
did a select distinct(id), it returned 1 million rows.
-------------------------------------------------------------------
db-data-config.xml
note: I limit the resultset to 1 million in the select query
-------------------------------------------------------------------
<dataConfig>
<dataSource type="JdbcDataSource" driver="com.mysql.jdbc.Driver"
url="jdbc:mysql://localhost:3306/climatedata" user="user" password="pw"
batchSize ="-1"/>
<document name="climateRecord">
<entity name="observation" query="select * from
climatedata.ws_record limit 1000000">
<field column="id" name="id" />
<field column="inst_code" name="inst_code" />
<field column="inst_name" name="inst_name" />
<field column="meas_name" name="meas_name" />
<field column="latitude" name="latitude" />
<field column="longitude" name="longitude" />
<field column="ob_id" name="ob_id" />
<field column="in_id" name="in_id" />
<field column="ob_name" name="ob_name" />
</entity>
</document>
</dataConfig>
-----------------------------------------------------------------
in the solr Schema.xml:
----------------------------------------------------------------
<fields>
<field name="id" type="string" indexed="true" stored="true"
multiValued="false"/>
<field name="inst_code" type="text" indexed="true" stored="true"
multiValued="true" required="false"/>
<field name="inst_name" type="text" indexed="true" stored="true"
multiValued="true" required="false"/>
<field name="meas_name" type="text" indexed="true" stored="true"
multiValued="true" required="false"/>
<field name="latitude" type="sfloat" class="solr.FloatField"
indexed="true" stored="true" required="false"/>
<field name="longitude" type="sfloat" class="solr.FloatField"
indexed="true" stored="true" required="false"/>
<field name="ob_id" type="string" indexed="true" stored="true"
multiValued="true"/>
<field name="in_id" type="string" indexed="true" stored="true"
multiValued="true"/>
<field name="ob_name" type="text" indexed="true" stored="true"
multiValued="true"/>
<!-- catchall field, containing all other searchable text fields
(implemented
via copyField further on in this schema -->
<field name="text" type="text" indexed="true" stored="false"
multiValued="true" required="false"/>
<!-- non-tokenized version of manufacturer to make it easier to sort or
group
results by manufacturer. copied from "manu" via copyField -->
<field name="manu_exact" type="string" indexed="true" stored="false"
required="false"/>
<!-- Dynamic field definitions. If a field name is not found,
dynamicFields
will be used if the name matches any of the patterns.
RESTRICTION: the glob-like pattern in the name attribute must have
a "*" only at the start or the end.
EXAMPLE: name="*_i" will match any field ending in _i (like myid_i,
z_i)
Longer patterns will be matched first. if equal size patterns
both match, the first appearing in the schema will be used. -->
<dynamicField name="*_i" type="sint" indexed="true" stored="true"/>
<dynamicField name="*_s" type="string" indexed="true" stored="true"/>
<dynamicField name="*_l" type="slong" indexed="true" stored="true"/>
<dynamicField name="*_t" type="text" indexed="true" stored="true"/>
<dynamicField name="*_b" type="boolean" indexed="true" stored="true"/>
<dynamicField name="*_f" type="sfloat" indexed="true" stored="true"/>
<dynamicField name="*_d" type="sdouble" indexed="true" stored="true"/>
<dynamicField name="*_dt" type="date" indexed="true" stored="true"/>
</fields>
----------------------------------------------------
I run the index via firefox browser using
http://localhost:8080/solr/dataimport?command=full-import
I checked the status using
http://localhost:8080/solr/dataimport?command=status
initially the status increased steadily, but after reaching 613071, the
status stayed for a while (as below), and then it displayed the completed
message :
----------------------------------------------------
<response>
-
<lst name="responseHeader">
<int name="status">0</int>
<int name="QTime">1</int>
</lst>
-
<lst name="initArgs">
-
<lst name="defaults">
<str name="config">db-data-config.xml</str>
</lst>
</lst>
<str name="command">status</str>
<str name="status">busy</str>
<str name="importResponse">A command is still running...</str>
-
<lst name="statusMessages">
<str name="Time Elapsed">0:3:24.266</str>
<str name="Total Requests made to DataSource">1</str>
<str name="Total Rows Fetched">613071</str>
<str name="Total Documents Processed">613070</str>
<str name="Total Documents Skipped">0</str>
<str name="Full Dump Started">2008-11-14 12:12:16</str>
</lst>
-
<str name="WARNING">
This response format is experimental. It is likely to change in the future.
</str>
</response>
-----------------------------------------------------------
>>NOTE: this is the status result after it completed
-----------------------------------------------------------
<response>
-
<lst name="responseHeader">
<int name="status">0</int>
<int name="QTime">1</int>
</lst>
-
<lst name="initArgs">
-
<lst name="defaults">
<str name="config">db-data-config.xml</str>
</lst>
</lst>
<str name="command">status</str>
<str name="status">idle</str>
<str name="importResponse"/>
-
<lst name="statusMessages">
<str name="Total Requests made to DataSource">1</str>
<str name="Total Rows Fetched">615360</str>
<str name="Total Documents Skipped">0</str>
<str name="Full Dump Started">2008-11-14 12:12:16</str>
-
<str name="">
Indexing completed. Added/Updated: 615360 documents. Deleted 0 documents.
</str>
<str name="Committed">2008-11-14 12:16:32</str>
<str name="Optimized">2008-11-14 12:16:32</str>
<str name="Time taken ">0:4:16.154</str>
</lst>
-
<str name="WARNING">
This response format is experimental. It is likely to change in the future.
</str>
</response>
-----------------------------------------------------
here is the full solr scehma.xml content:
----------------------------------------------------
<?xml version="1.0" ?>
<!-- The Solr schema file. This file should be named "schema.xml" and
should be in the conf directory under the solr home
(i.e. ./solr/conf/schema.xml by default)
or located where the classloader for the Solr webapp can find it.
For more information, on how to customize this file, please see...
http://wiki.apache.org/solr/SchemaXml
-->
<schema name="example" version="1.1">
<types>
<!-- field type definitions. The "name" attribute is
just a label to be used by field definitions. The "class"
attribute and any other attributes determine the real
behavior of the fieldtype. -->
<!-- The StringField type is not analyzed, but indexed/stored verbatim
-->
<fieldtype name="string" class="solr.StrField" sortMissingLast="true"/>
<!-- boolean type: "true" or "false" -->
<fieldtype name="boolean" class="solr.BoolField"
sortMissingLast="true"/>
<!-- The optional sortMissingLast and sortMissingFirst attributes are
currently supported on types that are sorted internally as a
strings.
- If sortMissingLast="true" then a sort on this field will cause
documents
without the field to come after documents with the field,
regardless of the requested sort order (asc or desc).
- If sortMissingFirst="true" then a sort on this field will cause
documents
without the field to come before documents with the field,
regardless of the requested sort order.
- If sortMissingLast="false" and sortMissingFirst="false" (the
default),
then default lucene sorting will be used which places docs without
the field
first in an ascending sort and last in a descending sort.
-->
<!-- numeric field types that store and index the text
value verbatim (and hence don't support range queries since the
lexicographic ordering isn't equal to the numeric ordering) -->
<fieldtype name="integer" class="solr.IntField"/>
<fieldtype name="long" class="solr.LongField"/>
<fieldtype name="float" class="solr.FloatField"/>
<fieldtype name="double" class="solr.DoubleField"/>
<!-- Numeric field types that manipulate the value into
a string value that isn't human readable in it's internal form,
but with a lexicographic ordering the same as the numeric ordering
so that range queries correctly work. -->
<fieldtype name="sint" class="solr.SortableIntField"
sortMissingLast="true"/>
<fieldtype name="slong" class="solr.SortableLongField"
sortMissingLast="true"/>
<fieldtype name="sfloat" class="solr.SortableFloatField"
sortMissingLast="true"/>
<fieldtype name="sdouble" class="solr.SortableDoubleField"
sortMissingLast="true"/>
<!-- The format for this date field is of the form 1995-12-31T23:59:59Z,
and
is a more restricted form of the canonical representation of
dateTime
http://www.w3.org/TR/xmlschema-2/#dateTime
The trailing "Z" designates UTC time and is mandatory.
Optional fractional seconds are allowed: 1995-12-31T23:59:59.999Z
All other components are mandatory. -->
<fieldtype name="date" class="solr.DateField" sortMissingLast="true"/>
<!-- solr.TextField allows the specification of custom text analyzers
specified as a tokenizer and a list of token filters. Different
analyzers may be specified for indexing and querying.
The optional positionIncrementGap puts space between multiple
fields of
this type on the same document, with the purpose of preventing
false phrase
matching across fields.
For more info on customizing your analyzer chain, please see...
http://wiki.apache.org/solr/AnalyzersTokenizersTokenFilters
-->
<!-- Standard analyzer commonly used by Lucene developers
-->
<!-- Standard analyzer commonly used by Lucene developers -->
<fieldtype name="text_lu" class="solr.TextField"
positionIncrementGap="100">
<analyzer>
<tokenizer class="solr.StandardTokenizerFactory"/>
<filter class="solr.StandardFilterFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
<filter class="solr.StopFilterFactory"/>
<filter class="solr.EnglishPorterFilterFactory"/>
</analyzer>
</fieldtype>
<!-- One could also specify an existing Analyzer implementation in Java
via the class attribute on the analyzer element:
<fieldtype name="text_lu" class="solr.TextField">
<analyzer
class="org.apache.lucene.analysis.snowball.SnowballAnalyzer"/>
</fieldType>
-->
<!-- A text field that only splits on whitespace for more exact matching
-->
<fieldtype name="text_ws" class="solr.TextField"
positionIncrementGap="100">
<analyzer>
<tokenizer class="solr.WhitespaceTokenizerFactory"/>
</analyzer>
</fieldtype>
<!-- A text field that uses WordDelimiterFilter to enable splitting and
matching of
words on case-change, alpha numeric boundaries, and non-alphanumeric
chars
so that a query of "wifi" or "wi fi" could match a document
containing "Wi-Fi".
Synonyms and stopwords are customized by external files, and
stemming is enabled -->
<fieldtype name="text" class="solr.TextField"
positionIncrementGap="100">
<analyzer type="index">
<tokenizer class="solr.WhitespaceTokenizerFactory"/>
<!-- in this example, we will only use synonyms at query time
<filter class="solr.SynonymFilterFactory"
synonyms="index_synonyms.txt" ignoreCase="true" expand="false"/>
-->
<!--<filter class="solr.WordDelimiterFilterFactory"
generateWordParts="1"/>-->
<filter class="solr.StopFilterFactory" ignoreCase="true"/>
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
<analyzer type="query">
<tokenizer class="solr.WhitespaceTokenizerFactory"/>
<filter class="solr.StopFilterFactory" ignoreCase="true"/>
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
</fieldtype>
<!-- Less flexible matching, but less false matches. Probably not ideal
for product names
but may be good for SKUs. Can insert dashes in the wrong place and
still match. -->
<fieldtype name="textTight" class="solr.TextField"
positionIncrementGap="100" >
<analyzer>
<tokenizer class="solr.WhitespaceTokenizerFactory"/>
<filter class="solr.SynonymFilterFactory" synonyms="synonyms.txt"
ignoreCase="true" expand="false"/>
<filter class="solr.StopFilterFactory" ignoreCase="true"/>
<filter class="solr.WordDelimiterFilterFactory"
generateWordParts="0" generateNumberParts="0" catenateWords="1"
catenateNumbers="1" catenateAll="0"/>
<filter class="solr.LowerCaseFilterFactory"/>
<filter class="solr.EnglishPorterFilterFactory"
protected="protwords.txt"/>
</analyzer>
</fieldtype>
</types>
<fields>
<!-- Valid attributes for fields:
name: mandatory - the name for the field
type: mandatory - the name of a previously defined type from the
<types> section
indexed: true if this field should be indexed (searchable)
stored: true if this field should be retrievable
multiValued: true if this field may contain multiple values per
document
omitNorms: (expert) set to true to omit the norms associated with
this field
(this disables length normalization and index-time
boosting for the field)
-->
<field name="id" type="string" indexed="true" stored="true"
multiValued="false"/>
<field name="inst_code" type="text" indexed="true" stored="true"
multiValued="true" required="false"/>
<field name="inst_name" type="text" indexed="true" stored="true"
multiValued="true" required="false"/>
<field name="meas_name" type="text" indexed="true" stored="true"
multiValued="true" required="false"/>
<field name="latitude" type="sfloat" class="solr.FloatField"
indexed="true" stored="true" required="false"/>
<field name="longitude" type="sfloat" class="solr.FloatField"
indexed="true" stored="true" required="false"/>
<field name="ob_id" type="string" indexed="true" stored="true"
multiValued="true"/>
<field name="in_id" type="string" indexed="true" stored="true"
multiValued="true"/>
<field name="ob_name" type="text" indexed="true" stored="true"
multiValued="true"/>
<!-- catchall field, containing all other searchable text fields
(implemented
via copyField further on in this schema -->
<field name="text" type="text" indexed="true" stored="false"
multiValued="true" required="false"/>
<!-- non-tokenized version of manufacturer to make it easier to sort or
group
results by manufacturer. copied from "manu" via copyField -->
<field name="manu_exact" type="string" indexed="true" stored="false"
required="false"/>
<!-- Dynamic field definitions. If a field name is not found,
dynamicFields
will be used if the name matches any of the patterns.
RESTRICTION: the glob-like pattern in the name attribute must have
a "*" only at the start or the end.
EXAMPLE: name="*_i" will match any field ending in _i (like myid_i,
z_i)
Longer patterns will be matched first. if equal size patterns
both match, the first appearing in the schema will be used. -->
<dynamicField name="*_i" type="sint" indexed="true" stored="true"/>
<dynamicField name="*_s" type="string" indexed="true" stored="true"/>
<dynamicField name="*_l" type="slong" indexed="true" stored="true"/>
<dynamicField name="*_t" type="text" indexed="true" stored="true"/>
<dynamicField name="*_b" type="boolean" indexed="true" stored="true"/>
<dynamicField name="*_f" type="sfloat" indexed="true" stored="true"/>
<dynamicField name="*_d" type="sdouble" indexed="true" stored="true"/>
<dynamicField name="*_dt" type="date" indexed="true" stored="true"/>
</fields>
<!-- field to use to determine and enforce document uniqueness. -->
<uniqueKey>id</uniqueKey>
<!-- field for the QueryParser to use when an explicit fieldname is absent
-->
<defaultSearchField>text</defaultSearchField>
<!-- SolrQueryParser configuration: defaultOperator="AND|OR" -->
<solrQueryParser defaultOperator="AND"/>
<!-- copyField commands copy one field to another at the time a document
is added to the index. It's used either to index the same field
different
ways, or to add multiple fields to the same field for easier/faster
searching. -->
<!-- Similarity is the scoring routine for each document vs a query.
A custom similarity may be specified here, but the default is fine
for most applications. -->
<!-- <similarity class="org.apache.lucene.search.DefaultSimilarity"/> -->
</schema>
-------------------------------------------------------------------------------------------------------------------------------------------------------------
On Wed, Nov 12, 2008 at 11:01 PM, Noble Paul നോബിള് नोब्ळ् <
noble.paul@gmail.com> wrote:
> the fact that it got committed in the end suggests there was no error in
> between
>
> look at the status url and see the no:of rows returned etc.
>
> It gives a clue as to what would have really happened. or you can
> paste your dataconfig and status xmls and we may be able to suggest
> something
>
> On Thu, Nov 13, 2008 at 9:26 AM, Giri <gi...@gmail.com> wrote:
> > Hi Noble,
> >
> > thanks for reply, my comments are below
> >
> >>>why is the id field multivalued?
> > I was just trying various options, yes, this ID is unique, and I check
> for
> > duplicates, when I did a distinct (id) query to the MySQL database, it
> > returned almost 2 million.
> >
> >>> look at the status host:post/dataimport gives you the status
> > I constantly checked the status using the dataimport URL, the status
> was
> > increased upto 600K records, then it stopped increasing, then took few
> > minutes to commit the indexed data.
> >
> >
> > On Tue, Nov 11, 2008 at 11:35 PM, Noble Paul നോബിള് नोब्ळ् <
> > noble.paul@gmail.com> wrote:
> >
> >> why is the id field multivalued? is there a uniqueKey in the schema ?
> >> Are you sure there are no duplicates?
> >>
> >> look at the status host:post/dataimport gives you the status
> >> it can give you some clue
> >>
> >> --Noble
> >>
> >>
> >> On Wed, Nov 12, 2008 at 4:53 AM, Giri <gi...@gmail.com> wrote:
> >> > Hi,
> >> >
> >> > I have about ~ 2 million records in a mySQL database table (about 9
> >> fields
> >> > from a single table), and I am trying to load it to the solr using
> >> > DataImportHandler using the command=full-import option. it only
> indexed
> >> > about 615360 records out of 2 millions.
> >> >
> >> > here is my db-data-config.xml
> >> > <dataConfig>
> >> > <dataSource type="JdbcDataSource" driver="com.mysql.jdbc.Driver"
> >> > url="jdbc:mysql://localhost:3306/mydb" user="ua" password="pw"
> batchSize
> >> > ="-1"/>
> >> > <document name="climate">
> >> > <entity name="occurence" query="select * from mylargetable">
> >> > <field column="id" name="id" />
> >> > <field column="title" name="title" />
> >> > <field column="url" name="url" />
> >> > </entity>
> >> > </document>
> >> > </dataConfig>
> >> >
> >> > and in my solr schema.xml, i define these fields as:
> >> >
> >> > <field name="id" type="string" indexed="true" stored="true"
> >> > multiValued="true"/>
> >> > <field name="title" type="text" indexed="true" stored="true"
> >> > multiValued="true" required="false"/>
> >> > <field name="url" type="text" indexed="true" stored="true"
> >> > multiValued="true" required="false"/>
> >> >
> >> >
> >> > If I try to index just one field (id), then it indexes about 960000
> >> records,
> >> > but if I try to index all the above three fields, it indexes only
> 615360
> >> > records.
> >> >
> >> > Any help will be appreciated.
> >> >
> >> > thanks!
> >> >
> >>
> >>
> >>
> >> --
> >> --Noble Paul
> >>
> >
>
>
>
> --
> --Noble Paul
>
Re: DataImportHandler not indexing all the records
Posted by Noble Paul നോബിള് नोब्ळ् <no...@gmail.com>.
the fact that it got committed in the end suggests there was no error in between
look at the status url and see the no:of rows returned etc.
It gives a clue as to what would have really happened. or you can
paste your dataconfig and status xmls and we may be able to suggest
something
On Thu, Nov 13, 2008 at 9:26 AM, Giri <gi...@gmail.com> wrote:
> Hi Noble,
>
> thanks for reply, my comments are below
>
>>>why is the id field multivalued?
> I was just trying various options, yes, this ID is unique, and I check for
> duplicates, when I did a distinct (id) query to the MySQL database, it
> returned almost 2 million.
>
>>> look at the status host:post/dataimport gives you the status
> I constantly checked the status using the dataimport URL, the status was
> increased upto 600K records, then it stopped increasing, then took few
> minutes to commit the indexed data.
>
>
> On Tue, Nov 11, 2008 at 11:35 PM, Noble Paul നോബിള് नोब्ळ् <
> noble.paul@gmail.com> wrote:
>
>> why is the id field multivalued? is there a uniqueKey in the schema ?
>> Are you sure there are no duplicates?
>>
>> look at the status host:post/dataimport gives you the status
>> it can give you some clue
>>
>> --Noble
>>
>>
>> On Wed, Nov 12, 2008 at 4:53 AM, Giri <gi...@gmail.com> wrote:
>> > Hi,
>> >
>> > I have about ~ 2 million records in a mySQL database table (about 9
>> fields
>> > from a single table), and I am trying to load it to the solr using
>> > DataImportHandler using the command=full-import option. it only indexed
>> > about 615360 records out of 2 millions.
>> >
>> > here is my db-data-config.xml
>> > <dataConfig>
>> > <dataSource type="JdbcDataSource" driver="com.mysql.jdbc.Driver"
>> > url="jdbc:mysql://localhost:3306/mydb" user="ua" password="pw" batchSize
>> > ="-1"/>
>> > <document name="climate">
>> > <entity name="occurence" query="select * from mylargetable">
>> > <field column="id" name="id" />
>> > <field column="title" name="title" />
>> > <field column="url" name="url" />
>> > </entity>
>> > </document>
>> > </dataConfig>
>> >
>> > and in my solr schema.xml, i define these fields as:
>> >
>> > <field name="id" type="string" indexed="true" stored="true"
>> > multiValued="true"/>
>> > <field name="title" type="text" indexed="true" stored="true"
>> > multiValued="true" required="false"/>
>> > <field name="url" type="text" indexed="true" stored="true"
>> > multiValued="true" required="false"/>
>> >
>> >
>> > If I try to index just one field (id), then it indexes about 960000
>> records,
>> > but if I try to index all the above three fields, it indexes only 615360
>> > records.
>> >
>> > Any help will be appreciated.
>> >
>> > thanks!
>> >
>>
>>
>>
>> --
>> --Noble Paul
>>
>
--
--Noble Paul
Re: DataImportHandler not indexing all the records
Posted by Giri <gi...@gmail.com>.
Hi Noble,
thanks for reply, my comments are below
>>why is the id field multivalued?
I was just trying various options, yes, this ID is unique, and I check for
duplicates, when I did a distinct (id) query to the MySQL database, it
returned almost 2 million.
>> look at the status host:post/dataimport gives you the status
I constantly checked the status using the dataimport URL, the status was
increased upto 600K records, then it stopped increasing, then took few
minutes to commit the indexed data.
On Tue, Nov 11, 2008 at 11:35 PM, Noble Paul നോബിള് नोब्ळ् <
noble.paul@gmail.com> wrote:
> why is the id field multivalued? is there a uniqueKey in the schema ?
> Are you sure there are no duplicates?
>
> look at the status host:post/dataimport gives you the status
> it can give you some clue
>
> --Noble
>
>
> On Wed, Nov 12, 2008 at 4:53 AM, Giri <gi...@gmail.com> wrote:
> > Hi,
> >
> > I have about ~ 2 million records in a mySQL database table (about 9
> fields
> > from a single table), and I am trying to load it to the solr using
> > DataImportHandler using the command=full-import option. it only indexed
> > about 615360 records out of 2 millions.
> >
> > here is my db-data-config.xml
> > <dataConfig>
> > <dataSource type="JdbcDataSource" driver="com.mysql.jdbc.Driver"
> > url="jdbc:mysql://localhost:3306/mydb" user="ua" password="pw" batchSize
> > ="-1"/>
> > <document name="climate">
> > <entity name="occurence" query="select * from mylargetable">
> > <field column="id" name="id" />
> > <field column="title" name="title" />
> > <field column="url" name="url" />
> > </entity>
> > </document>
> > </dataConfig>
> >
> > and in my solr schema.xml, i define these fields as:
> >
> > <field name="id" type="string" indexed="true" stored="true"
> > multiValued="true"/>
> > <field name="title" type="text" indexed="true" stored="true"
> > multiValued="true" required="false"/>
> > <field name="url" type="text" indexed="true" stored="true"
> > multiValued="true" required="false"/>
> >
> >
> > If I try to index just one field (id), then it indexes about 960000
> records,
> > but if I try to index all the above three fields, it indexes only 615360
> > records.
> >
> > Any help will be appreciated.
> >
> > thanks!
> >
>
>
>
> --
> --Noble Paul
>
Re: DataImportHandler not indexing all the records
Posted by Noble Paul നോബിള് नोब्ळ् <no...@gmail.com>.
why is the id field multivalued? is there a uniqueKey in the schema ?
Are you sure there are no duplicates?
look at the status host:post/dataimport gives you the status
it can give you some clue
--Noble
On Wed, Nov 12, 2008 at 4:53 AM, Giri <gi...@gmail.com> wrote:
> Hi,
>
> I have about ~ 2 million records in a mySQL database table (about 9 fields
> from a single table), and I am trying to load it to the solr using
> DataImportHandler using the command=full-import option. it only indexed
> about 615360 records out of 2 millions.
>
> here is my db-data-config.xml
> <dataConfig>
> <dataSource type="JdbcDataSource" driver="com.mysql.jdbc.Driver"
> url="jdbc:mysql://localhost:3306/mydb" user="ua" password="pw" batchSize
> ="-1"/>
> <document name="climate">
> <entity name="occurence" query="select * from mylargetable">
> <field column="id" name="id" />
> <field column="title" name="title" />
> <field column="url" name="url" />
> </entity>
> </document>
> </dataConfig>
>
> and in my solr schema.xml, i define these fields as:
>
> <field name="id" type="string" indexed="true" stored="true"
> multiValued="true"/>
> <field name="title" type="text" indexed="true" stored="true"
> multiValued="true" required="false"/>
> <field name="url" type="text" indexed="true" stored="true"
> multiValued="true" required="false"/>
>
>
> If I try to index just one field (id), then it indexes about 960000 records,
> but if I try to index all the above three fields, it indexes only 615360
> records.
>
> Any help will be appreciated.
>
> thanks!
>
--
--Noble Paul