You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Erik Earle <er...@yahoo.com> on 2009/08/19 00:16:47 UTC

DataImportHandler ignoring most rows

Using:
- apache-solr-1.3.0
- java 1.6 
- tomcat 6
- sql server 2005 w/ JSQLConnect 4.0 driver

I have a group table with 3007 rows.  I have confirmed the key is
unique with "select distinct id from group"  and it returns 3007.  When i re-index using http://host:port/solr/dataimport?command=full-import  I only get 7 records indexed.  Any insight into what is going on would be really great.  

A partial response:
    <lst name="statusMessages">
    <str name="Total Requests made to DataSource">1</str>
    <str name="Total Rows Fetched">7</str>
    <str name="Total Documents Skipped">0</str>


I have other entities that index all the rows without issue.

There are no errors in the logs.

I am not using any Transformers (and most of my config is not changed from install)

My schema.xml contains:

     <uniqueKey>key</uniqueKey>

and field defs (not a full list of fields):
   <field name="key" type="string" indexed="true" stored="true" required="true" /> 
   <field name="class" type="string" indexed="true" stored="true" required="true" />
   <field name="id" type="string" indexed="true" stored="true" /> 
   <field name="description" type="text" indexed="true" stored="true" />
   <field name="created" type="date" indexed="true" stored="true" />
   <field name="updated" type="date" indexed="true" stored="true" />

data-config.xml
################
<dataConfig>
    <!-- jdbc:JSQLConnect://se-eriearle-lt1/database=SocialSite2/user=SocialSite2/logfile=DB_TRACE.log -->
    <dataSource type="JdbcDataSource" 
        driver="com.jnetdirect.jsql.JSQLDriver"
        url="jdbc:JSQLConnect://se-eriearle-lt1/database=SocialSite2/user=SocialSite2" 
        user="SocialSite2" 
        password="SocialSite2" />

    <document>
       
    ....

        <entity name="Group"  pk="key"
            query="select 'group.'+id as 'key', 'group' as 'class', name, handle, description, created, updated from group order by created asc">
        </entity>

        <entity name="Message" pk="key"
            query="<...redacted...>">
        </entity>
   
    </document>
</dataConfig>


      

Re: DataImportHandler ignoring most rows

Posted by Lance Norskog <go...@gmail.com>.
It usually helps to make a database view of your query, and then load the
DIH from that view. There are cases where some query syntaxes are mangled on
the way to the DB.

2009/8/18 Noble Paul നോബിള്‍ नोब्ळ् <no...@corp.aol.com>

> this comment says that
>   <str name="Total Rows Fetched">7</str>
>
> the query fetched only 7 rows. If possible open a tool and just run
> the same query and see how many rows are returned
>
> On Wed, Aug 19, 2009 at 3:46 AM, Erik Earle<er...@yahoo.com> wrote:
> > Using:
> > - apache-solr-1.3.0
> > - java 1.6
> > - tomcat 6
> > - sql server 2005 w/ JSQLConnect 4.0 driver
> >
> > I have a group table with 3007 rows.  I have confirmed the key is
> > unique with "select distinct id from group"  and it returns 3007.  When i
> re-index using http://host:port/solr/dataimport?command=full-import  I
> only get 7 records indexed.  Any insight into what is going on would be
> really great.
> >
> > A partial response:
> >    <lst name="statusMessages">
> >    <str name="Total Requests made to DataSource">1</str>
> >    <str name="Total Rows Fetched">7</str>
> >    <str name="Total Documents Skipped">0</str>
> >
> >
> > I have other entities that index all the rows without issue.
> >
> > There are no errors in the logs.
> >
> > I am not using any Transformers (and most of my config is not changed
> from install)
> >
> > My schema.xml contains:
> >
> >     <uniqueKey>key</uniqueKey>
> >
> > and field defs (not a full list of fields):
> >   <field name="key" type="string" indexed="true" stored="true"
> required="true" />
> >   <field name="class" type="string" indexed="true" stored="true"
> required="true" />
> >   <field name="id" type="string" indexed="true" stored="true" />
> >   <field name="description" type="text" indexed="true" stored="true" />
> >   <field name="created" type="date" indexed="true" stored="true" />
> >   <field name="updated" type="date" indexed="true" stored="true" />
> >
> > data-config.xml
> > ################
> > <dataConfig>
> >    <!--
> jdbc:JSQLConnect://se-eriearle-lt1/database=SocialSite2/user=SocialSite2/logfile=DB_TRACE.log
> -->
> >    <dataSource type="JdbcDataSource"
> >        driver="com.jnetdirect.jsql.JSQLDriver"
> >
>  url="jdbc:JSQLConnect://se-eriearle-lt1/database=SocialSite2/user=SocialSite2"
> >        user="SocialSite2"
> >        password="SocialSite2" />
> >
> >    <document>
> >
> >    ....
> >
> >        <entity name="Group"  pk="key"
> >            query="select 'group.'+id as 'key', 'group' as 'class', name,
> handle, description, created, updated from group order by created asc">
> >        </entity>
> >
> >        <entity name="Message" pk="key"
> >            query="<...redacted...>">
> >        </entity>
> >
> >    </document>
> > </dataConfig>
> >
> >
> >
> >
>
>
>
> --
> -----------------------------------------------------
> Noble Paul | Principal Engineer| AOL | http://aol.com
>



-- 
Lance Norskog
goksron@gmail.com

Re: DataImportHandler ignoring most rows

Posted by Noble Paul നോബിള്‍ नोब्ळ् <no...@corp.aol.com>.
this comment says that
   <str name="Total Rows Fetched">7</str>

the query fetched only 7 rows. If possible open a tool and just run
the same query and see how many rows are returned

On Wed, Aug 19, 2009 at 3:46 AM, Erik Earle<er...@yahoo.com> wrote:
> Using:
> - apache-solr-1.3.0
> - java 1.6
> - tomcat 6
> - sql server 2005 w/ JSQLConnect 4.0 driver
>
> I have a group table with 3007 rows.  I have confirmed the key is
> unique with "select distinct id from group"  and it returns 3007.  When i re-index using http://host:port/solr/dataimport?command=full-import  I only get 7 records indexed.  Any insight into what is going on would be really great.
>
> A partial response:
>    <lst name="statusMessages">
>    <str name="Total Requests made to DataSource">1</str>
>    <str name="Total Rows Fetched">7</str>
>    <str name="Total Documents Skipped">0</str>
>
>
> I have other entities that index all the rows without issue.
>
> There are no errors in the logs.
>
> I am not using any Transformers (and most of my config is not changed from install)
>
> My schema.xml contains:
>
>     <uniqueKey>key</uniqueKey>
>
> and field defs (not a full list of fields):
>   <field name="key" type="string" indexed="true" stored="true" required="true" />
>   <field name="class" type="string" indexed="true" stored="true" required="true" />
>   <field name="id" type="string" indexed="true" stored="true" />
>   <field name="description" type="text" indexed="true" stored="true" />
>   <field name="created" type="date" indexed="true" stored="true" />
>   <field name="updated" type="date" indexed="true" stored="true" />
>
> data-config.xml
> ################
> <dataConfig>
>    <!-- jdbc:JSQLConnect://se-eriearle-lt1/database=SocialSite2/user=SocialSite2/logfile=DB_TRACE.log -->
>    <dataSource type="JdbcDataSource"
>        driver="com.jnetdirect.jsql.JSQLDriver"
>        url="jdbc:JSQLConnect://se-eriearle-lt1/database=SocialSite2/user=SocialSite2"
>        user="SocialSite2"
>        password="SocialSite2" />
>
>    <document>
>
>    ....
>
>        <entity name="Group"  pk="key"
>            query="select 'group.'+id as 'key', 'group' as 'class', name, handle, description, created, updated from group order by created asc">
>        </entity>
>
>        <entity name="Message" pk="key"
>            query="<...redacted...>">
>        </entity>
>
>    </document>
> </dataConfig>
>
>
>
>



-- 
-----------------------------------------------------
Noble Paul | Principal Engineer| AOL | http://aol.com