You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Erik Earle <er...@yahoo.com> on 2009/08/19 00:16:47 UTC
DataImportHandler ignoring most rows
Using:
- apache-solr-1.3.0
- java 1.6
- tomcat 6
- sql server 2005 w/ JSQLConnect 4.0 driver
I have a group table with 3007 rows. I have confirmed the key is
unique with "select distinct id from group" and it returns 3007. When i re-index using http://host:port/solr/dataimport?command=full-import I only get 7 records indexed. Any insight into what is going on would be really great.
A partial response:
<lst name="statusMessages">
<str name="Total Requests made to DataSource">1</str>
<str name="Total Rows Fetched">7</str>
<str name="Total Documents Skipped">0</str>
I have other entities that index all the rows without issue.
There are no errors in the logs.
I am not using any Transformers (and most of my config is not changed from install)
My schema.xml contains:
<uniqueKey>key</uniqueKey>
and field defs (not a full list of fields):
<field name="key" type="string" indexed="true" stored="true" required="true" />
<field name="class" type="string" indexed="true" stored="true" required="true" />
<field name="id" type="string" indexed="true" stored="true" />
<field name="description" type="text" indexed="true" stored="true" />
<field name="created" type="date" indexed="true" stored="true" />
<field name="updated" type="date" indexed="true" stored="true" />
data-config.xml
################
<dataConfig>
<!-- jdbc:JSQLConnect://se-eriearle-lt1/database=SocialSite2/user=SocialSite2/logfile=DB_TRACE.log -->
<dataSource type="JdbcDataSource"
driver="com.jnetdirect.jsql.JSQLDriver"
url="jdbc:JSQLConnect://se-eriearle-lt1/database=SocialSite2/user=SocialSite2"
user="SocialSite2"
password="SocialSite2" />
<document>
....
<entity name="Group" pk="key"
query="select 'group.'+id as 'key', 'group' as 'class', name, handle, description, created, updated from group order by created asc">
</entity>
<entity name="Message" pk="key"
query="<...redacted...>">
</entity>
</document>
</dataConfig>
Re: DataImportHandler ignoring most rows
Posted by Lance Norskog <go...@gmail.com>.
It usually helps to make a database view of your query, and then load the
DIH from that view. There are cases where some query syntaxes are mangled on
the way to the DB.
2009/8/18 Noble Paul നോബിള് नोब्ळ् <no...@corp.aol.com>
> this comment says that
> <str name="Total Rows Fetched">7</str>
>
> the query fetched only 7 rows. If possible open a tool and just run
> the same query and see how many rows are returned
>
> On Wed, Aug 19, 2009 at 3:46 AM, Erik Earle<er...@yahoo.com> wrote:
> > Using:
> > - apache-solr-1.3.0
> > - java 1.6
> > - tomcat 6
> > - sql server 2005 w/ JSQLConnect 4.0 driver
> >
> > I have a group table with 3007 rows. I have confirmed the key is
> > unique with "select distinct id from group" and it returns 3007. When i
> re-index using http://host:port/solr/dataimport?command=full-import I
> only get 7 records indexed. Any insight into what is going on would be
> really great.
> >
> > A partial response:
> > <lst name="statusMessages">
> > <str name="Total Requests made to DataSource">1</str>
> > <str name="Total Rows Fetched">7</str>
> > <str name="Total Documents Skipped">0</str>
> >
> >
> > I have other entities that index all the rows without issue.
> >
> > There are no errors in the logs.
> >
> > I am not using any Transformers (and most of my config is not changed
> from install)
> >
> > My schema.xml contains:
> >
> > <uniqueKey>key</uniqueKey>
> >
> > and field defs (not a full list of fields):
> > <field name="key" type="string" indexed="true" stored="true"
> required="true" />
> > <field name="class" type="string" indexed="true" stored="true"
> required="true" />
> > <field name="id" type="string" indexed="true" stored="true" />
> > <field name="description" type="text" indexed="true" stored="true" />
> > <field name="created" type="date" indexed="true" stored="true" />
> > <field name="updated" type="date" indexed="true" stored="true" />
> >
> > data-config.xml
> > ################
> > <dataConfig>
> > <!--
> jdbc:JSQLConnect://se-eriearle-lt1/database=SocialSite2/user=SocialSite2/logfile=DB_TRACE.log
> -->
> > <dataSource type="JdbcDataSource"
> > driver="com.jnetdirect.jsql.JSQLDriver"
> >
> url="jdbc:JSQLConnect://se-eriearle-lt1/database=SocialSite2/user=SocialSite2"
> > user="SocialSite2"
> > password="SocialSite2" />
> >
> > <document>
> >
> > ....
> >
> > <entity name="Group" pk="key"
> > query="select 'group.'+id as 'key', 'group' as 'class', name,
> handle, description, created, updated from group order by created asc">
> > </entity>
> >
> > <entity name="Message" pk="key"
> > query="<...redacted...>">
> > </entity>
> >
> > </document>
> > </dataConfig>
> >
> >
> >
> >
>
>
>
> --
> -----------------------------------------------------
> Noble Paul | Principal Engineer| AOL | http://aol.com
>
--
Lance Norskog
goksron@gmail.com
Re: DataImportHandler ignoring most rows
Posted by Noble Paul നോബിള് नोब्ळ् <no...@corp.aol.com>.
this comment says that
<str name="Total Rows Fetched">7</str>
the query fetched only 7 rows. If possible open a tool and just run
the same query and see how many rows are returned
On Wed, Aug 19, 2009 at 3:46 AM, Erik Earle<er...@yahoo.com> wrote:
> Using:
> - apache-solr-1.3.0
> - java 1.6
> - tomcat 6
> - sql server 2005 w/ JSQLConnect 4.0 driver
>
> I have a group table with 3007 rows. I have confirmed the key is
> unique with "select distinct id from group" and it returns 3007. When i re-index using http://host:port/solr/dataimport?command=full-import I only get 7 records indexed. Any insight into what is going on would be really great.
>
> A partial response:
> <lst name="statusMessages">
> <str name="Total Requests made to DataSource">1</str>
> <str name="Total Rows Fetched">7</str>
> <str name="Total Documents Skipped">0</str>
>
>
> I have other entities that index all the rows without issue.
>
> There are no errors in the logs.
>
> I am not using any Transformers (and most of my config is not changed from install)
>
> My schema.xml contains:
>
> <uniqueKey>key</uniqueKey>
>
> and field defs (not a full list of fields):
> <field name="key" type="string" indexed="true" stored="true" required="true" />
> <field name="class" type="string" indexed="true" stored="true" required="true" />
> <field name="id" type="string" indexed="true" stored="true" />
> <field name="description" type="text" indexed="true" stored="true" />
> <field name="created" type="date" indexed="true" stored="true" />
> <field name="updated" type="date" indexed="true" stored="true" />
>
> data-config.xml
> ################
> <dataConfig>
> <!-- jdbc:JSQLConnect://se-eriearle-lt1/database=SocialSite2/user=SocialSite2/logfile=DB_TRACE.log -->
> <dataSource type="JdbcDataSource"
> driver="com.jnetdirect.jsql.JSQLDriver"
> url="jdbc:JSQLConnect://se-eriearle-lt1/database=SocialSite2/user=SocialSite2"
> user="SocialSite2"
> password="SocialSite2" />
>
> <document>
>
> ....
>
> <entity name="Group" pk="key"
> query="select 'group.'+id as 'key', 'group' as 'class', name, handle, description, created, updated from group order by created asc">
> </entity>
>
> <entity name="Message" pk="key"
> query="<...redacted...>">
> </entity>
>
> </document>
> </dataConfig>
>
>
>
>
--
-----------------------------------------------------
Noble Paul | Principal Engineer| AOL | http://aol.com