You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Jay Fisher <ja...@gmail.com> on 2009/12/27 06:55:15 UTC
Problem with simple use of DIH
I'm trying to use DataImportHandler to load my index and having some strange
results. I have two tables in my database. DPRODUC contains products and
FSKUMAS contains the skus related to each product.
This is the data-config I'm using.
<dataConfig>
<dataSource type="JdbcDataSource"
driver="com.ibm.as400.access.AS400JDBCDriver"
url="jdbc:as400:IWAVE;prompt=false;naming=system"
user="IPGUI"
password="IPGUI"/>
<document>
<entity name="dproduc"
query="select dprprd, dprdes from dproduc where dprprd like 'F%'">
<field column="dprprd" name="id" />
<field column="dprdes" name="name" />
<entity name="fskumas"
query="select fsksku, fcoclr, fszsiz, fskret
from fskumas where dprprd='${dproduc.DPRPRD}'">
<field column="fsksku" name="sku" />
<field column="fcoclr" name="color" />
<field column="fszsiz" name="size" />
<field column="fskret" name="price" />
</entity>
</entity>
</document>
</dataConfig>
and this is the relevant part of my schema.xml
<field name="id" type="string" indexed="true" stored="true"
required="true" />
<field name="name" type="textgen" indexed="true" stored="true"/>
<field name="sku" multiValued="true" type="textTight" indexed="true"
stored="true" omitNorms="true"/>
<field name="color" multiValued="true" type="textgen" indexed="true"
stored="true"/>
<field name="size" multiValued="true" type="textgen" indexed="true"
stored="true"/>
<field name="price" multiValued="true" type="float" indexed="true"
stored="true"/>
The queries appear to be executing correctly. Here is what the DIH console
shows when I debug it for the first two products. But notice that the nested
query on fskumas shows up twice on the second product. That continues the
more products I include; 3 for the 3rd product, 4 for the fourth, etc. And
also notice that zero documents end up being processed.
Can anyone spot what I'm doing wrong?
<?xml version="1.0" encoding="UTF-8"?>
<response>
<lst name="responseHeader">
<int name="status">0</int>
<int name="QTime">656</int>
</lst>
<lst name="initArgs">
<lst name="defaults">
<str name="config">data-config.xml</str>
</lst>
</lst>
<str name="command">full-import</str>
<str name="mode">debug</str>
<null name="documents" />
<lst name="verbose-output">
<lst name="entity:dproduc">
<lst name="document#1">
<str name="query">select dprprd, dprdes from dproduc where dprprd like
'F%'</str>
<str name="time-taken">0:0:0.313</str>
<str>----------- row #1-------------</str>
<str name="DPRDES">ADIRON/CEDAR/SEAT </str>
<str name="DPRPRD">F001 </str>
<str>---------------------------------------------</str>
<lst name="entity:fskumas">
<str name="query">select fsksku, fcoclr, fszsiz, fskret from fskumas
where dprprd='F001 '</str>
<str name="time-taken">0:0:0.313</str>
<str>----------- row #1-------------</str>
<str name="FSZSIZ" />
<str name="FSKSKU">java.math.BigDecimal:100032</str>
<str name="FSKRET">java.math.BigDecimal:0.01</str>
<str name="FCOCLR" />
<str>---------------------------------------------</str>
</lst>
</ lst>
<lst name="document#1">
<str>----------- row #1-------------</str>
<str name="DPRDES">ADIRON/CEDAR/BACK </str>
<str name="DPRPRD">F002 </str>
<str>---------------------------------------------</str>
<lst name="entity:fskumas">
<str name="query">select fsksku, fcoclr, fszsiz, fskret from fskumas
where dprprd='F002 '</str>
<str name="query">select fsksku, fcoclr, fszsiz, fskret from fskumas
where dprprd='F002 '</str>
<str name="time-taken">0:0:0.15</str>
<str name="time-taken">0:0:0.15</str>
<str>----------- row #1-------------</str>
<str name="FSZSIZ" />
<str name="FSKSKU">java.math.BigDecimal:100040</str>
<str name="FSKRET">java.math.BigDecimal:0.01</str>
<str name="FCOCLR" />
<str>---------------------------------------------</str>
</lst>
</ lst>
<lst name="document#1" />
</lst>
</lst>
<str name="status">idle</str>
<str name="importResponse">Configuration Re-loaded sucessfully</str>
<lst name="statusMessages">
<str name="Total Requests made to DataSource">3</str>
<str name="Total Rows Fetched">4</str>
<str name="Total Documents Skipped">0</str>
<str name="Full Dump Started">2009-12-26 21:46:45</str>
<str name="Total Documents Processed">0</str>
<str name="Time taken ">0:0:0.656</str>
</lst>
<str name="WARNING">This response format is experimental. It is likely to
change in the future.</str>
</lst>
</lst>
</response>
Re: Problem with simple use of DIH
Posted by Noble Paul നോബിള് नोब्ळ् <no...@corp.aol.com>.
The field names are case sensitive. But if the <field> tags are
missing they are mapped to corresponding solr fields in a case
insensistive way.apparently all the fields come out of you ALL CAPS
you should put the 'column' values in ALL CAPS too
On Sun, Dec 27, 2009 at 9:03 PM, Jay Fisher <ja...@gmail.com> wrote:
> I did run it without debug and the result was that 0 documents were
> processed.
>
> The problem seems to be with the <field> tags that I was using to map from
> the table column names to the schema.xml field names. I switched to using an
> AS clause in the SQL statement instead and it worked.
>
> I think the column names may be case-sensitive, although I haven't proven
> that to be the case. I did discover that references to column names in the
> velocity template are case sensitive; ${dproduc.DPRPRD} works
> and ${dproduc.dprprd} does not.
>
> Thanks, Jay
>
> 2009/12/27 Noble Paul നോബിള് नोब्ळ् <no...@corp.aol.com>
>
>> did you run it w/o the debug?
>>
>> On Sun, Dec 27, 2009 at 6:31 PM, AHMET ARSLAN <io...@yahoo.com> wrote:
>> >> I'm trying to use DataImportHandler
>> >> to load my index and having some strange
>> >> results. I have two tables in my database. DPRODUC contains
>> >> products and
>> >> FSKUMAS contains the skus related to each product.
>> >>
>> >> This is the data-config I'm using.
>> >>
>> >> <dataConfig>
>> >> <dataSource type="JdbcDataSource"
>> >>
>> >> driver="com.ibm.as400.access.AS400JDBCDriver"
>> >>
>> >> url="jdbc:as400:IWAVE;prompt=false;naming=system"
>> >>
>> >> user="IPGUI"
>> >>
>> >> password="IPGUI"/>
>> >> <document>
>> >> <entity name="dproduc"
>> >> query="select dprprd, dprdes from
>> >> dproduc where dprprd like 'F%'">
>> >> <field column="dprprd" name="id"
>> >> />
>> >> <field column="dprdes" name="name"
>> >> />
>> >> <entity name="fskumas"
>> >> query="select fsksku, fcoclr,
>> >> fszsiz, fskret
>> >> from fskumas where
>> >> dprprd='${dproduc.DPRPRD}'">
>> >> <field
>> >> column="fsksku" name="sku" />
>> >> <field
>> >> column="fcoclr" name="color" />
>> >> <field
>> >> column="fszsiz" name="size" />
>> >> <field
>> >> column="fskret" name="price" />
>> >> </entity>
>> >> </entity>
>> >> </document>
>> >> </dataConfig>
>> >
>> > What is the primary key of dproduc table? If it is dprprd can you try
>> adding pk="dprprd" to <entity name="dproduc"?
>> >
>> > <entity name="dproduc" pk="dprprd" query="select dprprd, dprdes from
>> dproduc where dprprd like 'F%'">
>> >
>> >
>> >
>> >
>>
>>
>>
>> --
>> -----------------------------------------------------
>> Noble Paul | Systems Architect| AOL | http://aol.com
>>
>
--
-----------------------------------------------------
Noble Paul | Systems Architect| AOL | http://aol.com
Re: Problem with simple use of DIH
Posted by Jay Fisher <ja...@gmail.com>.
I did run it without debug and the result was that 0 documents were
processed.
The problem seems to be with the <field> tags that I was using to map from
the table column names to the schema.xml field names. I switched to using an
AS clause in the SQL statement instead and it worked.
I think the column names may be case-sensitive, although I haven't proven
that to be the case. I did discover that references to column names in the
velocity template are case sensitive; ${dproduc.DPRPRD} works
and ${dproduc.dprprd} does not.
Thanks, Jay
2009/12/27 Noble Paul നോബിള് नोब्ळ् <no...@corp.aol.com>
> did you run it w/o the debug?
>
> On Sun, Dec 27, 2009 at 6:31 PM, AHMET ARSLAN <io...@yahoo.com> wrote:
> >> I'm trying to use DataImportHandler
> >> to load my index and having some strange
> >> results. I have two tables in my database. DPRODUC contains
> >> products and
> >> FSKUMAS contains the skus related to each product.
> >>
> >> This is the data-config I'm using.
> >>
> >> <dataConfig>
> >> <dataSource type="JdbcDataSource"
> >>
> >> driver="com.ibm.as400.access.AS400JDBCDriver"
> >>
> >> url="jdbc:as400:IWAVE;prompt=false;naming=system"
> >>
> >> user="IPGUI"
> >>
> >> password="IPGUI"/>
> >> <document>
> >> <entity name="dproduc"
> >> query="select dprprd, dprdes from
> >> dproduc where dprprd like 'F%'">
> >> <field column="dprprd" name="id"
> >> />
> >> <field column="dprdes" name="name"
> >> />
> >> <entity name="fskumas"
> >> query="select fsksku, fcoclr,
> >> fszsiz, fskret
> >> from fskumas where
> >> dprprd='${dproduc.DPRPRD}'">
> >> <field
> >> column="fsksku" name="sku" />
> >> <field
> >> column="fcoclr" name="color" />
> >> <field
> >> column="fszsiz" name="size" />
> >> <field
> >> column="fskret" name="price" />
> >> </entity>
> >> </entity>
> >> </document>
> >> </dataConfig>
> >
> > What is the primary key of dproduc table? If it is dprprd can you try
> adding pk="dprprd" to <entity name="dproduc"?
> >
> > <entity name="dproduc" pk="dprprd" query="select dprprd, dprdes from
> dproduc where dprprd like 'F%'">
> >
> >
> >
> >
>
>
>
> --
> -----------------------------------------------------
> Noble Paul | Systems Architect| AOL | http://aol.com
>
Re: Problem with simple use of DIH
Posted by Noble Paul നോബിള് नोब्ळ् <no...@corp.aol.com>.
did you run it w/o the debug?
On Sun, Dec 27, 2009 at 6:31 PM, AHMET ARSLAN <io...@yahoo.com> wrote:
>> I'm trying to use DataImportHandler
>> to load my index and having some strange
>> results. I have two tables in my database. DPRODUC contains
>> products and
>> FSKUMAS contains the skus related to each product.
>>
>> This is the data-config I'm using.
>>
>> <dataConfig>
>> <dataSource type="JdbcDataSource"
>>
>> driver="com.ibm.as400.access.AS400JDBCDriver"
>>
>> url="jdbc:as400:IWAVE;prompt=false;naming=system"
>>
>> user="IPGUI"
>>
>> password="IPGUI"/>
>> <document>
>> <entity name="dproduc"
>> query="select dprprd, dprdes from
>> dproduc where dprprd like 'F%'">
>> <field column="dprprd" name="id"
>> />
>> <field column="dprdes" name="name"
>> />
>> <entity name="fskumas"
>> query="select fsksku, fcoclr,
>> fszsiz, fskret
>> from fskumas where
>> dprprd='${dproduc.DPRPRD}'">
>> <field
>> column="fsksku" name="sku" />
>> <field
>> column="fcoclr" name="color" />
>> <field
>> column="fszsiz" name="size" />
>> <field
>> column="fskret" name="price" />
>> </entity>
>> </entity>
>> </document>
>> </dataConfig>
>
> What is the primary key of dproduc table? If it is dprprd can you try adding pk="dprprd" to <entity name="dproduc"?
>
> <entity name="dproduc" pk="dprprd" query="select dprprd, dprdes from dproduc where dprprd like 'F%'">
>
>
>
>
--
-----------------------------------------------------
Noble Paul | Systems Architect| AOL | http://aol.com
Re: Problem with simple use of DIH
Posted by AHMET ARSLAN <io...@yahoo.com>.
> I'm trying to use DataImportHandler
> to load my index and having some strange
> results. I have two tables in my database. DPRODUC contains
> products and
> FSKUMAS contains the skus related to each product.
>
> This is the data-config I'm using.
>
> <dataConfig>
> <dataSource type="JdbcDataSource"
>
> driver="com.ibm.as400.access.AS400JDBCDriver"
>
> url="jdbc:as400:IWAVE;prompt=false;naming=system"
>
> user="IPGUI"
>
> password="IPGUI"/>
> <document>
> <entity name="dproduc"
> query="select dprprd, dprdes from
> dproduc where dprprd like 'F%'">
> <field column="dprprd" name="id"
> />
> <field column="dprdes" name="name"
> />
> <entity name="fskumas"
> query="select fsksku, fcoclr,
> fszsiz, fskret
> from fskumas where
> dprprd='${dproduc.DPRPRD}'">
> <field
> column="fsksku" name="sku" />
> <field
> column="fcoclr" name="color" />
> <field
> column="fszsiz" name="size" />
> <field
> column="fskret" name="price" />
> </entity>
> </entity>
> </document>
> </dataConfig>
What is the primary key of dproduc table? If it is dprprd can you try adding pk="dprprd" to <entity name="dproduc"?
<entity name="dproduc" pk="dprprd" query="select dprprd, dprdes from dproduc where dprprd like 'F%'">