You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by "K, Baraneetharan" <ba...@hp.com> on 2012/02/16 12:53:42 UTC

How to loop through the DataImportHandler query results?

Hi Solr community,

I'm new to Solr and DataImportHandler., I've a requirement to fetch the data from a database table and pass it to solr.

Part of existing data-config.xml and solr schema.xml are given below,

data-config.xml

<dataConfig>
    <dataSource driver="com.mysql.jdbc.Driver" url="jdbc:mysql://demo22122011.com" user="XXXX" password="1234" />
    <document name="sample">
        <entity name="adaptation" pk=" sample _id"
          query="Select * from adap"
         transformer="TemplateTransformer,DateFormatTransformer" >

         <field column="field_mrmid_value" name="mrm_id_camp_s_t" />
         <field column="field_ sample _scope_value" name=" sample _scope_camp_s_s" />
         <field column="field_quarterly_plan_value" name="quarterly_plan_camp_s_s" />
         <field column="field_business_unit_value" name="field_business_unit_value_camp_s_t" />
         <field column="field_sub_business_value" name="field_sub_business_value_camp_s_t" />
         <field column="field_cdescription_value" name=" sample _description_camp_s_t" />
        <field column="field_ sample _owner_value" name=" sample _owner_camp_s_s" />

       </entity>
    </document>
</dataConfig>


Schema.xml

<schema name="example" version="1.2">
<fields>
   <dynamicField name="*_camp_m_i"  type="int"    indexed="true"  stored="true" multiValued="true"/>
   <dynamicField name="*_camp_s_i"  type="int"    indexed="true"  stored="true" multiValued="false"/>
   <dynamicField name="*_camp_m_t"  type="text"    indexed="true"  stored="true" multiValued="true"/>
   <dynamicField name="*_camp_s_t"  type="text"    indexed="true"  stored="true" multiValued="false"/>
   <dynamicField name="*_camp_m_s"  type="string"  indexed="true"  stored="true" multiValued="true"/>
   <dynamicField name="*_camp_s_s"  type="string"  indexed="true"  stored="true" multiValued="false"/>
   <dynamicField name="*_camp_m_l"  type="long"  indexed="true"  stored="true" multiValued="true"/>
   <dynamicField name="*_camp_s_l"  type="long"  indexed="true"  stored="true" multiValued="false"/>
</fields>
</schema>


The table used in the query (adap) is often modified, number of columns in this table are changing frequently. Hence we are supposed to change the data-config.xml whenever a field is added or deleted.
To avoid that we don't want to mention the column names in the field tag , but want to write a query to map all the fields in the table with solr fileds even if we don't know, how many columns are there in the table.  I need a kind of loop which runs through all the query results and map that with solr fileds.

Please help me.

Regards,
Baranee

Re: How to loop through the DataImportHandler query results?

Posted by Mikhail Khludnev <mk...@griddynamics.com>.
Chantal,

if you prefer java here is http://wiki.apache.org/solr/DIHCustomTransform



On Thu, Feb 16, 2012 at 7:24 PM, Chantal Ackermann <
chantal.ackermann@btelligent.de> wrote:

> If your script turns out too complex to maintain, and you are developing
> in Java, anyway, you could extend EntityProcessor and handle the data in
> a custom way. I've done that to transform a datamart like data structure
> back into a row based one.
>
> Basically you override the method that gets the data in a Map and
> transform it into a different Map which contains the fields as
> understood by your schema.
>
> Chantal
>
>
> On Thu, 2012-02-16 at 14:59 +0100, Mikhail Khludnev wrote:
> > Hi Baranee,
> >
> > Some time ago I played with
> > http://wiki.apache.org/solr/DataImportHandler#ScriptTransformer - it
> was a
> > pretty good stuff.
> >
> > Regards
> >
> >
> > On Thu, Feb 16, 2012 at 3:53 PM, K, Baraneetharan <
> baraneetharan.k@hp.com>wrote:
> >
> > > To avoid that we don't want to mention the column names in the field
> tag ,
> > > but want to write a query to map all the fields in the table with solr
> > > fileds even if we don't know, how many columns are there in the table.
>  I
> > > need a kind of loop which runs through all the query results and map
> that
> > > with solr fileds.
> >
> >
> >
> >
>
>


-- 
Sincerely yours
Mikhail Khludnev
Lucid Certified
Apache Lucene/Solr Developer
Grid Dynamics

<http://www.griddynamics.com>
 <mk...@griddynamics.com>

Re: How to loop through the DataImportHandler query results?

Posted by Chantal Ackermann <ch...@btelligent.de>.
If your script turns out too complex to maintain, and you are developing
in Java, anyway, you could extend EntityProcessor and handle the data in
a custom way. I've done that to transform a datamart like data structure
back into a row based one.

Basically you override the method that gets the data in a Map and
transform it into a different Map which contains the fields as
understood by your schema.

Chantal


On Thu, 2012-02-16 at 14:59 +0100, Mikhail Khludnev wrote:
> Hi Baranee,
> 
> Some time ago I played with
> http://wiki.apache.org/solr/DataImportHandler#ScriptTransformer - it was a
> pretty good stuff.
> 
> Regards
> 
> 
> On Thu, Feb 16, 2012 at 3:53 PM, K, Baraneetharan <ba...@hp.com>wrote:
> 
> > To avoid that we don't want to mention the column names in the field tag ,
> > but want to write a query to map all the fields in the table with solr
> > fileds even if we don't know, how many columns are there in the table.  I
> > need a kind of loop which runs through all the query results and map that
> > with solr fileds.
> 
> 
> 
>

Re: How to loop through the DataImportHandler query results?

Posted by Mikhail Khludnev <mk...@griddynamics.com>.
Hi Baranee,

Some time ago I played with
http://wiki.apache.org/solr/DataImportHandler#ScriptTransformer - it was a
pretty good stuff.

Regards


On Thu, Feb 16, 2012 at 3:53 PM, K, Baraneetharan <ba...@hp.com>wrote:

> To avoid that we don't want to mention the column names in the field tag ,
> but want to write a query to map all the fields in the table with solr
> fileds even if we don't know, how many columns are there in the table.  I
> need a kind of loop which runs through all the query results and map that
> with solr fileds.




-- 
Sincerely yours
Mikhail Khludnev
Lucid Certified
Apache Lucene/Solr Developer
Grid Dynamics

<http://www.griddynamics.com>
 <mk...@griddynamics.com>