You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by harrysmith <ha...@gmail.com> on 2010/09/30 04:01:48 UTC
DataImportHandler dynamic fields clarification
Looking for some clarification on DIH to make sure I am interpreting this
correctly.
I have a wide DB table, 100 columns. I'd rather not have to add 100 values
in schema.xml and data-config.xml. I was under the impression that if the
column name matched a dynamic Field name, it would be added. I am not
finding this is the case, but only works when the column name is explicitly
listed as a static field.
Example: 100 column table, columns named 'COLUMN_1, COLUMN_2 ... COLUMN_100'
If I add something like:
<field name="column_60" type="string" indexed="true" stored="true"/>
to schema.xml, and don't reference the column in data-config entity/field
tag, it gets imported, as expected.
However, if I use:
<dynamicField name="column_*" type="string" indexed="true"
stored="true"/>
It does not get imported into Solr, I would expect it would.
Is this the expected behavior?
--
View this message in context: http://lucene.472066.n3.nabble.com/DataImportHandler-dynamic-fields-clarification-tp1606159p1606159.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: DataImportHandler dynamic fields clarification
Posted by Alexey Serba <as...@gmail.com>.
Harry, could you please file a jira for this and I'll address this in
a patch. I fixed related issue (SOLR-2102) and I think it's pretty
similar.
> Interesting, I was under the impression that case does not matter.
>
> From http://wiki.apache.org/solr/DataImportHandler#A_shorter_data-config :
> "It is possible to totally avoid the field entries in entities if the names
> of the fields are same (case does not matter) as those in Solr schema"
>
Yeah, case does not matter only for explicit mapping of sql columns to
Solr fields. The reason is that DIH populates hash map for case
insensitive match only for explicit mappings.
You can also workaround this upper case column names in Oracle using
the following SQL clause:
=========================
data-config.xml
<entity name="item" query="select column_1 as "e;column_1"e;,
column_100 as "e;column_100"e; from wide_table">
</entity>
schema.xml
<dynamicField name="column_*" type="string" indexed="true" stored="true"
multiValued="true" />
=========================
HTH,
Alexey
On Thu, Sep 30, 2010 at 9:10 PM, harrysmith <ha...@gmail.com> wrote:
>
>>
>>Two things, one are your DB column uppercase as this would effect the out.
>>
>>
>
> Interesting, I was under the impression that case does not matter.
>
> From http://wiki.apache.org/solr/DataImportHandler#A_shorter_data-config :
> "It is possible to totally avoid the field entries in entities if the names
> of the fields are same (case does not matter) as those in Solr schema"
>
> I confirmed that matching the schema.xml field case to the database table is
> needed for dynamic fields, and the wiki statement above is incorrect, or at
> the very least confusing, possibly a bug.
>
> My database is Oracle 10g and the column names have been created in all
> uppercase in the database.
>
> In Oracle:
> Table name: wide_table
> Column names: COLUMN_1 ... COLUMN_100 (yes, uppercase)
>
> Please see following scenarios and results I found:
>
> data-config.xml
> <entity name="item" query="select column_1,column_100 from wide_table">
> <field column="column_100" name="id"/>
> </entity>
>
> schema.xml
> <dynamicField name="column_*" type="string" indexed="true" stored="true"
> multiValued="true" />
>
> Result:
> Nothing Imported
>
> =========
>
> data-config.xml
> <entity name="item" query="select COLUMN_1,COLUMN_100 from wide_table">
> <field column="column_100" name="id"/>
> </entity>
>
> schema.xml
> <dynamicField name="column_*" type="string" indexed="true" stored="true"
> multiValued="true" />
>
> Result:
> Note query column names changed to uppercase.
> Nothing Imported
>
> =========
>
>
> data-config.xml
> <entity name="item" query="select column_1,column_100 from wide_table">
> <field column="COLUMN_100" name="id"/>
> </entity>
>
> schema.xml
> <dynamicField name="column_*" type="string" indexed="true" stored="true"
> multiValued="true" />
>
> Result:
> Note ONLY the field entry was changed to caps
>
> All records imported, with only COLUMN_100 id field.
>
> ============
>
> data-config.xml
> <entity name="item" query="select column_1,column_100 from wide_table">
> <field column="COLUMN_100" name="id"/>
> </entity>
>
> schema.xml
> <dynamicField name="COLUMN_*" type="string" indexed="true" stored="true"
> multiValued="true" />
>
> Result:
> Note BOTH the field entry was changed to caps in data-config.xml, and the
> dynamicField wildcard in schema.xml
>
> All records imported, with all fields specified. This is the behavior
> desired.
>
> =============
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>>
>>Second what does your db-data-config.xml look like
>>
>>
>
> The relevant data-config.xml is as follows:
>
> <document name="">
> <entity name="item" query="select COLUMN_1,COLUMN_100 from wide_table">
> <field column="COLUMN_100" name="id"/>
> </entity>
> </document>
>
> Ideally, I would rather have the query be 'select * from wide_table" with
> the fields being dynamically matched by the column name from the
> dynamicField wildcard from the schema.xml.
>
> <dynamicField name="COLUMN_*" type="string" indexed="true" stored="true"/>
>
>
> --
> View this message in context: http://lucene.472066.n3.nabble.com/DataImportHandler-dynamic-fields-clarification-tp1606159p1609578.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
Re: DataImportHandler dynamic fields clarification
Posted by harrysmith <ha...@gmail.com>.
>
>Two things, one are your DB column uppercase as this would effect the out.
>
>
Interesting, I was under the impression that case does not matter.
>From http://wiki.apache.org/solr/DataImportHandler#A_shorter_data-config :
"It is possible to totally avoid the field entries in entities if the names
of the fields are same (case does not matter) as those in Solr schema"
I confirmed that matching the schema.xml field case to the database table is
needed for dynamic fields, and the wiki statement above is incorrect, or at
the very least confusing, possibly a bug.
My database is Oracle 10g and the column names have been created in all
uppercase in the database.
In Oracle:
Table name: wide_table
Column names: COLUMN_1 ... COLUMN_100 (yes, uppercase)
Please see following scenarios and results I found:
data-config.xml
<entity name="item" query="select column_1,column_100 from wide_table">
<field column="column_100" name="id"/>
</entity>
schema.xml
<dynamicField name="column_*" type="string" indexed="true" stored="true"
multiValued="true" />
Result:
Nothing Imported
=========
data-config.xml
<entity name="item" query="select COLUMN_1,COLUMN_100 from wide_table">
<field column="column_100" name="id"/>
</entity>
schema.xml
<dynamicField name="column_*" type="string" indexed="true" stored="true"
multiValued="true" />
Result:
Note query column names changed to uppercase.
Nothing Imported
=========
data-config.xml
<entity name="item" query="select column_1,column_100 from wide_table">
<field column="COLUMN_100" name="id"/>
</entity>
schema.xml
<dynamicField name="column_*" type="string" indexed="true" stored="true"
multiValued="true" />
Result:
Note ONLY the field entry was changed to caps
All records imported, with only COLUMN_100 id field.
============
data-config.xml
<entity name="item" query="select column_1,column_100 from wide_table">
<field column="COLUMN_100" name="id"/>
</entity>
schema.xml
<dynamicField name="COLUMN_*" type="string" indexed="true" stored="true"
multiValued="true" />
Result:
Note BOTH the field entry was changed to caps in data-config.xml, and the
dynamicField wildcard in schema.xml
All records imported, with all fields specified. This is the behavior
desired.
=============
>
>Second what does your db-data-config.xml look like
>
>
The relevant data-config.xml is as follows:
<document name="">
<entity name="item" query="select COLUMN_1,COLUMN_100 from wide_table">
<field column="COLUMN_100" name="id"/>
</entity>
</document>
Ideally, I would rather have the query be 'select * from wide_table" with
the fields being dynamically matched by the column name from the
dynamicField wildcard from the schema.xml.
<dynamicField name="COLUMN_*" type="string" indexed="true" stored="true"/>
--
View this message in context: http://lucene.472066.n3.nabble.com/DataImportHandler-dynamic-fields-clarification-tp1606159p1609578.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Solr Cluster Indexing data question
Posted by Chris Hostetter <ho...@fucit.org>.
: Subject: Solr Cluster Indexing data question
: References: <12...@n3.nabble.com>
: <1E...@progressivealliance.co.uk>
http://people.apache.org/~hossman/#threadhijack
Thread Hijacking on Mailing Lists
When starting a new discussion on a mailing list, please do not reply to
an existing message, instead start a fresh email. Even if you change the
subject line of your email, other mail headers still track which thread
you replied to and your question is "hidden" in that thread and gets less
attention. It makes following discussions in the mailing list archives
particularly difficult.
See Also: http://en.wikipedia.org/wiki/User:DonDiego/Thread_hijacking
-Hoss
--
http://lucenerevolution.org/ ... October 7-8, Boston
http://bit.ly/stump-hoss ... Stump The Chump!
RE: Solr Cluster Indexing data question
Posted by "ZAROGKIKAS,GIORGOS" <g....@multirama.gr>.
Thank you very much Jak
I 'll take a look
-----Original Message-----
From: Jak Akdemir [mailto:jakdemir@gmail.com]
Sent: Thursday, September 30, 2010 5:58 PM
To: solr-user@lucene.apache.org
Subject: Re: Solr Cluster Indexing data question
If you want to use both of your nodes for building index (which means
two master), it makes them unified and collapses master slave
relation.
Would you take a look the link below for index snapshot problem?
http://wiki.apache.org/solr/SolrCollectionDistributionScripts
On Thu, Sep 30, 2010 at 11:03 AM, ZAROGKIKAS,GIORGOS
<g....@multirama.gr> wrote:
> Hi there solr experts
>
>
> I have an solr cluster with two nodes and separate index files for each
> node
>
> Node1 is master
> Node2 is slave
>
>
> Node1 is the one that I index my data and replicate them to Node2
>
> How can I index my data at both nodes simultaneously ?????
> Is there any specific setup ????
>
>
> The problem is when my Node1 is down and I index the data from Node2 ,
> Solr creates backup index folders like this "index.20100929060410"
> and reduce the space of my hard disk
>
> Thanks in advance
>
>
>
>
>
>
>
RE: Solr Cluster Indexing data question
Posted by "ZAROGKIKAS,GIORGOS" <g....@multirama.gr>.
Thanks Steve
I'll check it out
-----Original Message-----
From: Steve Cohen [mailto:mail4steve@gmail.com]
Sent: Thursday, September 30, 2010 9:30 PM
To: solr-user@lucene.apache.org
Subject: Re: Solr Cluster Indexing data question
So how would one set it up to use multiple nodes for building an index?
I
see a document for solr + hadoop
(http://wiki.apache.org/solr/HadoopIndexing)
and it says it has an example but the example is missing.
Thanks,
Steve Cohen
On Thu, Sep 30, 2010 at 10:58 AM, Jak Akdemir <ja...@gmail.com>
wrote:
> If you want to use both of your nodes for building index (which means
> two master), it makes them unified and collapses master slave
> relation.
>
> Would you take a look the link below for index snapshot problem?
> http://wiki.apache.org/solr/SolrCollectionDistributionScripts
>
> On Thu, Sep 30, 2010 at 11:03 AM, ZAROGKIKAS,GIORGOS
> <g....@multirama.gr> wrote:
> > Hi there solr experts
> >
> >
> > I have an solr cluster with two nodes and separate index files for
each
> > node
> >
> > Node1 is master
> > Node2 is slave
> >
> >
> > Node1 is the one that I index my data and replicate them to Node2
> >
> > How can I index my data at both nodes simultaneously ?????
> > Is there any specific setup ????
> >
> >
> > The problem is when my Node1 is down and I index the data from Node2
,
> > Solr creates backup index folders like this "index.20100929060410"
> > and reduce the space of my hard disk
> >
> > Thanks in advance
> >
> >
> >
> >
> >
> >
> >
>
Re: Solr Cluster Indexing data question
Posted by Steve Cohen <ma...@gmail.com>.
So how would one set it up to use multiple nodes for building an index? I
see a document for solr + hadoop (http://wiki.apache.org/solr/HadoopIndexing)
and it says it has an example but the example is missing.
Thanks,
Steve Cohen
On Thu, Sep 30, 2010 at 10:58 AM, Jak Akdemir <ja...@gmail.com> wrote:
> If you want to use both of your nodes for building index (which means
> two master), it makes them unified and collapses master slave
> relation.
>
> Would you take a look the link below for index snapshot problem?
> http://wiki.apache.org/solr/SolrCollectionDistributionScripts
>
> On Thu, Sep 30, 2010 at 11:03 AM, ZAROGKIKAS,GIORGOS
> <g....@multirama.gr> wrote:
> > Hi there solr experts
> >
> >
> > I have an solr cluster with two nodes and separate index files for each
> > node
> >
> > Node1 is master
> > Node2 is slave
> >
> >
> > Node1 is the one that I index my data and replicate them to Node2
> >
> > How can I index my data at both nodes simultaneously ?????
> > Is there any specific setup ????
> >
> >
> > The problem is when my Node1 is down and I index the data from Node2 ,
> > Solr creates backup index folders like this "index.20100929060410"
> > and reduce the space of my hard disk
> >
> > Thanks in advance
> >
> >
> >
> >
> >
> >
> >
>
Re: Solr Cluster Indexing data question
Posted by Jak Akdemir <ja...@gmail.com>.
If you want to use both of your nodes for building index (which means
two master), it makes them unified and collapses master slave
relation.
Would you take a look the link below for index snapshot problem?
http://wiki.apache.org/solr/SolrCollectionDistributionScripts
On Thu, Sep 30, 2010 at 11:03 AM, ZAROGKIKAS,GIORGOS
<g....@multirama.gr> wrote:
> Hi there solr experts
>
>
> I have an solr cluster with two nodes and separate index files for each
> node
>
> Node1 is master
> Node2 is slave
>
>
> Node1 is the one that I index my data and replicate them to Node2
>
> How can I index my data at both nodes simultaneously ?????
> Is there any specific setup ????
>
>
> The problem is when my Node1 is down and I index the data from Node2 ,
> Solr creates backup index folders like this "index.20100929060410"
> and reduce the space of my hard disk
>
> Thanks in advance
>
>
>
>
>
>
>
Solr Cluster Indexing data question
Posted by "ZAROGKIKAS,GIORGOS" <g....@multirama.gr>.
Hi there solr experts
I have an solr cluster with two nodes and separate index files for each
node
Node1 is master
Node2 is slave
Node1 is the one that I index my data and replicate them to Node2
How can I index my data at both nodes simultaneously ?????
Is there any specific setup ????
The problem is when my Node1 is down and I index the data from Node2 ,
Solr creates backup index folders like this "index.20100929060410"
and reduce the space of my hard disk
Thanks in advance
Re: DataImportHandler dynamic fields clarification
Posted by David Stuart <da...@progressivealliance.co.uk>.
Two things, one are your DB column uppercase as this would effect the out.
Second what does your db-data-config.xml look like
Regards,
Dave
On 30 Sep 2010, at 03:01, harrysmith wrote:
>
> Looking for some clarification on DIH to make sure I am interpreting this
> correctly.
>
> I have a wide DB table, 100 columns. I'd rather not have to add 100 values
> in schema.xml and data-config.xml. I was under the impression that if the
> column name matched a dynamic Field name, it would be added. I am not
> finding this is the case, but only works when the column name is explicitly
> listed as a static field.
>
> Example: 100 column table, columns named 'COLUMN_1, COLUMN_2 ... COLUMN_100'
>
> If I add something like:
> <field name="column_60" type="string" indexed="true" stored="true"/>
> to schema.xml, and don't reference the column in data-config entity/field
> tag, it gets imported, as expected.
>
> However, if I use:
> <dynamicField name="column_*" type="string" indexed="true"
> stored="true"/>
> It does not get imported into Solr, I would expect it would.
>
>
> Is this the expected behavior?
> --
> View this message in context: http://lucene.472066.n3.nabble.com/DataImportHandler-dynamic-fields-clarification-tp1606159p1606159.html
> Sent from the Solr - User mailing list archive at Nabble.com.