index new discovered fileds of different types

[Thaer Sammar <t....@geophy.com>]

Hi,
We are trying to index documents of different types. Document have different fields. fields are known at indexing time. We run a query on a database and we index what comes using query variables as field names in solr. Our current solution: we use dynamic fields with prefix, for example feature_i_*, the issue with that
1) we need to define the type of the dynamic field and to be able to cover the type of discovered fields we define the following
feature_i_* for integers, feature_t_* for string, feature_d_* for double, ....
1.a) this means we need to check the type of the discovered field and then put in the corresponding dynamic field
2) at search time, we need to know the right prefix
We are looking for help to find away to ignore the prefix and check of the type

regards,
Thaer

Re: index new discovered fileds of different types

[Thaer Sammar <t....@geophy.com>]

Hi Rick,

yes the RDF structure has subject, predicate and object. The object data
type is not only text, it can be integer or double as well or other data
types. The structure of our solar document doesn't only contain these three
fields. We compose one document per subject and we use all found objects as
fields. Currently, in the schema we define two static fields uri (subject)
and geo filed which contain the geographic point. When we find a message in
the kafka queue, which means something change in the DB, we query DB to get
all subject,predicate,object of the found subjects, based on that we create
the document. For example, for subjects s1 and s2, we might get the
following from the DB

s1,geo,(latitude, longitude)
s1,are,200.0
s1,type,office
s2,geo,(latitude, longitude)

for s1, there are more information available and we like to include it in
the solr doc, therefore we used the dynamic filed
feature_double_*, and feature_text_*. based on the object data type we add
to appropriate dynamic field

<doc>
<uri>s1<uri>
<geo>(latitude,longitude)</geo>
<feature_double_area>200.0</feature_double_area>
<feature_text_type>office</feature_text_type>
</doc>
 we appended the predicate name with dynamic filed prefix, and we used pdf
data type to decide which dynamic filed to use

regards,
Thaer

On 8 July 2017 at 02:36, Rick Leir <rl...@leirtech.com> wrote:

> Thaer
> Whoa, hold everything! You said RDF, meaning resource description
> framework? If so, you have exactly​ three fields: subject, predicate, and
> object. Maybe they are text type, or for exact matches you might want
> string fields. Add an ID field, which could be automatically generated by
> Solr, so now you have four fields. Or am I on a tangent again? Cheers --
> Rick
>
> On July 7, 2017 6:01:00 AM EDT, Thaer Sammar <t....@geophy.com> wrote:
> >Hi Jan,
> >
> >Thanks!, I am exploring the schemaless option based on Furkan
> >suggestion. I
> >need the the flexibility because not all fields are known. We get the
> >data
> >from RDF database (which changes continuously). To be more specific, we
> >have a database and all changes on it are sent to a kafka queue. and we
> >have a consumer which listen to the queue and update the Solr index.
> >
> >regards,
> >Thaer
> >
> >On 7 July 2017 at 10:53, Jan Høydahl <ja...@cominvent.com> wrote:
> >
> >> If you do not need the flexibility of dynamic fields, don’t use them.
> >> Sounds to me that you really want a field “price” to be float and a
> >field
> >> “birthdate” to be of type date etc.
> >> If so, simply create your schema (either manually, through Schema API
> >or
> >> using schemaless) up front and index each field as correct type
> >without
> >> messing with field name prefixes.
> >>
> >> --
> >> Jan Høydahl, search solution architect
> >> Cominvent AS - www.cominvent.com
> >>
> >> > 5. jul. 2017 kl. 15.23 skrev Thaer Sammar <t....@geophy.com>:
> >> >
> >> > Hi,
> >> > We are trying to index documents of different types. Document have
> >> different fields. fields are known at indexing time. We run a query
> >on a
> >> database and we index what comes using query variables as field names
> >in
> >> solr. Our current solution: we use dynamic fields with prefix, for
> >example
> >> feature_i_*, the issue with that
> >> > 1) we need to define the type of the dynamic field and to be able
> >to
> >> cover the type of discovered fields we define the following
> >> > feature_i_* for integers, feature_t_* for string, feature_d_* for
> >> double, ....
> >> > 1.a) this means we need to check the type of the discovered field
> >and
> >> then put in the corresponding dynamic field
> >> > 2) at search time, we need to know the right prefix
> >> > We are looking for help to find away to ignore the prefix and check
> >of
> >> the type
> >> >
> >> > regards,
> >> > Thaer
> >>
> >>
>
> --
> Sorry for being brief. Alternate email is rickleir at yahoo dot com

Re: index new discovered fileds of different types

[Jan Høydahl <ja...@cominvent.com>]

I think Thaer’s answer clarify how they do it.
So at the time they assemble the full Solr doc to index, there may be a new field name not known in advance,
but to my understanding the RDF source contains information on the type (else they could not do the mapping
to dynamic field either) and so adding a field to the managed schema on the fly once an unknown field is detected
should work just fine!

--
Jan Høydahl, search solution architect
Cominvent AS - www.cominvent.com

> 10. jul. 2017 kl. 02.08 skrev Rick Leir <rl...@leirtech.com>:
> 
> Jan
> 
> I hope this is not off-topic, but I am curious: if you do not use the three fields, subject, predicate, and object for indexing RDF
> then what is your algorithm? Maybe document nesting is appropriate for this? cheers -- Rick
> 
> 
> On 2017-07-09 05:52 PM, Jan Høydahl wrote:
>> Hi,
>> 
>> I have personally written a Python script to parse RDF files into an in-memory graph structure and then pull data from that structure to index to Solr.
>> I.e. you may perfectly well have RDF (nt, turtle, whatever) as source but index sub structures in very specific ways.
>> Anyway, as Erick points out, that’s probably where in your code that you should use Managed Schema REST API in order to
>> 1. Query Solr for what fields are defined
>> 2. If you need to index a field that is not yet in Solr, add it, using the correct field type (your app should know)
>> 3. Push the data
>> 4. Repeat
>> 
>> --
>> Jan Høydahl, search solution architect
>> Cominvent AS - www.cominvent.com
>> 
>>> 8. jul. 2017 kl. 02.36 skrev Rick Leir <rl...@leirtech.com>:
>>> 
>>> Thaer
>>> Whoa, hold everything! You said RDF, meaning resource description framework? If so, you have exactly​ three fields: subject, predicate, and object. Maybe they are text type, or for exact matches you might want string fields. Add an ID field, which could be automatically generated by Solr, so now you have four fields. Or am I on a tangent again? Cheers -- Rick
>>> 
>>> On July 7, 2017 6:01:00 AM EDT, Thaer Sammar <t....@geophy.com> wrote:
>>>> Hi Jan,
>>>> 
>>>> Thanks!, I am exploring the schemaless option based on Furkan
>>>> suggestion. I
>>>> need the the flexibility because not all fields are known. We get the
>>>> data
>>>> from RDF database (which changes continuously). To be more specific, we
>>>> have a database and all changes on it are sent to a kafka queue. and we
>>>> have a consumer which listen to the queue and update the Solr index.
>>>> 
>>>> regards,
>>>> Thaer
>>>> 
>>>> On 7 July 2017 at 10:53, Jan Høydahl <ja...@cominvent.com> wrote:
>>>> 
>>>>> If you do not need the flexibility of dynamic fields, don’t use them.
>>>>> Sounds to me that you really want a field “price” to be float and a
>>>> field
>>>>> “birthdate” to be of type date etc.
>>>>> If so, simply create your schema (either manually, through Schema API
>>>> or
>>>>> using schemaless) up front and index each field as correct type
>>>> without
>>>>> messing with field name prefixes.
>>>>> 
>>>>> --
>>>>> Jan Høydahl, search solution architect
>>>>> Cominvent AS - www.cominvent.com
>>>>> 
>>>>>> 5. jul. 2017 kl. 15.23 skrev Thaer Sammar <t....@geophy.com>:
>>>>>> 
>>>>>> Hi,
>>>>>> We are trying to index documents of different types. Document have
>>>>> different fields. fields are known at indexing time. We run a query
>>>> on a
>>>>> database and we index what comes using query variables as field names
>>>> in
>>>>> solr. Our current solution: we use dynamic fields with prefix, for
>>>> example
>>>>> feature_i_*, the issue with that
>>>>>> 1) we need to define the type of the dynamic field and to be able
>>>> to
>>>>> cover the type of discovered fields we define the following
>>>>>> feature_i_* for integers, feature_t_* for string, feature_d_* for
>>>>> double, ....
>>>>>> 1.a) this means we need to check the type of the discovered field
>>>> and
>>>>> then put in the corresponding dynamic field
>>>>>> 2) at search time, we need to know the right prefix
>>>>>> We are looking for help to find away to ignore the prefix and check
>>>> of
>>>>> the type
>>>>>> regards,
>>>>>> Thaer
>>>>> 
>>> -- 
>>> Sorry for being brief. Alternate email is rickleir at yahoo dot com
> 

Re: index new discovered fileds of different types

[Rick Leir <rl...@leirtech.com>]

Jan

I hope this is not off-topic, but I am curious: if you do not use the 
three fields, subject, predicate, and object for indexing RDF
then what is your algorithm? Maybe document nesting is appropriate for 
this? cheers -- Rick


On 2017-07-09 05:52 PM, Jan Høydahl wrote:
> Hi,
>
> I have personally written a Python script to parse RDF files into an in-memory graph structure and then pull data from that structure to index to Solr.
> I.e. you may perfectly well have RDF (nt, turtle, whatever) as source but index sub structures in very specific ways.
> Anyway, as Erick points out, that’s probably where in your code that you should use Managed Schema REST API in order to
> 1. Query Solr for what fields are defined
> 2. If you need to index a field that is not yet in Solr, add it, using the correct field type (your app should know)
> 3. Push the data
> 4. Repeat
>
> --
> Jan Høydahl, search solution architect
> Cominvent AS - www.cominvent.com
>
>> 8. jul. 2017 kl. 02.36 skrev Rick Leir <rl...@leirtech.com>:
>>
>> Thaer
>> Whoa, hold everything! You said RDF, meaning resource description framework? If so, you have exactly​ three fields: subject, predicate, and object. Maybe they are text type, or for exact matches you might want string fields. Add an ID field, which could be automatically generated by Solr, so now you have four fields. Or am I on a tangent again? Cheers -- Rick
>>
>> On July 7, 2017 6:01:00 AM EDT, Thaer Sammar <t....@geophy.com> wrote:
>>> Hi Jan,
>>>
>>> Thanks!, I am exploring the schemaless option based on Furkan
>>> suggestion. I
>>> need the the flexibility because not all fields are known. We get the
>>> data
>>> from RDF database (which changes continuously). To be more specific, we
>>> have a database and all changes on it are sent to a kafka queue. and we
>>> have a consumer which listen to the queue and update the Solr index.
>>>
>>> regards,
>>> Thaer
>>>
>>> On 7 July 2017 at 10:53, Jan Høydahl <ja...@cominvent.com> wrote:
>>>
>>>> If you do not need the flexibility of dynamic fields, don’t use them.
>>>> Sounds to me that you really want a field “price” to be float and a
>>> field
>>>> “birthdate” to be of type date etc.
>>>> If so, simply create your schema (either manually, through Schema API
>>> or
>>>> using schemaless) up front and index each field as correct type
>>> without
>>>> messing with field name prefixes.
>>>>
>>>> --
>>>> Jan Høydahl, search solution architect
>>>> Cominvent AS - www.cominvent.com
>>>>
>>>>> 5. jul. 2017 kl. 15.23 skrev Thaer Sammar <t....@geophy.com>:
>>>>>
>>>>> Hi,
>>>>> We are trying to index documents of different types. Document have
>>>> different fields. fields are known at indexing time. We run a query
>>> on a
>>>> database and we index what comes using query variables as field names
>>> in
>>>> solr. Our current solution: we use dynamic fields with prefix, for
>>> example
>>>> feature_i_*, the issue with that
>>>>> 1) we need to define the type of the dynamic field and to be able
>>> to
>>>> cover the type of discovered fields we define the following
>>>>> feature_i_* for integers, feature_t_* for string, feature_d_* for
>>>> double, ....
>>>>> 1.a) this means we need to check the type of the discovered field
>>> and
>>>> then put in the corresponding dynamic field
>>>>> 2) at search time, we need to know the right prefix
>>>>> We are looking for help to find away to ignore the prefix and check
>>> of
>>>> the type
>>>>> regards,
>>>>> Thaer
>>>>
>> -- 
>> Sorry for being brief. Alternate email is rickleir at yahoo dot com

Re: index new discovered fileds of different types

[Jan Høydahl <ja...@cominvent.com>]

Hi,

I have personally written a Python script to parse RDF files into an in-memory graph structure and then pull data from that structure to index to Solr.
I.e. you may perfectly well have RDF (nt, turtle, whatever) as source but index sub structures in very specific ways.
Anyway, as Erick points out, that’s probably where in your code that you should use Managed Schema REST API in order to
1. Query Solr for what fields are defined
2. If you need to index a field that is not yet in Solr, add it, using the correct field type (your app should know)
3. Push the data
4. Repeat

--
Jan Høydahl, search solution architect
Cominvent AS - www.cominvent.com

> 8. jul. 2017 kl. 02.36 skrev Rick Leir <rl...@leirtech.com>:
> 
> Thaer
> Whoa, hold everything! You said RDF, meaning resource description framework? If so, you have exactly​ three fields: subject, predicate, and object. Maybe they are text type, or for exact matches you might want string fields. Add an ID field, which could be automatically generated by Solr, so now you have four fields. Or am I on a tangent again? Cheers -- Rick
> 
> On July 7, 2017 6:01:00 AM EDT, Thaer Sammar <t....@geophy.com> wrote:
>> Hi Jan,
>> 
>> Thanks!, I am exploring the schemaless option based on Furkan
>> suggestion. I
>> need the the flexibility because not all fields are known. We get the
>> data
>> from RDF database (which changes continuously). To be more specific, we
>> have a database and all changes on it are sent to a kafka queue. and we
>> have a consumer which listen to the queue and update the Solr index.
>> 
>> regards,
>> Thaer
>> 
>> On 7 July 2017 at 10:53, Jan Høydahl <ja...@cominvent.com> wrote:
>> 
>>> If you do not need the flexibility of dynamic fields, don’t use them.
>>> Sounds to me that you really want a field “price” to be float and a
>> field
>>> “birthdate” to be of type date etc.
>>> If so, simply create your schema (either manually, through Schema API
>> or
>>> using schemaless) up front and index each field as correct type
>> without
>>> messing with field name prefixes.
>>> 
>>> --
>>> Jan Høydahl, search solution architect
>>> Cominvent AS - www.cominvent.com
>>> 
>>>> 5. jul. 2017 kl. 15.23 skrev Thaer Sammar <t....@geophy.com>:
>>>> 
>>>> Hi,
>>>> We are trying to index documents of different types. Document have
>>> different fields. fields are known at indexing time. We run a query
>> on a
>>> database and we index what comes using query variables as field names
>> in
>>> solr. Our current solution: we use dynamic fields with prefix, for
>> example
>>> feature_i_*, the issue with that
>>>> 1) we need to define the type of the dynamic field and to be able
>> to
>>> cover the type of discovered fields we define the following
>>>> feature_i_* for integers, feature_t_* for string, feature_d_* for
>>> double, ....
>>>> 1.a) this means we need to check the type of the discovered field
>> and
>>> then put in the corresponding dynamic field
>>>> 2) at search time, we need to know the right prefix
>>>> We are looking for help to find away to ignore the prefix and check
>> of
>>> the type
>>>> 
>>>> regards,
>>>> Thaer
>>> 
>>> 
> 
> -- 
> Sorry for being brief. Alternate email is rickleir at yahoo dot com

Re: index new discovered fileds of different types

[Rick Leir <rl...@leirtech.com>]

Thaer
Whoa, hold everything! You said RDF, meaning resource description framework? If so, you have exactly​ three fields: subject, predicate, and object. Maybe they are text type, or for exact matches you might want string fields. Add an ID field, which could be automatically generated by Solr, so now you have four fields. Or am I on a tangent again? Cheers -- Rick

On July 7, 2017 6:01:00 AM EDT, Thaer Sammar <t....@geophy.com> wrote:
>Hi Jan,
>
>Thanks!, I am exploring the schemaless option based on Furkan
>suggestion. I
>need the the flexibility because not all fields are known. We get the
>data
>from RDF database (which changes continuously). To be more specific, we
>have a database and all changes on it are sent to a kafka queue. and we
>have a consumer which listen to the queue and update the Solr index.
>
>regards,
>Thaer
>
>On 7 July 2017 at 10:53, Jan Høydahl <ja...@cominvent.com> wrote:
>
>> If you do not need the flexibility of dynamic fields, don’t use them.
>> Sounds to me that you really want a field “price” to be float and a
>field
>> “birthdate” to be of type date etc.
>> If so, simply create your schema (either manually, through Schema API
>or
>> using schemaless) up front and index each field as correct type
>without
>> messing with field name prefixes.
>>
>> --
>> Jan Høydahl, search solution architect
>> Cominvent AS - www.cominvent.com
>>
>> > 5. jul. 2017 kl. 15.23 skrev Thaer Sammar <t....@geophy.com>:
>> >
>> > Hi,
>> > We are trying to index documents of different types. Document have
>> different fields. fields are known at indexing time. We run a query
>on a
>> database and we index what comes using query variables as field names
>in
>> solr. Our current solution: we use dynamic fields with prefix, for
>example
>> feature_i_*, the issue with that
>> > 1) we need to define the type of the dynamic field and to be able
>to
>> cover the type of discovered fields we define the following
>> > feature_i_* for integers, feature_t_* for string, feature_d_* for
>> double, ....
>> > 1.a) this means we need to check the type of the discovered field
>and
>> then put in the corresponding dynamic field
>> > 2) at search time, we need to know the right prefix
>> > We are looking for help to find away to ignore the prefix and check
>of
>> the type
>> >
>> > regards,
>> > Thaer
>>
>>

-- 
Sorry for being brief. Alternate email is rickleir at yahoo dot com 

Re: index new discovered fileds of different types

[Erick Erickson <er...@gmail.com>]

I'd recommend "managed schema" rather than schemaless. They're related
but distinct.

The problem is that schemaless makes assumptions based on the first
field it finds. So if it finds a field with a "1" in it, it guesses
"int". That'll break if the next doc has a 1.0 since it doesn't parse
to an int.

Managed schema uses the same underlying mechanism to change the
schema, it just let's you control exactly what gets changed.

Best,
Erick

On Fri, Jul 7, 2017 at 3:01 AM, Thaer Sammar <t....@geophy.com> wrote:
> Hi Jan,
>
> Thanks!, I am exploring the schemaless option based on Furkan suggestion. I
> need the the flexibility because not all fields are known. We get the data
> from RDF database (which changes continuously). To be more specific, we
> have a database and all changes on it are sent to a kafka queue. and we
> have a consumer which listen to the queue and update the Solr index.
>
> regards,
> Thaer
>
> On 7 July 2017 at 10:53, Jan Høydahl <ja...@cominvent.com> wrote:
>
>> If you do not need the flexibility of dynamic fields, don’t use them.
>> Sounds to me that you really want a field “price” to be float and a field
>> “birthdate” to be of type date etc.
>> If so, simply create your schema (either manually, through Schema API or
>> using schemaless) up front and index each field as correct type without
>> messing with field name prefixes.
>>
>> --
>> Jan Høydahl, search solution architect
>> Cominvent AS - www.cominvent.com
>>
>> > 5. jul. 2017 kl. 15.23 skrev Thaer Sammar <t....@geophy.com>:
>> >
>> > Hi,
>> > We are trying to index documents of different types. Document have
>> different fields. fields are known at indexing time. We run a query on a
>> database and we index what comes using query variables as field names in
>> solr. Our current solution: we use dynamic fields with prefix, for example
>> feature_i_*, the issue with that
>> > 1) we need to define the type of the dynamic field and to be able to
>> cover the type of discovered fields we define the following
>> > feature_i_* for integers, feature_t_* for string, feature_d_* for
>> double, ....
>> > 1.a) this means we need to check the type of the discovered field and
>> then put in the corresponding dynamic field
>> > 2) at search time, we need to know the right prefix
>> > We are looking for help to find away to ignore the prefix and check of
>> the type
>> >
>> > regards,
>> > Thaer
>>
>>

Re: index new discovered fileds of different types

[Thaer Sammar <t....@geophy.com>]

Hi Jan,

Thanks!, I am exploring the schemaless option based on Furkan suggestion. I
need the the flexibility because not all fields are known. We get the data
from RDF database (which changes continuously). To be more specific, we
have a database and all changes on it are sent to a kafka queue. and we
have a consumer which listen to the queue and update the Solr index.

regards,
Thaer

On 7 July 2017 at 10:53, Jan Høydahl <ja...@cominvent.com> wrote:

> If you do not need the flexibility of dynamic fields, don’t use them.
> Sounds to me that you really want a field “price” to be float and a field
> “birthdate” to be of type date etc.
> If so, simply create your schema (either manually, through Schema API or
> using schemaless) up front and index each field as correct type without
> messing with field name prefixes.
>
> --
> Jan Høydahl, search solution architect
> Cominvent AS - www.cominvent.com
>
> > 5. jul. 2017 kl. 15.23 skrev Thaer Sammar <t....@geophy.com>:
> >
> > Hi,
> > We are trying to index documents of different types. Document have
> different fields. fields are known at indexing time. We run a query on a
> database and we index what comes using query variables as field names in
> solr. Our current solution: we use dynamic fields with prefix, for example
> feature_i_*, the issue with that
> > 1) we need to define the type of the dynamic field and to be able to
> cover the type of discovered fields we define the following
> > feature_i_* for integers, feature_t_* for string, feature_d_* for
> double, ....
> > 1.a) this means we need to check the type of the discovered field and
> then put in the corresponding dynamic field
> > 2) at search time, we need to know the right prefix
> > We are looking for help to find away to ignore the prefix and check of
> the type
> >
> > regards,
> > Thaer
>
>

Re: index new discovered fileds of different types

[Jan Høydahl <ja...@cominvent.com>]

If you do not need the flexibility of dynamic fields, don’t use them.
Sounds to me that you really want a field “price” to be float and a field “birthdate” to be of type date etc.
If so, simply create your schema (either manually, through Schema API or using schemaless) up front and index each field as correct type without messing with field name prefixes.

--
Jan Høydahl, search solution architect
Cominvent AS - www.cominvent.com

> 5. jul. 2017 kl. 15.23 skrev Thaer Sammar <t....@geophy.com>:
> 
> Hi,
> We are trying to index documents of different types. Document have different fields. fields are known at indexing time. We run a query on a database and we index what comes using query variables as field names in solr. Our current solution: we use dynamic fields with prefix, for example feature_i_*, the issue with that
> 1) we need to define the type of the dynamic field and to be able to cover the type of discovered fields we define the following
> feature_i_* for integers, feature_t_* for string, feature_d_* for double, ....
> 1.a) this means we need to check the type of the discovered field and then put in the corresponding dynamic field
> 2) at search time, we need to know the right prefix
> We are looking for help to find away to ignore the prefix and check of the type
> 
> regards,
> Thaer

Re: index new discovered fileds of different types

[Erick Erickson <er...@gmail.com>]

I really have no idea what "to ignore the prefix and check of the type" means.

When? How? Can you give an example of inputs and outputs? You might
want to review:
https://wiki.apache.org/solr/UsingMailingLists

And to add to what Furkan mentioned, in addition to schemaless you can
use "managed schema"
which will allow you to add fields and types on the fly.

Best,
Erick

On Wed, Jul 5, 2017 at 8:12 AM, Thaer Sammar <t....@geophy.com> wrote:
> Hi Furkan,
>
> No, In the schema we also defined some static fields such as uri and geo
> field.
>
> On 5 July 2017 at 17:07, Furkan KAMACI <fu...@gmail.com> wrote:
>
>> Hi Thaer,
>>
>> Do you use schemeless mode [1] ?
>>
>> Kind Regards,
>> Furkan KAMACI
>>
>> [1] https://cwiki.apache.org/confluence/display/solr/Schemaless+Mode
>>
>> On Wed, Jul 5, 2017 at 4:23 PM, Thaer Sammar <t....@geophy.com> wrote:
>>
>> > Hi,
>> > We are trying to index documents of different types. Document have
>> > different fields. fields are known at indexing time. We run a query on a
>> > database and we index what comes using query variables as field names in
>> > solr. Our current solution: we use dynamic fields with prefix, for
>> example
>> > feature_i_*, the issue with that
>> > 1) we need to define the type of the dynamic field and to be able to
>> cover
>> > the type of discovered fields we define the following
>> > feature_i_* for integers, feature_t_* for string, feature_d_* for double,
>> > ....
>> > 1.a) this means we need to check the type of the discovered field and
>> then
>> > put in the corresponding dynamic field
>> > 2) at search time, we need to know the right prefix
>> > We are looking for help to find away to ignore the prefix and check of
>> the
>> > type
>> >
>> > regards,
>> > Thaer
>>

Re: index new discovered fileds of different types

[Thaer Sammar <t....@geophy.com>]

Hi Furkan,

No, In the schema we also defined some static fields such as uri and geo
field.

On 5 July 2017 at 17:07, Furkan KAMACI <fu...@gmail.com> wrote:

> Hi Thaer,
>
> Do you use schemeless mode [1] ?
>
> Kind Regards,
> Furkan KAMACI
>
> [1] https://cwiki.apache.org/confluence/display/solr/Schemaless+Mode
>
> On Wed, Jul 5, 2017 at 4:23 PM, Thaer Sammar <t....@geophy.com> wrote:
>
> > Hi,
> > We are trying to index documents of different types. Document have
> > different fields. fields are known at indexing time. We run a query on a
> > database and we index what comes using query variables as field names in
> > solr. Our current solution: we use dynamic fields with prefix, for
> example
> > feature_i_*, the issue with that
> > 1) we need to define the type of the dynamic field and to be able to
> cover
> > the type of discovered fields we define the following
> > feature_i_* for integers, feature_t_* for string, feature_d_* for double,
> > ....
> > 1.a) this means we need to check the type of the discovered field and
> then
> > put in the corresponding dynamic field
> > 2) at search time, we need to know the right prefix
> > We are looking for help to find away to ignore the prefix and check of
> the
> > type
> >
> > regards,
> > Thaer
>

Re: index new discovered fileds of different types

[Furkan KAMACI <fu...@gmail.com>]

Hi Thaer,

Do you use schemeless mode [1] ?

Kind Regards,
Furkan KAMACI

[1] https://cwiki.apache.org/confluence/display/solr/Schemaless+Mode

On Wed, Jul 5, 2017 at 4:23 PM, Thaer Sammar <t....@geophy.com> wrote:

> Hi,
> We are trying to index documents of different types. Document have
> different fields. fields are known at indexing time. We run a query on a
> database and we index what comes using query variables as field names in
> solr. Our current solution: we use dynamic fields with prefix, for example
> feature_i_*, the issue with that
> 1) we need to define the type of the dynamic field and to be able to cover
> the type of discovered fields we define the following
> feature_i_* for integers, feature_t_* for string, feature_d_* for double,
> ....
> 1.a) this means we need to check the type of the discovered field and then
> put in the corresponding dynamic field
> 2) at search time, we need to know the right prefix
> We are looking for help to find away to ignore the prefix and check of the
> type
>
> regards,
> Thaer