Maximum Number of entires in External Field?

[Aman Tandon <am...@gmail.com>]

Hi,

How much ids information can I define in External File? Currently I am
having the 100 Million records in my index.

With Regards
Aman Tandon

Re: Maximum Number of entires in External Field?

[Aman Tandon <am...@gmail.com>]

>
> I can provide examples if needed.

Yes that will be so much helpful. Thank you so much.

Then I will try both methodology. And will report the results back here.

With Regards
Aman Tandon

On Tue, Sep 8, 2015 at 2:11 PM, Upayavira <uv...@odoko.co.uk> wrote:

> If you have just 5-7 items, then an external file will work, as will the
> join query. You'll need to handle the 'default' case with the join
> query, that is, making sure you do <some query> OR <join query> so that
> documents matching the join are boosted above those matching the main
> query, rather than the join being a filter on the main query.
>
> I can provide examples if needed.
>
> Upayavira
>
> On Mon, Sep 7, 2015, at 07:21 PM, Aman Tandon wrote:
> > I am currently doing boosting for 5-7 things. will  it work great with
> > this
> > too?
> >
> > With Regards
> > Aman Tandon
> >
> > On Mon, Sep 7, 2015 at 11:42 PM, Upayavira <uv...@odoko.co.uk> wrote:
> >
> > > External file field would work, but requires a full import of the
> > > external file field every time you change a single entry, which is
> > > pretty extreme.
> > >
> > > I've tested out "score joins" which seemed to perform very well and
> > > achieved the same effect, but using another core, rather than an
> > > external file.
> > >
> > > Thus:
> > >
> > > {!join score=max fromIndex=prices from=id to=id}{!boost b=price}*:*
> > >
> > > seemed to do the job of using the price as a boost. Of course you could
> > > extend this like so:
> > >
> > > q={!join score=max fromIndex=prices from=id to=id}{!boost b=$b}*:*
> > > b=sqrt(price)
> > >
> > > or such things to make the price a more reasonable value.
> > >
> > > Upayavira
> > >
> > > On Mon, Sep 7, 2015, at 06:21 PM, Aman Tandon wrote:
> > > > Any suggestions?
> > > >
> > > > With Regards
> > > > Aman Tandon
> > > >
> > > > On Mon, Sep 7, 2015 at 1:07 PM, Aman Tandon <amantandon.10@gmail.com
> >
> > > > wrote:
> > > >
> > > > > Hi Upayavira,
> > > > >
> > > > > Have you tried it?
> > > > >
> > > > >
> > > > > No
> > > > >
> > > > > E.g. external file fields don't play nice with Solr Cloud
> > > > >
> > > > >
> > > > > We are not using Solr Cloud.
> > > > >
> > > > >
> > > > >> What are you using the external file for?
> > > > >
> > > > >
> > > > > We are doing the boosting in the search result which are *having
> price
> > > by
> > > > > 1.2* &  *country is India by 1.1*. We are doing by using the
> boosting
> > > > > parameter in conjucation with query & map function e.g.
> > > *&boost=map(query({!dismax
> > > > > qf=hasPrice v='yes' pf=''},0),1,1,1,1)*
> > > > >
> > > > > This is being done with 5/6 parameters. And I am hoping it will
> > > increase
> > > > > query time. So I am planning to make the single score and populate
> it
> > > in
> > > > > external file field. And this might reduce some time.
> > > > >
> > > > > Just to mention we are doing incremental updates after every 10
> > > minutes.
> > > > >
> > > > > With Regards
> > > > > Aman Tandon
> > > > >
> > > > > On Mon, Sep 7, 2015 at 12:53 PM, Upayavira <uv...@odoko.co.uk> wrote:
> > > > >
> > > > >> Have you tried it? I suspect your issue will be with the process
> of
> > > > >> reloading the external file rather than consuming it once loaded.
> > > > >>
> > > > >> What are you using the external file for? There may be other ways
> > > also.
> > > > >> E.g. external file fields don't play nice with Solr Cloud.
> > > > >>
> > > > >> Upayavira
> > > > >>
> > > > >> On Mon, Sep 7, 2015, at 07:05 AM, Aman Tandon wrote:
> > > > >> > Hi,
> > > > >> >
> > > > >> > How much ids information can I define in External File?
> Currently I
> > > am
> > > > >> > having the 100 Million records in my index.
> > > > >> >
> > > > >> > With Regards
> > > > >> > Aman Tandon
> > > > >>
> > > > >
> > > > >
> > >
>

Re: Maximum Number of entires in External Field?

[Upayavira <uv...@odoko.co.uk>]

If you have just 5-7 items, then an external file will work, as will the
join query. You'll need to handle the 'default' case with the join
query, that is, making sure you do <some query> OR <join query> so that
documents matching the join are boosted above those matching the main
query, rather than the join being a filter on the main query.

I can provide examples if needed.

Upayavira

On Mon, Sep 7, 2015, at 07:21 PM, Aman Tandon wrote:
> I am currently doing boosting for 5-7 things. will  it work great with
> this
> too?
> 
> With Regards
> Aman Tandon
> 
> On Mon, Sep 7, 2015 at 11:42 PM, Upayavira <uv...@odoko.co.uk> wrote:
> 
> > External file field would work, but requires a full import of the
> > external file field every time you change a single entry, which is
> > pretty extreme.
> >
> > I've tested out "score joins" which seemed to perform very well and
> > achieved the same effect, but using another core, rather than an
> > external file.
> >
> > Thus:
> >
> > {!join score=max fromIndex=prices from=id to=id}{!boost b=price}*:*
> >
> > seemed to do the job of using the price as a boost. Of course you could
> > extend this like so:
> >
> > q={!join score=max fromIndex=prices from=id to=id}{!boost b=$b}*:*
> > b=sqrt(price)
> >
> > or such things to make the price a more reasonable value.
> >
> > Upayavira
> >
> > On Mon, Sep 7, 2015, at 06:21 PM, Aman Tandon wrote:
> > > Any suggestions?
> > >
> > > With Regards
> > > Aman Tandon
> > >
> > > On Mon, Sep 7, 2015 at 1:07 PM, Aman Tandon <am...@gmail.com>
> > > wrote:
> > >
> > > > Hi Upayavira,
> > > >
> > > > Have you tried it?
> > > >
> > > >
> > > > No
> > > >
> > > > E.g. external file fields don't play nice with Solr Cloud
> > > >
> > > >
> > > > We are not using Solr Cloud.
> > > >
> > > >
> > > >> What are you using the external file for?
> > > >
> > > >
> > > > We are doing the boosting in the search result which are *having price
> > by
> > > > 1.2* &  *country is India by 1.1*. We are doing by using the boosting
> > > > parameter in conjucation with query & map function e.g.
> > *&boost=map(query({!dismax
> > > > qf=hasPrice v='yes' pf=''},0),1,1,1,1)*
> > > >
> > > > This is being done with 5/6 parameters. And I am hoping it will
> > increase
> > > > query time. So I am planning to make the single score and populate it
> > in
> > > > external file field. And this might reduce some time.
> > > >
> > > > Just to mention we are doing incremental updates after every 10
> > minutes.
> > > >
> > > > With Regards
> > > > Aman Tandon
> > > >
> > > > On Mon, Sep 7, 2015 at 12:53 PM, Upayavira <uv...@odoko.co.uk> wrote:
> > > >
> > > >> Have you tried it? I suspect your issue will be with the process of
> > > >> reloading the external file rather than consuming it once loaded.
> > > >>
> > > >> What are you using the external file for? There may be other ways
> > also.
> > > >> E.g. external file fields don't play nice with Solr Cloud.
> > > >>
> > > >> Upayavira
> > > >>
> > > >> On Mon, Sep 7, 2015, at 07:05 AM, Aman Tandon wrote:
> > > >> > Hi,
> > > >> >
> > > >> > How much ids information can I define in External File? Currently I
> > am
> > > >> > having the 100 Million records in my index.
> > > >> >
> > > >> > With Regards
> > > >> > Aman Tandon
> > > >>
> > > >
> > > >
> >

Re: Maximum Number of entires in External Field?

[Aman Tandon <am...@gmail.com>]

I am currently doing boosting for 5-7 things. will  it work great with this
too?

With Regards
Aman Tandon

On Mon, Sep 7, 2015 at 11:42 PM, Upayavira <uv...@odoko.co.uk> wrote:

> External file field would work, but requires a full import of the
> external file field every time you change a single entry, which is
> pretty extreme.
>
> I've tested out "score joins" which seemed to perform very well and
> achieved the same effect, but using another core, rather than an
> external file.
>
> Thus:
>
> {!join score=max fromIndex=prices from=id to=id}{!boost b=price}*:*
>
> seemed to do the job of using the price as a boost. Of course you could
> extend this like so:
>
> q={!join score=max fromIndex=prices from=id to=id}{!boost b=$b}*:*
> b=sqrt(price)
>
> or such things to make the price a more reasonable value.
>
> Upayavira
>
> On Mon, Sep 7, 2015, at 06:21 PM, Aman Tandon wrote:
> > Any suggestions?
> >
> > With Regards
> > Aman Tandon
> >
> > On Mon, Sep 7, 2015 at 1:07 PM, Aman Tandon <am...@gmail.com>
> > wrote:
> >
> > > Hi Upayavira,
> > >
> > > Have you tried it?
> > >
> > >
> > > No
> > >
> > > E.g. external file fields don't play nice with Solr Cloud
> > >
> > >
> > > We are not using Solr Cloud.
> > >
> > >
> > >> What are you using the external file for?
> > >
> > >
> > > We are doing the boosting in the search result which are *having price
> by
> > > 1.2* &  *country is India by 1.1*. We are doing by using the boosting
> > > parameter in conjucation with query & map function e.g.
> *&boost=map(query({!dismax
> > > qf=hasPrice v='yes' pf=''},0),1,1,1,1)*
> > >
> > > This is being done with 5/6 parameters. And I am hoping it will
> increase
> > > query time. So I am planning to make the single score and populate it
> in
> > > external file field. And this might reduce some time.
> > >
> > > Just to mention we are doing incremental updates after every 10
> minutes.
> > >
> > > With Regards
> > > Aman Tandon
> > >
> > > On Mon, Sep 7, 2015 at 12:53 PM, Upayavira <uv...@odoko.co.uk> wrote:
> > >
> > >> Have you tried it? I suspect your issue will be with the process of
> > >> reloading the external file rather than consuming it once loaded.
> > >>
> > >> What are you using the external file for? There may be other ways
> also.
> > >> E.g. external file fields don't play nice with Solr Cloud.
> > >>
> > >> Upayavira
> > >>
> > >> On Mon, Sep 7, 2015, at 07:05 AM, Aman Tandon wrote:
> > >> > Hi,
> > >> >
> > >> > How much ids information can I define in External File? Currently I
> am
> > >> > having the 100 Million records in my index.
> > >> >
> > >> > With Regards
> > >> > Aman Tandon
> > >>
> > >
> > >
>

Re: Maximum Number of entires in External Field?

[Upayavira <uv...@odoko.co.uk>]

External file field would work, but requires a full import of the
external file field every time you change a single entry, which is
pretty extreme.

I've tested out "score joins" which seemed to perform very well and
achieved the same effect, but using another core, rather than an
external file.

Thus:

{!join score=max fromIndex=prices from=id to=id}{!boost b=price}*:*

seemed to do the job of using the price as a boost. Of course you could
extend this like so:

q={!join score=max fromIndex=prices from=id to=id}{!boost b=$b}*:*
b=sqrt(price)

or such things to make the price a more reasonable value.

Upayavira

On Mon, Sep 7, 2015, at 06:21 PM, Aman Tandon wrote:
> Any suggestions?
> 
> With Regards
> Aman Tandon
> 
> On Mon, Sep 7, 2015 at 1:07 PM, Aman Tandon <am...@gmail.com>
> wrote:
> 
> > Hi Upayavira,
> >
> > Have you tried it?
> >
> >
> > No
> >
> > E.g. external file fields don't play nice with Solr Cloud
> >
> >
> > We are not using Solr Cloud.
> >
> >
> >> What are you using the external file for?
> >
> >
> > We are doing the boosting in the search result which are *having price by
> > 1.2* &  *country is India by 1.1*. We are doing by using the boosting
> > parameter in conjucation with query & map function e.g. *&boost=map(query({!dismax
> > qf=hasPrice v='yes' pf=''},0),1,1,1,1)*
> >
> > This is being done with 5/6 parameters. And I am hoping it will increase
> > query time. So I am planning to make the single score and populate it in
> > external file field. And this might reduce some time.
> >
> > Just to mention we are doing incremental updates after every 10 minutes.
> >
> > With Regards
> > Aman Tandon
> >
> > On Mon, Sep 7, 2015 at 12:53 PM, Upayavira <uv...@odoko.co.uk> wrote:
> >
> >> Have you tried it? I suspect your issue will be with the process of
> >> reloading the external file rather than consuming it once loaded.
> >>
> >> What are you using the external file for? There may be other ways also.
> >> E.g. external file fields don't play nice with Solr Cloud.
> >>
> >> Upayavira
> >>
> >> On Mon, Sep 7, 2015, at 07:05 AM, Aman Tandon wrote:
> >> > Hi,
> >> >
> >> > How much ids information can I define in External File? Currently I am
> >> > having the 100 Million records in my index.
> >> >
> >> > With Regards
> >> > Aman Tandon
> >>
> >
> >

Re: Maximum Number of entires in External Field?

[Aman Tandon <am...@gmail.com>]

Any suggestions?

With Regards
Aman Tandon

On Mon, Sep 7, 2015 at 1:07 PM, Aman Tandon <am...@gmail.com> wrote:

> Hi Upayavira,
>
> Have you tried it?
>
>
> No
>
> E.g. external file fields don't play nice with Solr Cloud
>
>
> We are not using Solr Cloud.
>
>
>> What are you using the external file for?
>
>
> We are doing the boosting in the search result which are *having price by
> 1.2* &  *country is India by 1.1*. We are doing by using the boosting
> parameter in conjucation with query & map function e.g. *&boost=map(query({!dismax
> qf=hasPrice v='yes' pf=''},0),1,1,1,1)*
>
> This is being done with 5/6 parameters. And I am hoping it will increase
> query time. So I am planning to make the single score and populate it in
> external file field. And this might reduce some time.
>
> Just to mention we are doing incremental updates after every 10 minutes.
>
> With Regards
> Aman Tandon
>
> On Mon, Sep 7, 2015 at 12:53 PM, Upayavira <uv...@odoko.co.uk> wrote:
>
>> Have you tried it? I suspect your issue will be with the process of
>> reloading the external file rather than consuming it once loaded.
>>
>> What are you using the external file for? There may be other ways also.
>> E.g. external file fields don't play nice with Solr Cloud.
>>
>> Upayavira
>>
>> On Mon, Sep 7, 2015, at 07:05 AM, Aman Tandon wrote:
>> > Hi,
>> >
>> > How much ids information can I define in External File? Currently I am
>> > having the 100 Million records in my index.
>> >
>> > With Regards
>> > Aman Tandon
>>
>
>

Re: Maximum Number of entires in External Field?

[Aman Tandon <am...@gmail.com>]

Hi Upayavira,

Have you tried it?


No

E.g. external file fields don't play nice with Solr Cloud


We are not using Solr Cloud.


> What are you using the external file for?


We are doing the boosting in the search result which are *having price by
1.2* &  *country is India by 1.1*. We are doing by using the boosting
parameter in conjucation with query & map function e.g.
*&boost=map(query({!dismax
qf=hasPrice v='yes' pf=''},0),1,1,1,1)*

This is being done with 5/6 parameters. And I am hoping it will increase
query time. So I am planning to make the single score and populate it in
external file field. And this might reduce some time.

Just to mention we are doing incremental updates after every 10 minutes.

With Regards
Aman Tandon

On Mon, Sep 7, 2015 at 12:53 PM, Upayavira <uv...@odoko.co.uk> wrote:

> Have you tried it? I suspect your issue will be with the process of
> reloading the external file rather than consuming it once loaded.
>
> What are you using the external file for? There may be other ways also.
> E.g. external file fields don't play nice with Solr Cloud.
>
> Upayavira
>
> On Mon, Sep 7, 2015, at 07:05 AM, Aman Tandon wrote:
> > Hi,
> >
> > How much ids information can I define in External File? Currently I am
> > having the 100 Million records in my index.
> >
> > With Regards
> > Aman Tandon
>

Re: Maximum Number of entires in External Field?

[Upayavira <uv...@odoko.co.uk>]

Have you tried it? I suspect your issue will be with the process of
reloading the external file rather than consuming it once loaded.

What are you using the external file for? There may be other ways also.
E.g. external file fields don't play nice with Solr Cloud.

Upayavira

On Mon, Sep 7, 2015, at 07:05 AM, Aman Tandon wrote:
> Hi,
> 
> How much ids information can I define in External File? Currently I am
> having the 100 Million records in my index.
> 
> With Regards
> Aman Tandon