You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by matthew sporleder <ms...@gmail.com> on 2020/09/01 15:21:31 UTC

external field file size

We are researching the canonical use case for external fields --
traffic-based rankings

What are the practical limits on the size of the external field file?
A k=v text file seems like it might fall over if it grows into the GB
range?

Our other thought is to use rolling cores where we stream in web logs
and use !join queries.

Does anyone have practical experience with this that they might want to share?

Thanks,
Matt

Re: external field file size

Posted by matthew sporleder <ms...@gmail.com>.
Okay thanks for the tip.  I am pretty wary of streaming logs into my
main set of documents + tons of $stat_updated_at fields + resetting
stats on ~every document every day + whatever else we feel like
trending.  It just feels like a lot of churn.

I will lean towards the !join on stats-$DATE probably.

On Tue, Sep 1, 2020 at 11:32 AM Erick Erickson <er...@gmail.com> wrote:
>
> I wouldn’t use ExternalFileField if your use-case is served by in-place updates. See
>
> https://lucene.apache.org/solr/guide/8_1/updating-parts-of-documents.html#in-place-updates
>
> EFFs were put in in order to have _some_ capability to change individual fields in a doc
> long before in-place updates were around and long before SolrCloud. Using EFF in any
> kind of sharded system will cause you significant heartburn in terms of keeping the
> file up to date on all replicas.
>
> Best,
> Erick
>
> > On Sep 1, 2020, at 11:21 AM, matthew sporleder <ms...@gmail.com> wrote:
> >
> > We are researching the canonical use case for external fields --
> > traffic-based rankings
> >
> > What are the practical limits on the size of the external field file?
> > A k=v text file seems like it might fall over if it grows into the GB
> > range?
> >
> > Our other thought is to use rolling cores where we stream in web logs
> > and use !join queries.
> >
> > Does anyone have practical experience with this that they might want to share?
> >
> > Thanks,
> > Matt
>

Re: external field file size

Posted by Erick Erickson <er...@gmail.com>.
I wouldn’t use ExternalFileField if your use-case is served by in-place updates. See 

https://lucene.apache.org/solr/guide/8_1/updating-parts-of-documents.html#in-place-updates

EFFs were put in in order to have _some_ capability to change individual fields in a doc
long before in-place updates were around and long before SolrCloud. Using EFF in any
kind of sharded system will cause you significant heartburn in terms of keeping the
file up to date on all replicas.

Best,
Erick

> On Sep 1, 2020, at 11:21 AM, matthew sporleder <ms...@gmail.com> wrote:
> 
> We are researching the canonical use case for external fields --
> traffic-based rankings
> 
> What are the practical limits on the size of the external field file?
> A k=v text file seems like it might fall over if it grows into the GB
> range?
> 
> Our other thought is to use rolling cores where we stream in web logs
> and use !join queries.
> 
> Does anyone have practical experience with this that they might want to share?
> 
> Thanks,
> Matt