scale

[John Blythe <jo...@curvolabs.com>]

hi all,

having lots of processing happening using multiple solr cores to do some
data linkage with our customers' transactional data. it runs pretty slowly
at the moment. we were wondering if there were some solr or jetty tunings
that we could implement to help make it more powerful and efficient. it
currently is using less than 2GB on our box, can we open it up to use more
memory and get speedier as a result?

thanks for any tips!

Re: scale

[John Blythe <jo...@curvolabs.com>]

sure.

the processes we run to do linkage take hours. we're processing ~600k
records, bouncing our users data up against a few data sources that act as
'sources of truth' for us for the sake of this linkage. we get the top 3
results and run some quick checks on it algorithmically to determine if we
have a match. we use parallel requests of 100 at a time.

solr isn't built for this sort of purpose specifically, i'm pretty sure,
but even so i'm imagining/hoping there is a way to give it a bit more
processing power.

thanks for any continued discussion!


-- 
*John Blythe*
Product Manager & Lead Developer

251.605.3071 | john@curvolabs.com
www.curvolabs.com

58 Adams Ave
Evansville, IN 47713

On Thu, Jun 2, 2016 at 12:49 PM, Erick Erickson <er...@gmail.com>
wrote:

> Without having a lot more data it's hard to say anything helpful.
>
> _What_ is slow? What does "data linkage" mean exactly? Etc.
>
> Best,
> Erick
>
> On Thu, Jun 2, 2016 at 9:33 AM, John Blythe <jo...@curvolabs.com> wrote:
> > hi all,
> >
> > having lots of processing happening using multiple solr cores to do some
> > data linkage with our customers' transactional data. it runs pretty
> slowly
> > at the moment. we were wondering if there were some solr or jetty tunings
> > that we could implement to help make it more powerful and efficient. it
> > currently is using less than 2GB on our box, can we open it up to use
> more
> > memory and get speedier as a result?
> >
> > thanks for any tips!
>

Re: scale

[Erick Erickson <er...@gmail.com>]

Without having a lot more data it's hard to say anything helpful.

_What_ is slow? What does "data linkage" mean exactly? Etc.

Best,
Erick

On Thu, Jun 2, 2016 at 9:33 AM, John Blythe <jo...@curvolabs.com> wrote:
> hi all,
>
> having lots of processing happening using multiple solr cores to do some
> data linkage with our customers' transactional data. it runs pretty slowly
> at the moment. we were wondering if there were some solr or jetty tunings
> that we could implement to help make it more powerful and efficient. it
> currently is using less than 2GB on our box, can we open it up to use more
> memory and get speedier as a result?
>
> thanks for any tips!