You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Santanu8939967892 <mi...@gmail.com> on 2013/07/29 14:00:12 UTC
DIH to index the data - 250 millions - Need a best architecture
Hi,
I have a huge volume of DB records, which is close to 250 millions.
I am going to use DIH to index the data into Solr.
I need a best architecture to index and query the data in an efficient
manner.
I am using windows server 2008 with 16 GB RAM, zion processor and Solr 4.4.
With Regards,
Santanu
Re: DIH to index the data - 250 millions - Need a best architecture
Posted by Jack Krupansky <ja...@basetechnology.com>.
You neglected to provide information about the filters or the "20 or 30 meta
data information".
Did you mean to imply that you will not be querying against the metadata
(only returning it)?
-- Jack Krupansky
-----Original Message-----
From: Santanu8939967892
Sent: Monday, July 29, 2013 9:41 AM
To: solr-user@lucene.apache.org
Subject: Re: DIH to index the data - 250 millions - Need a best architecture
Hi Jack,
My sample query will be with a keyword (text) and probably 2 to 3
filters.
There is a java interface for display of data, which will consume a class,
and the class returns a data set object using SolrJ.
So for display we will use a list for binding. we may display 20 or 30 meta
data information.
I believe I have provided the information you have asked for.
With Regards,
Santanu
On Mon, Jul 29, 2013 at 5:50 PM, Jack Krupansky
<ja...@basetechnology.com>wrote:
> The initial question is not how to index the data, but how you want to use
> or query the data. Use cases for query and data access should drive the
> data model that you will use to index the data.
>
> So, what are some sample queries? How will users want to search and access
> the data? What data will they expect to see and in what form? Not so much
> from a UI perspective, but in terms of how the client app(s) will access
> data.
>
> -- Jack Krupansky
>
> -----Original Message----- From: Santanu8939967892
> Sent: Monday, July 29, 2013 8:00 AM
> To: solr-user@lucene.apache.org
> Subject: DIH to index the data - 250 millions - Need a best architecture
>
>
> Hi,
> I have a huge volume of DB records, which is close to 250 millions.
> I am going to use DIH to index the data into Solr.
> I need a best architecture to index and query the data in an efficient
> manner.
> I am using windows server 2008 with 16 GB RAM, zion processor and Solr
> 4.4.
>
>
> With Regards,
> Santanu
>
Re: DIH to index the data - 250 millions - Need a best architecture
Posted by Santanu8939967892 <mi...@gmail.com>.
Hi Jack,
My sample query will be with a keyword (text) and probably 2 to 3
filters.
There is a java interface for display of data, which will consume a class,
and the class returns a data set object using SolrJ.
So for display we will use a list for binding. we may display 20 or 30 meta
data information.
I believe I have provided the information you have asked for.
With Regards,
Santanu
On Mon, Jul 29, 2013 at 5:50 PM, Jack Krupansky <ja...@basetechnology.com>wrote:
> The initial question is not how to index the data, but how you want to use
> or query the data. Use cases for query and data access should drive the
> data model that you will use to index the data.
>
> So, what are some sample queries? How will users want to search and access
> the data? What data will they expect to see and in what form? Not so much
> from a UI perspective, but in terms of how the client app(s) will access
> data.
>
> -- Jack Krupansky
>
> -----Original Message----- From: Santanu8939967892
> Sent: Monday, July 29, 2013 8:00 AM
> To: solr-user@lucene.apache.org
> Subject: DIH to index the data - 250 millions - Need a best architecture
>
>
> Hi,
> I have a huge volume of DB records, which is close to 250 millions.
> I am going to use DIH to index the data into Solr.
> I need a best architecture to index and query the data in an efficient
> manner.
> I am using windows server 2008 with 16 GB RAM, zion processor and Solr 4.4.
>
>
> With Regards,
> Santanu
>
Re: DIH to index the data - 250 millions - Need a best architecture
Posted by Jack Krupansky <ja...@basetechnology.com>.
The initial question is not how to index the data, but how you want to use
or query the data. Use cases for query and data access should drive the data
model that you will use to index the data.
So, what are some sample queries? How will users want to search and access
the data? What data will they expect to see and in what form? Not so much
from a UI perspective, but in terms of how the client app(s) will access
data.
-- Jack Krupansky
-----Original Message-----
From: Santanu8939967892
Sent: Monday, July 29, 2013 8:00 AM
To: solr-user@lucene.apache.org
Subject: DIH to index the data - 250 millions - Need a best architecture
Hi,
I have a huge volume of DB records, which is close to 250 millions.
I am going to use DIH to index the data into Solr.
I need a best architecture to index and query the data in an efficient
manner.
I am using windows server 2008 with 16 GB RAM, zion processor and Solr 4.4.
With Regards,
Santanu
Re: DIH to index the data - 250 millions - Need a best architecture
Posted by Shawn Heisey <so...@elyograg.org>.
On 7/29/2013 6:00 AM, Santanu8939967892 wrote:
> Hi,
> I have a huge volume of DB records, which is close to 250 millions.
> I am going to use DIH to index the data into Solr.
> I need a best architecture to index and query the data in an efficient
> manner.
> I am using windows server 2008 with 16 GB RAM, zion processor and Solr 4.4.
Gora and Jack have given you great information. I would add that when
you are dealing with an index of this size, you need to be prepared to
spend some real money on hardware if you want maximum performance.
With 20-30 fields, I would imagine that each document is probably a few
KB in size. Even if they will be much smaller than that, with 250
million of them, your index will be pretty large.
I'd be VERY surprised if the index is less than 100GB, and something
larger than 500GB is probably more likely. For illustration purposes,
let's be conservative and say it's 200GB.
16GB of RAM isn't enough for an index that size. An ideal "round"
memory size for a 200GB index would be 256GB - 200GB of RAM for the OS
disk cache and enough memory for whatever size java heap you might need.
In truth, you probably don't need to cache the ENTIRE index ... most
searches will involve only certain parts of the index and won't touch
the entire thing. A "good enough" memory size might be 128GB which
would keep the most relevant parts of the index in RAM at all times.
If you were to put a 200GB index onto a disk that's SSD, you could
probably get away with 64GB of RAM - 50GB or so for the OS disk cache
and the rest for the java heap.
If your index will be larger than 200GB, then the numbers I have given
you will go up. These numbers also assume that you have your entire
index on one server, which is probably not a good idea.
http://wiki.apache.org/solr/SolrPerformanceProblems
SolrCloud would likely be the best architecture. It would spread out
your system requirements and load across multiple machines. If you had
20 machines, each with 16-32GB of RAM, you could do a SolrCloud
installation with 10 shards and a replicationFactor of 2, and there
wouldn't be any memory problems. Each machine would have 25 million
records on it, and you'd have two complete copies of your index so you'd
be able to keep running if a machine completely failed -- which DOES happen.
The information I've given you is for an ideal setup. You can go
smaller, and budget needs might indeed cause you to go smaller. If you
don't need extremely good performance from Solr, then you don't need to
spend the money required for an architecture like I've described.
Thanks,
Shawn
Re: DIH to index the data - 250 millions - Need a best architecture
Posted by Santanu8939967892 <mi...@gmail.com>.
Hi Gora,
I wanted to highlight one point here.
As my content volume is large what should be my index architecture single
core, cloud... ?
With Regards,
Santanu
On Mon, Jul 29, 2013 at 5:37 PM, Gora Mohanty <go...@mimirtech.com> wrote:
> On 29 July 2013 17:30, Santanu8939967892 <mi...@gmail.com> wrote:
> > Hi,
> > I have a huge volume of DB records, which is close to 250 millions.
> > I am going to use DIH to index the data into Solr.
> > I need a best architecture to index and query the data in an efficient
> > manner.
> [...]
>
> This is difficult to answer without knowing details of your
> particular use case. Your best bet would be to prototype
> a system, and measure performance on at least a subset
> of the data. If you search through earlier message on the
> list, you should also come across some numbers for
> performance, but it is best to test for your own needs.
>
> Regards,
> Gora
>
Re: DIH to index the data - 250 millions - Need a best architecture
Posted by Gora Mohanty <go...@mimirtech.com>.
On 29 July 2013 17:30, Santanu8939967892 <mi...@gmail.com> wrote:
> Hi,
> I have a huge volume of DB records, which is close to 250 millions.
> I am going to use DIH to index the data into Solr.
> I need a best architecture to index and query the data in an efficient
> manner.
[...]
This is difficult to answer without knowing details of your
particular use case. Your best bet would be to prototype
a system, and measure performance on at least a subset
of the data. If you search through earlier message on the
list, you should also come across some numbers for
performance, but it is best to test for your own needs.
Regards,
Gora
Re: DIH to index the data - 250 millions - Need a best architecture
Posted by Santanu8939967892 <mi...@gmail.com>.
Hi,
In addition to my last mail one further query.
Can we automate the deployment process for multinode environment (N..
nodes)?
With Regards,
Santanu
On Tue, Jul 30, 2013 at 11:53 AM, Santanu8939967892 <
mishra.santanu@gmail.com> wrote:
> Hi Shawn,
> Yes, your assumption is correct. The index size is around 250 GB and
> we index 20/30 meta data and store around 50.
> We have plan for a Solr cloud architecture having two nodes one
> Master and other one is replica of the master (replication factor 2) with
> multiple zookeeper ensemble. We will have multiple shards for each Master
> and replica node.
> Is above architecture a fit for production deployment for an improved
> index and query performance.
> Do we require 64 GB RAM or less will work for us.
>
> With Regards,
> Santanu
>
>
>
> On Tue, Jul 30, 2013 at 12:59 AM, Mikhail Khludnev <
> mkhludnev@griddynamics.com> wrote:
>
>> Mishra,
>> What if you setup DIH with single SQLEntityProcessor without caching, does
>> it works for you?
>>
>>
>> On Mon, Jul 29, 2013 at 4:00 PM, Santanu8939967892 <
>> mishra.santanu@gmail.com
>> > wrote:
>>
>> > Hi,
>> > I have a huge volume of DB records, which is close to 250 millions.
>> > I am going to use DIH to index the data into Solr.
>> > I need a best architecture to index and query the data in an efficient
>> > manner.
>> > I am using windows server 2008 with 16 GB RAM, zion processor and Solr
>> 4.4.
>> >
>> >
>> > With Regards,
>> > Santanu
>> >
>>
>>
>>
>> --
>> Sincerely yours
>> Mikhail Khludnev
>> Principal Engineer,
>> Grid Dynamics
>>
>> <http://www.griddynamics.com>
>> <mk...@griddynamics.com>
>>
>
>
Re: DIH to index the data - 250 millions - Need a best architecture
Posted by Santanu8939967892 <mi...@gmail.com>.
Hi Shawn,
Thanks for your detailed explanation.
Will do a POC and finalize the arch.
With Regards,
Santanu
On Tue, Jul 30, 2013 at 12:20 PM, Shawn Heisey <so...@elyograg.org> wrote:
> On 7/30/2013 12:23 AM, Santanu8939967892 wrote:
> > Yes, your assumption is correct. The index size is around 250 GB and
> > we index 20/30 meta data and store around 50.
> > We have plan for a Solr cloud architecture having two nodes one
> Master
> > and other one is replica of the master (replication factor 2) with
> multiple
> > zookeeper ensemble. We will have multiple shards for each Master and
> > replica node.
> > Is above architecture a fit for production deployment for an improved
> index
> > and query performance.
> > Do we require 64 GB RAM or less will work for us.
>
> It sounds like you're planning to put the entire index on one server,
> and then have a replica on another server. You'll have multiple shards,
> but they won't be running on separate hardware. Running multiple shards
> per server is a strategy that can work well if you have a lot CPU cores
> and a low query volume. When the query volume gets really high, you
> will want fewer shards per server and more servers.
>
> If your index is on spinning disks, I wouldn't try to run an index of
> that size on a host with less than 128GB RAM, and I'd try to get 256GB.
> If you have to choose between super-high-end CPUs and memory, choose
> memory ... but don't skimp TOO much on the CPUs. The amount of RAM
> required for each server will go down if you spread the shards out
> across more servers.
>
> If the index is on SSD, 64GB might work OK, but 128GB would be better.
> If your query volume is low, 64GB might even work for spinning disks,
> but the query latency might be fairly high.
>
> If you require a very high query volume, two replicas might not be
> enough, and you wouldn't want to run a lot of shards per server. You'd
> have to actually set up a proof of concept and run tests with real data
> and real queries to find out for sure what you need.
>
> In case it isn't clear by now - assuming you've got enough RAM for good
> disk caching, query volume will dictate how many actual servers you need.
>
> Thanks,
> Shawn
>
>
Re: DIH to index the data - 250 millions - Need a best architecture
Posted by Shawn Heisey <so...@elyograg.org>.
On 7/30/2013 12:23 AM, Santanu8939967892 wrote:
> Yes, your assumption is correct. The index size is around 250 GB and
> we index 20/30 meta data and store around 50.
> We have plan for a Solr cloud architecture having two nodes one Master
> and other one is replica of the master (replication factor 2) with multiple
> zookeeper ensemble. We will have multiple shards for each Master and
> replica node.
> Is above architecture a fit for production deployment for an improved index
> and query performance.
> Do we require 64 GB RAM or less will work for us.
It sounds like you're planning to put the entire index on one server,
and then have a replica on another server. You'll have multiple shards,
but they won't be running on separate hardware. Running multiple shards
per server is a strategy that can work well if you have a lot CPU cores
and a low query volume. When the query volume gets really high, you
will want fewer shards per server and more servers.
If your index is on spinning disks, I wouldn't try to run an index of
that size on a host with less than 128GB RAM, and I'd try to get 256GB.
If you have to choose between super-high-end CPUs and memory, choose
memory ... but don't skimp TOO much on the CPUs. The amount of RAM
required for each server will go down if you spread the shards out
across more servers.
If the index is on SSD, 64GB might work OK, but 128GB would be better.
If your query volume is low, 64GB might even work for spinning disks,
but the query latency might be fairly high.
If you require a very high query volume, two replicas might not be
enough, and you wouldn't want to run a lot of shards per server. You'd
have to actually set up a proof of concept and run tests with real data
and real queries to find out for sure what you need.
In case it isn't clear by now - assuming you've got enough RAM for good
disk caching, query volume will dictate how many actual servers you need.
Thanks,
Shawn
Re: DIH to index the data - 250 millions - Need a best architecture
Posted by Santanu8939967892 <mi...@gmail.com>.
Hi Shawn,
Yes, your assumption is correct. The index size is around 250 GB and
we index 20/30 meta data and store around 50.
We have plan for a Solr cloud architecture having two nodes one Master
and other one is replica of the master (replication factor 2) with multiple
zookeeper ensemble. We will have multiple shards for each Master and
replica node.
Is above architecture a fit for production deployment for an improved index
and query performance.
Do we require 64 GB RAM or less will work for us.
With Regards,
Santanu
On Tue, Jul 30, 2013 at 12:59 AM, Mikhail Khludnev <
mkhludnev@griddynamics.com> wrote:
> Mishra,
> What if you setup DIH with single SQLEntityProcessor without caching, does
> it works for you?
>
>
> On Mon, Jul 29, 2013 at 4:00 PM, Santanu8939967892 <
> mishra.santanu@gmail.com
> > wrote:
>
> > Hi,
> > I have a huge volume of DB records, which is close to 250 millions.
> > I am going to use DIH to index the data into Solr.
> > I need a best architecture to index and query the data in an efficient
> > manner.
> > I am using windows server 2008 with 16 GB RAM, zion processor and Solr
> 4.4.
> >
> >
> > With Regards,
> > Santanu
> >
>
>
>
> --
> Sincerely yours
> Mikhail Khludnev
> Principal Engineer,
> Grid Dynamics
>
> <http://www.griddynamics.com>
> <mk...@griddynamics.com>
>
Re: DIH to index the data - 250 millions - Need a best architecture
Posted by Mikhail Khludnev <mk...@griddynamics.com>.
Mishra,
What if you setup DIH with single SQLEntityProcessor without caching, does
it works for you?
On Mon, Jul 29, 2013 at 4:00 PM, Santanu8939967892 <mishra.santanu@gmail.com
> wrote:
> Hi,
> I have a huge volume of DB records, which is close to 250 millions.
> I am going to use DIH to index the data into Solr.
> I need a best architecture to index and query the data in an efficient
> manner.
> I am using windows server 2008 with 16 GB RAM, zion processor and Solr 4.4.
>
>
> With Regards,
> Santanu
>
--
Sincerely yours
Mikhail Khludnev
Principal Engineer,
Grid Dynamics
<http://www.griddynamics.com>
<mk...@griddynamics.com>