You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Vijay B <vi...@gmail.com> on 2014/12/04 15:19:05 UTC
Index replication strategy
Hello,
We index docs coming from database nightly. Current index is sitting on
NFS. Due to obvious performance reasons, we are switching are planning to
switch to local index. W have cluster of 4 servers and with NFS it was not
a problem for us until now to share the index. but going forward, we are
looking for our design options for index replication on to local storage.
Our setup:
Index size: 8GB (grows by 2GB every year)
Lucene: 4.2.1
64-bit Java
The options we considered:
* Each server instance,hosting a nightly job to pull delta of data from
db. But, this would result in high DB load. (4 severs =4 times the load)
* An additional nightly job sitting on another sever, that pushes the data
on to local disks of each instances..This may not work out as the local
disk may not be visible.
* each sever hosting a replication job that pulls delta of data from NFS
and stores in the local index...so far this is the only promising option we
have.
* Does solr an option for us in this case? (I know it's a question for solr
group..but experts here might have some thoughts..)..
Thank you for your attention.
Re: Index replication strategy
Posted by Vijay B <vi...@gmail.com>.
Looks very promising. I will check out your blog.
On Thu, Dec 4, 2014 at 9:38 AM, Shai Erera <se...@gmail.com> wrote:
> Ooops, didn't notice that :).
>
> So you'll need to upgrade to Lucene 4.4.0 in order to use it. You can read
> some details as well as example code here:
> http://shaierera.blogspot.com/2013/05/the-replicator.html.
>
> Shai
>
> On Thu, Dec 4, 2014 at 4:36 PM, Vijay B <vi...@gmail.com> wrote:
>
> > As indicated in my post, we use Lucene 4.2.1.
> >
> > On Thu, Dec 4, 2014 at 9:29 AM, Shai Erera <se...@gmail.com> wrote:
> >
> > > Do you use Lucene or Solr? Lucene also has a replication module, which
> > will
> > > allow you to replicate index changes.
> > >
> > > On Thu, Dec 4, 2014 at 4:19 PM, Vijay B <vi...@gmail.com>
> wrote:
> > >
> > > > Hello,
> > > >
> > > > We index docs coming from database nightly. Current index is sitting
> on
> > > > NFS. Due to obvious performance reasons, we are switching are
> planning
> > to
> > > > switch to local index. W have cluster of 4 servers and with NFS it
> was
> > > not
> > > > a problem for us until now to share the index. but going forward, we
> > are
> > > > looking for our design options for index replication on to local
> > storage.
> > > >
> > > > Our setup:
> > > > Index size: 8GB (grows by 2GB every year)
> > > > Lucene: 4.2.1
> > > > 64-bit Java
> > > >
> > > > The options we considered:
> > > > * Each server instance,hosting a nightly job to pull delta of data
> > from
> > > > db. But, this would result in high DB load. (4 severs =4 times the
> > load)
> > > >
> > > > * An additional nightly job sitting on another sever, that pushes
> the
> > > data
> > > > on to local disks of each instances..This may not work out as the
> local
> > > > disk may not be visible.
> > > >
> > > > * each sever hosting a replication job that pulls delta of data from
> > NFS
> > > > and stores in the local index...so far this is the only promising
> > option
> > > we
> > > > have.
> > > >
> > > > * Does solr an option for us in this case? (I know it's a question
> for
> > > solr
> > > > group..but experts here might have some thoughts..)..
> > > >
> > > > Thank you for your attention.
> > > >
> > >
> >
>
Re: Index replication strategy
Posted by Michael Sokolov <ms...@safaribooksonline.com>.
There are also Solr replication options - older snapshot-style
replication, and newer Solr Cloud, but if you are not using solr now,
you will incur some transitional costs since you would need to alter
your indexing and possibly querying code to use it
-Mike
On 12/04/2014 09:38 AM, Shai Erera wrote:
> Ooops, didn't notice that :).
>
> So you'll need to upgrade to Lucene 4.4.0 in order to use it. You can read
> some details as well as example code here:
> http://shaierera.blogspot.com/2013/05/the-replicator.html.
>
> Shai
>
> On Thu, Dec 4, 2014 at 4:36 PM, Vijay B <vi...@gmail.com> wrote:
>
>> As indicated in my post, we use Lucene 4.2.1.
>>
>> On Thu, Dec 4, 2014 at 9:29 AM, Shai Erera <se...@gmail.com> wrote:
>>
>>> Do you use Lucene or Solr? Lucene also has a replication module, which
>> will
>>> allow you to replicate index changes.
>>>
>>> On Thu, Dec 4, 2014 at 4:19 PM, Vijay B <vi...@gmail.com> wrote:
>>>
>>>> Hello,
>>>>
>>>> We index docs coming from database nightly. Current index is sitting on
>>>> NFS. Due to obvious performance reasons, we are switching are planning
>> to
>>>> switch to local index. W have cluster of 4 servers and with NFS it was
>>> not
>>>> a problem for us until now to share the index. but going forward, we
>> are
>>>> looking for our design options for index replication on to local
>> storage.
>>>> Our setup:
>>>> Index size: 8GB (grows by 2GB every year)
>>>> Lucene: 4.2.1
>>>> 64-bit Java
>>>>
>>>> The options we considered:
>>>> * Each server instance,hosting a nightly job to pull delta of data
>> from
>>>> db. But, this would result in high DB load. (4 severs =4 times the
>> load)
>>>> * An additional nightly job sitting on another sever, that pushes the
>>> data
>>>> on to local disks of each instances..This may not work out as the local
>>>> disk may not be visible.
>>>>
>>>> * each sever hosting a replication job that pulls delta of data from
>> NFS
>>>> and stores in the local index...so far this is the only promising
>> option
>>> we
>>>> have.
>>>>
>>>> * Does solr an option for us in this case? (I know it's a question for
>>> solr
>>>> group..but experts here might have some thoughts..)..
>>>>
>>>> Thank you for your attention.
>>>>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Index replication strategy
Posted by Shai Erera <se...@gmail.com>.
Ooops, didn't notice that :).
So you'll need to upgrade to Lucene 4.4.0 in order to use it. You can read
some details as well as example code here:
http://shaierera.blogspot.com/2013/05/the-replicator.html.
Shai
On Thu, Dec 4, 2014 at 4:36 PM, Vijay B <vi...@gmail.com> wrote:
> As indicated in my post, we use Lucene 4.2.1.
>
> On Thu, Dec 4, 2014 at 9:29 AM, Shai Erera <se...@gmail.com> wrote:
>
> > Do you use Lucene or Solr? Lucene also has a replication module, which
> will
> > allow you to replicate index changes.
> >
> > On Thu, Dec 4, 2014 at 4:19 PM, Vijay B <vi...@gmail.com> wrote:
> >
> > > Hello,
> > >
> > > We index docs coming from database nightly. Current index is sitting on
> > > NFS. Due to obvious performance reasons, we are switching are planning
> to
> > > switch to local index. W have cluster of 4 servers and with NFS it was
> > not
> > > a problem for us until now to share the index. but going forward, we
> are
> > > looking for our design options for index replication on to local
> storage.
> > >
> > > Our setup:
> > > Index size: 8GB (grows by 2GB every year)
> > > Lucene: 4.2.1
> > > 64-bit Java
> > >
> > > The options we considered:
> > > * Each server instance,hosting a nightly job to pull delta of data
> from
> > > db. But, this would result in high DB load. (4 severs =4 times the
> load)
> > >
> > > * An additional nightly job sitting on another sever, that pushes the
> > data
> > > on to local disks of each instances..This may not work out as the local
> > > disk may not be visible.
> > >
> > > * each sever hosting a replication job that pulls delta of data from
> NFS
> > > and stores in the local index...so far this is the only promising
> option
> > we
> > > have.
> > >
> > > * Does solr an option for us in this case? (I know it's a question for
> > solr
> > > group..but experts here might have some thoughts..)..
> > >
> > > Thank you for your attention.
> > >
> >
>
Re: Index replication strategy
Posted by Vijay B <vi...@gmail.com>.
As indicated in my post, we use Lucene 4.2.1.
On Thu, Dec 4, 2014 at 9:29 AM, Shai Erera <se...@gmail.com> wrote:
> Do you use Lucene or Solr? Lucene also has a replication module, which will
> allow you to replicate index changes.
>
> On Thu, Dec 4, 2014 at 4:19 PM, Vijay B <vi...@gmail.com> wrote:
>
> > Hello,
> >
> > We index docs coming from database nightly. Current index is sitting on
> > NFS. Due to obvious performance reasons, we are switching are planning to
> > switch to local index. W have cluster of 4 servers and with NFS it was
> not
> > a problem for us until now to share the index. but going forward, we are
> > looking for our design options for index replication on to local storage.
> >
> > Our setup:
> > Index size: 8GB (grows by 2GB every year)
> > Lucene: 4.2.1
> > 64-bit Java
> >
> > The options we considered:
> > * Each server instance,hosting a nightly job to pull delta of data from
> > db. But, this would result in high DB load. (4 severs =4 times the load)
> >
> > * An additional nightly job sitting on another sever, that pushes the
> data
> > on to local disks of each instances..This may not work out as the local
> > disk may not be visible.
> >
> > * each sever hosting a replication job that pulls delta of data from NFS
> > and stores in the local index...so far this is the only promising option
> we
> > have.
> >
> > * Does solr an option for us in this case? (I know it's a question for
> solr
> > group..but experts here might have some thoughts..)..
> >
> > Thank you for your attention.
> >
>
Re: Index replication strategy
Posted by Shai Erera <se...@gmail.com>.
Do you use Lucene or Solr? Lucene also has a replication module, which will
allow you to replicate index changes.
On Thu, Dec 4, 2014 at 4:19 PM, Vijay B <vi...@gmail.com> wrote:
> Hello,
>
> We index docs coming from database nightly. Current index is sitting on
> NFS. Due to obvious performance reasons, we are switching are planning to
> switch to local index. W have cluster of 4 servers and with NFS it was not
> a problem for us until now to share the index. but going forward, we are
> looking for our design options for index replication on to local storage.
>
> Our setup:
> Index size: 8GB (grows by 2GB every year)
> Lucene: 4.2.1
> 64-bit Java
>
> The options we considered:
> * Each server instance,hosting a nightly job to pull delta of data from
> db. But, this would result in high DB load. (4 severs =4 times the load)
>
> * An additional nightly job sitting on another sever, that pushes the data
> on to local disks of each instances..This may not work out as the local
> disk may not be visible.
>
> * each sever hosting a replication job that pulls delta of data from NFS
> and stores in the local index...so far this is the only promising option we
> have.
>
> * Does solr an option for us in this case? (I know it's a question for solr
> group..but experts here might have some thoughts..)..
>
> Thank you for your attention.
>