You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Roberto Franchini <ro...@gmail.com> on 2014/03/13 17:21:46 UTC
Replicator: how to use it?
Hi to all,
I'm trying to use the Replicator
(http://lucene.apache.org/core/4_7_0/replicator/index.html?org/apache/lucene/replicator/package-summary.html)
to backup my indexes while indexing.
We have a lot of writer opened at the same time committed every 5 minutes.
So for each writer I associated a localreplicator:
writer = new IndexWriter(directory, config);
replicator = new LocalReplicator();
then, when I do a commit
writer.commit();
replicator.publish(new IndexRevision(writer));
On the other side, I've a service running on a separate thread that
for each IndexDir creates this:
private final List<ReplicationClient> replicators;
public void initialize() {
Directory dir = getDirFromConf) ; //internal
Replicator replicator = new LocalReplicator();
Callable<Boolean> callback = null;
ReplicationHandler handler = new IndexReplicationHandler(dir, callback);
File workDir = new File("/mnt/storage/backup/" + card.getId());
SourceDirectoryFactory factory = new PerSessionDirectoryFactory(workDir);
ReplicationClient client = new ReplicationClient(replicator, handler, factory);
replicators.add(client); //a list of client
}
Then it is called by a schedule to to replication:
public void doBackup() {
for (ReplicationClient client : replicators) {
log.info("activating client:: " + client);
client.updateNow();
}
}
This is not working.
I'm doing it wrong, but I don't know where.
The java doc and the old blog abut replicator aren't helping me.
Regards,
RF
--
Roberto Franchini
"L'impossibile è inevitabile"
jabber:ro.franchini@gmail.com skype:ro.franchini
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Replicator: how to use it?
Posted by Roberto Franchini <fr...@celi.it>.
On Thu, Mar 20, 2014 at 8:47 AM, Shai Erera <se...@gmail.com> wrote:
>>
>> Even if the commit is called just before the close, the close triggers
>> a last commit.
>>
>
> That seems wrong. If you do writer.commit() and them immediately
> writer.close(), and there are no changes to the writer in between (i.e. a
> thread comes in and adds/updates/deletes a document), then close() should
> not create a new commit point. Do you see that it does?
Hi,
I've added a waitForMerges call before the last commit, then I close the writer:
writer.waitForMerges();
writer.commit();
replicator. replicate(new IndexRevision(writer));
writer.close();
Now it works.
Thank you very much for the support.
RF
--
Roberto Franchini
The impossible is inevitable.
http://www.celi.it http://www.blogmeter.it
http://github.com/celi-uim http://github.com/robfrank
Tel +39.011.562.71.15
jabber:ro.franchini@gmail.com skype:ro.franchini tw:@robfrankie
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Replicator: how to use it?
Posted by Shai Erera <se...@gmail.com>.
>
> Even if the commit is called just before the close, the close triggers
> a last commit.
>
That seems wrong. If you do writer.commit() and them immediately
writer.close(), and there are no changes to the writer in between (i.e. a
thread comes in and adds/updates/deletes a document), then close() should
not create a new commit point. Do you see that it does?
Shai
On Wed, Mar 19, 2014 at 11:09 PM, Roberto Franchini <fr...@celi.it>wrote:
> On Sat, Mar 15, 2014 at 12:56 PM, Roberto Franchini <fr...@celi.it>
> wrote:
> > On Sat, Mar 15, 2014 at 12:47 PM, Shai Erera <se...@gmail.com> wrote:
> >> If you use LocalReplicator on both sides, you have to use the same
> instance
> >> on both sides. Otherwise the replicas will never see the published
> >> revisions the which are done in a separate instance. Can you try that?
> >
> > Ok, I missed it. I was using different instances.
> > I'll try this afternoon.
>
> Hi,
> the replicator works fine on live writer, but when the writer is
> closed it does a last commit that isn't replicated.
>
> Even if the commit is called just before the close, the close triggers
> a last commit.
>
> And trying to use the writer after close is impossible:
>
> writer.close();
> revision= new IndexRevision(writer);
>
> produce:
> org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed
> ....
>
> So, I can replicate the last commit before the close, and don't worry
> about the inner commit that close does.
> May I'll lost something?
> RF
>
> --
> Roberto Franchini
> The impossible is inevitable.
> http://www.celi.it http://www.blogmeter.it
> http://github.com/celi-uim http://github.com/robfrank
> Tel +39.011.562.71.15
> jabber:ro.franchini@gmail.com skype:ro.franchini tw:@robfrankie
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
Re: Replicator: how to use it?
Posted by Roberto Franchini <fr...@celi.it>.
On Sat, Mar 15, 2014 at 12:56 PM, Roberto Franchini <fr...@celi.it> wrote:
> On Sat, Mar 15, 2014 at 12:47 PM, Shai Erera <se...@gmail.com> wrote:
>> If you use LocalReplicator on both sides, you have to use the same instance
>> on both sides. Otherwise the replicas will never see the published
>> revisions the which are done in a separate instance. Can you try that?
>
> Ok, I missed it. I was using different instances.
> I'll try this afternoon.
Hi,
the replicator works fine on live writer, but when the writer is
closed it does a last commit that isn't replicated.
Even if the commit is called just before the close, the close triggers
a last commit.
And trying to use the writer after close is impossible:
writer.close();
revision= new IndexRevision(writer);
produce:
org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed
....
So, I can replicate the last commit before the close, and don't worry
about the inner commit that close does.
May I'll lost something?
RF
--
Roberto Franchini
The impossible is inevitable.
http://www.celi.it http://www.blogmeter.it
http://github.com/celi-uim http://github.com/robfrank
Tel +39.011.562.71.15
jabber:ro.franchini@gmail.com skype:ro.franchini tw:@robfrankie
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Replicator: how to use it?
Posted by Roberto Franchini <fr...@celi.it>.
On Sat, Mar 15, 2014 at 12:47 PM, Shai Erera <se...@gmail.com> wrote:
> If you use LocalReplicator on both sides, you have to use the same instance
> on both sides. Otherwise the replicas will never see the published
> revisions the which are done in a separate instance. Can you try that?
Ok, I missed it. I was using different instances.
I'll try this afternoon.
thank you.
RF
--
Roberto Franchini
The impossible is inevitable.
http://www.celi.it http://www.blogmeter.it
http://github.com/celi-uim http://github.com/robfrank
Tel +39.011.562.71.15
jabber:ro.franchini@gmail.com skype:ro.franchini tw:@robfrankie
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Replicator: how to use it?
Posted by Shai Erera <se...@gmail.com>.
If you use LocalReplicator on both sides, you have to use the same instance
on both sides. Otherwise the replicas will never see the published
revisions the which are done in a separate instance. Can you try that?
Shai
On Mar 15, 2014 1:10 PM, "Roberto Franchini" <fr...@celi.it> wrote:
> On Sat, Mar 15, 2014 at 11:58 AM, Michael McCandless
> <lu...@mikemccandless.com> wrote:
> > I think maybe the problem is you are using LocalReplicator on the
> > replicas? I think you should only use that on the master. I think
> > e.g. you should use HttpReplicator on the clients? Or, your own
> > implementation that moves the files its own way.
> >
> > Have you seen Shai's blog post about this?
> > http://shaierera.blogspot.com/2013/05/the-replicator.html
>
> Yes, I've seen it. I checkout the replicator code and looked at test code.
>
> I'm trying to use the local replicator because, as a first step, I
> want only to incrementally backup indexes.
>
> So I've implemented a sort of producer/consumer where the indexer is
> the producer, it runs on it's own thread and publish revisions, and
> the consumer will be the replicator client that's on it's onw thread.
>
> Code samples aren't, at least for me, very clear in how to use the
> replicator.
> So, if someone has a clean sample of use of replicator I would appreciate
> it.
>
> REgards,
> RF
>
> --
> Roberto Franchini
> The impossible is inevitable.
> http://www.celi.it http://www.blogmeter.it
> http://github.com/celi-uim http://github.com/robfrank
> Tel +39.011.562.71.15
> jabber:ro.franchini@gmail.com skype:ro.franchini tw:@robfrankie
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
Re: Replicator: how to use it?
Posted by Roberto Franchini <fr...@celi.it>.
On Sat, Mar 15, 2014 at 11:58 AM, Michael McCandless
<lu...@mikemccandless.com> wrote:
> I think maybe the problem is you are using LocalReplicator on the
> replicas? I think you should only use that on the master. I think
> e.g. you should use HttpReplicator on the clients? Or, your own
> implementation that moves the files its own way.
>
> Have you seen Shai's blog post about this?
> http://shaierera.blogspot.com/2013/05/the-replicator.html
Yes, I've seen it. I checkout the replicator code and looked at test code.
I'm trying to use the local replicator because, as a first step, I
want only to incrementally backup indexes.
So I've implemented a sort of producer/consumer where the indexer is
the producer, it runs on it's own thread and publish revisions, and
the consumer will be the replicator client that's on it's onw thread.
Code samples aren't, at least for me, very clear in how to use the replicator.
So, if someone has a clean sample of use of replicator I would appreciate it.
REgards,
RF
--
Roberto Franchini
The impossible is inevitable.
http://www.celi.it http://www.blogmeter.it
http://github.com/celi-uim http://github.com/robfrank
Tel +39.011.562.71.15
jabber:ro.franchini@gmail.com skype:ro.franchini tw:@robfrankie
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Replicator: how to use it?
Posted by Michael McCandless <lu...@mikemccandless.com>.
I think maybe the problem is you are using LocalReplicator on the
replicas? I think you should only use that on the master. I think
e.g. you should use HttpReplicator on the clients? Or, your own
implementation that moves the files its own way.
Have you seen Shai's blog post about this?
http://shaierera.blogspot.com/2013/05/the-replicator.html
Mike McCandless
http://blog.mikemccandless.com
On Thu, Mar 13, 2014 at 12:21 PM, Roberto Franchini
<ro...@gmail.com> wrote:
> Hi to all,
> I'm trying to use the Replicator
> (http://lucene.apache.org/core/4_7_0/replicator/index.html?org/apache/lucene/replicator/package-summary.html)
> to backup my indexes while indexing.
>
>
> We have a lot of writer opened at the same time committed every 5 minutes.
>
> So for each writer I associated a localreplicator:
>
> writer = new IndexWriter(directory, config);
> replicator = new LocalReplicator();
>
> then, when I do a commit
>
> writer.commit();
> replicator.publish(new IndexRevision(writer));
>
> On the other side, I've a service running on a separate thread that
> for each IndexDir creates this:
>
> private final List<ReplicationClient> replicators;
>
>
> public void initialize() {
> Directory dir = getDirFromConf) ; //internal
> Replicator replicator = new LocalReplicator();
> Callable<Boolean> callback = null;
> ReplicationHandler handler = new IndexReplicationHandler(dir, callback);
>
> File workDir = new File("/mnt/storage/backup/" + card.getId());
> SourceDirectoryFactory factory = new PerSessionDirectoryFactory(workDir);
> ReplicationClient client = new ReplicationClient(replicator, handler, factory);
>
> replicators.add(client); //a list of client
> }
>
> Then it is called by a schedule to to replication:
>
> public void doBackup() {
> for (ReplicationClient client : replicators) {
> log.info("activating client:: " + client);
> client.updateNow();
> }
> }
>
> This is not working.
> I'm doing it wrong, but I don't know where.
> The java doc and the old blog abut replicator aren't helping me.
>
> Regards,
> RF
>
> --
> Roberto Franchini
> "L'impossibile è inevitabile"
> jabber:ro.franchini@gmail.com skype:ro.franchini
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org