You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by "Wu, Daniel" <Da...@sonyconnect.com> on 2007/09/15 01:25:30 UTC
Triggering snapshooter through web admin interface
Hi,
I am not sure if this can be done. Let's say if periodically there is a
big batch to be indexed and we don't want to replicate the data befor
the batch is completely indexed. We would like to avoid post commit
hook as we will be periodically committing to reduce the memory usage
and we also would like to avoid post optimize hook as optimizing causes
the replication of new big optimized file every time. Is there a way to
trigger snap shooting through Solr's admin interface?
For that matter, is there a generic way to trigger the Solr server to
execute some pre-defined scripts? I know it can be achieved with some
ssh scripts. It seems more elegant to be able to do so through the
admin interface so the client doesn't need to know the physical location
of the Solr installation.
Daniel
Re: Triggering snapshooter through web admin interface
Posted by Bill Au <bi...@gmail.com>.
Lance, do start a new thread if you run into this problem again and please
include as much info as possible.
Once a snapshot has been taken, the files it contains should not change so I
am not sure why tar was telling you
a file had changed while it was being copied.
Bill
On 9/19/07, Chris Hostetter <ho...@fucit.org> wrote:
>
>
> lance: since the topic you are describing is not directly related to
> triggering a snapshot from the web interface can you please start a new
> thread with a unique subejct describing in more details exactly what it
> was you were doing and the problem you encountered?
>
> this will make it easier for your problem to get visibility (some people
> don't read every thread, and archive searching is frequently done by
> thread, so people looking for similar problems may not realize this new
> thread is burried inside an old one)
>
> -Hoss
>
> : Date: Wed, 19 Sep 2007 11:33:30 -0700
> : From: Lance Norskog <go...@gmail.com>
> : Reply-To: solr-user@lucene.apache.org
> : To: solr-user@lucene.apache.org
> : Subject: RE: Triggering snapshooter through web admin interface
> :
> : Is there a ticket for this yet? I have a bug report and request: I just
> did
> : a snapshot while indexing 700 records/sec. and got an inconsistency. I
> was
> : tarring off the snapshot and tar reported that a file changed while it
> was
> : being copied. The error rolled off my screen, so I cannot report the
> file
> : name or extension.
> :
> : If a solr command to do a snapshot is implemented, please make sure that
> it
> : is 100% consistent.
> :
> : Thanks,
> :
> : Lance Norskog
>
>
RE: Triggering snapshooter through web admin interface
Posted by Chris Hostetter <ho...@fucit.org>.
lance: since the topic you are describing is not directly related to
triggering a snapshot from the web interface can you please start a new
thread with a unique subejct describing in more details exactly what it
was you were doing and the problem you encountered?
this will make it easier for your problem to get visibility (some people
don't read every thread, and archive searching is frequently done by
thread, so people looking for similar problems may not realize this new
thread is burried inside an old one)
-Hoss
: Date: Wed, 19 Sep 2007 11:33:30 -0700
: From: Lance Norskog <go...@gmail.com>
: Reply-To: solr-user@lucene.apache.org
: To: solr-user@lucene.apache.org
: Subject: RE: Triggering snapshooter through web admin interface
:
: Is there a ticket for this yet? I have a bug report and request: I just did
: a snapshot while indexing 700 records/sec. and got an inconsistency. I was
: tarring off the snapshot and tar reported that a file changed while it was
: being copied. The error rolled off my screen, so I cannot report the file
: name or extension.
:
: If a solr command to do a snapshot is implemented, please make sure that it
: is 100% consistent.
:
: Thanks,
:
: Lance Norskog
RE: Triggering snapshooter through web admin interface
Posted by Lance Norskog <go...@gmail.com>.
Is there a ticket for this yet? I have a bug report and request: I just did
a snapshot while indexing 700 records/sec. and got an inconsistency. I was
tarring off the snapshot and tar reported that a file changed while it was
being copied. The error rolled off my screen, so I cannot report the file
name or extension.
If a solr command to do a snapshot is implemented, please make sure that it
is 100% consistent.
Thanks,
Lance Norskog
-----Original Message-----
From: Chris Hostetter [mailto:hossman_lucene@fucit.org]
Sent: Tuesday, September 18, 2007 11:11 AM
To: solr-user@lucene.apache.org
Subject: RE: Triggering snapshooter through web admin interface
: [Wu, Daniel] That sounds great. Do I need to create a JIRA ticket?
Sure, JIRA is a great way to track feature requests (since they can be
"watched" and "voted" on, and if you want to start on an implementation you
can attach patches...
http://wiki.apache.org/solr/HowToContribute
-Hoss
RE: Triggering snapshooter through web admin interface
Posted by Chris Hostetter <ho...@fucit.org>.
: [Wu, Daniel] That sounds great. Do I need to create a JIRA ticket?
Sure, JIRA is a great way to track feature requests (since they can be
"watched" and "voted" on, and if you want to start on an implementation
you can attach patches...
http://wiki.apache.org/solr/HowToContribute
-Hoss
RE: Triggering snapshooter through web admin interface
Posted by "Wu, Daniel" <Da...@sonyconnect.com>.
> -----Original Message-----
> From: Chris Hostetter [mailto:hossman_lucene@fucit.org]
> Sent: Monday, September 17, 2007 1:28 PM
> To: solr-user@lucene.apache.org
> Subject: RE: Triggering snapshooter through web admin interface
>
>
> : I was also suggesting a new feature to allow sending messages to
Solr
> : through http interface and a mechanism to handling the message on
the
> : Solr server; in this case, a message to trigger snapshooter script.
It
> : seems to me, a very useful feature to help simplify operational
issues.
>
> it's been a while since i looked at the SolrEventListener stuff, but i
> think that would be pretty easy to develop as a plugin.
>
> The existing postCommit/postOptimizefirstSearcher/newSearcher event
> listener tracking are part of hte SolrCore because it needs to know
about
> them when managing the index ... but if you just wanted a way to
trigger
> arbitrary events by name, the utility functions used in SolrCore could
be
> reused by a custom plugin ... then you could reuse things like the
> RunExecutableListener from your own RequestHandler with the same
> solrconfig.xml syntax.
>
> that would be a pretty cool addition to Solr ... an
"EventRequestHandler"
> that takes in a single "event" param and triggers all of the Listeners
> configured for that even in the solrconfig.xml
>
>
> -Hoss
>
[Wu, Daniel] That sounds great. Do I need to create a JIRA ticket?
RE: Triggering snapshooter through web admin interface
Posted by Chris Hostetter <ho...@fucit.org>.
: I was also suggesting a new feature to allow sending messages to Solr
: through http interface and a mechanism to handling the message on the
: Solr server; in this case, a message to trigger snapshooter script. It
: seems to me, a very useful feature to help simplify operational issues.
it's been a while since i looked at the SolrEventListener stuff, but i
think that would be pretty easy to develop as a plugin.
The existing postCommit/postOptimizefirstSearcher/newSearcher event
listener tracking are part of hte SolrCore because it needs to know about
them when managing the index ... but if you just wanted a way to trigger
arbitrary events by name, the utility functions used in SolrCore could be
reused by a custom plugin ... then you could reuse things like the
RunExecutableListener from your own RequestHandler with the same
solrconfig.xml syntax.
that would be a pretty cool addition to Solr ... an "EventRequestHandler"
that takes in a single "event" param and triggers all of the Listeners
configured for that even in the solrconfig.xml
-Hoss
RE: Triggering snapshooter through web admin interface
Posted by "Wu, Daniel" <Da...@sonyconnect.com>.
> There is no way to trigger snapshots taking through Solr's admin
interface
> now. Taking a snapshot is a very light-weight operation. It uses
hard
> links so each snapshot doesn't take up much additional disk space. If
you
[Wu, Daniel]
It is not a concern on the snapshot performance. Rather, it is the data
consistency requirement.
I was also suggesting a new feature to allow sending messages to Solr
through http interface and a mechanism to handling the message on the
Solr server; in this case, a message to trigger snapshooter script. It
seems to me, a very useful feature to help simplify operational issues.
> don't want to replicate your index while the big batch job is still
> running,
> you can disable snappuller on the slave while the batch job is running
and
> enable it after the batch job has completed.
>
[Wu, Daniel]
Yes, it can be done that way but will not be as elegant. The Solr
master can be well-known among the applications; however, the slaves
could be anywhere. To turn off snapulling requires the knowledge of all
the Solr slaves as well as timing of the indexing. It gets ugly when
there are multiple environments (e.g. dev, qa, stage, production) and
multiple indexes.
Re: Triggering snapshooter through web admin interface
Posted by Bill Au <bi...@gmail.com>.
There is no way to trigger snapshots taking through Solr's admin interface
now. Taking a snapshot is a very light-weight operation. It uses hard
links so each snapshot doesn't take up much additional disk space. If you
don't want to replicate your index while the big batch job is still running,
you can disable snappuller on the slave while the batch job is running and
enable it after the batch job has completed.
Bill
On 9/14/07, Wu, Daniel <Da...@sonyconnect.com> wrote:
>
> Hi,
>
>
>
> I am not sure if this can be done. Let's say if periodically there is a
> big batch to be indexed and we don't want to replicate the data befor
> the batch is completely indexed. We would like to avoid post commit
> hook as we will be periodically committing to reduce the memory usage
> and we also would like to avoid post optimize hook as optimizing causes
> the replication of new big optimized file every time. Is there a way to
> trigger snap shooting through Solr's admin interface?
>
>
>
> For that matter, is there a generic way to trigger the Solr server to
> execute some pre-defined scripts? I know it can be achieved with some
> ssh scripts. It seems more elegant to be able to do so through the
> admin interface so the client doesn't need to know the physical location
> of the Solr installation.
>
>
>
> Daniel
>
>