You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by Chun Wei Ho <cw...@gmail.com> on 2006/01/27 05:03:13 UTC
Updating the search index
Hi,
We are running a nutch crawler on one machine and a web search
application on searching an index using NutchBean on another.
Periodically we would like to copy the updated crawl index from the
crawl machine to replace the one used by the search application,
without resulting in any broken queries or search application
downtime.
Could anyone give us some pointers on how they are doing this or
setting it up? Thanks :)
Re: Updating the search index
Posted by Raghavendra Prabhu <rr...@gmail.com>.
Hi,
The present system as far as i know does not allow this i guess
If somebody else is also facing the problem .can you tell us how you get
around this issue
Rgds
Prabhu
On 1/27/06, Chun Wei Ho <cw...@gmail.com> wrote:
>
> Hi,
>
> We are running a nutch crawler on one machine and a web search
> application on searching an index using NutchBean on another.
> Periodically we would like to copy the updated crawl index from the
> crawl machine to replace the one used by the search application,
> without resulting in any broken queries or search application
> downtime.
>
> Could anyone give us some pointers on how they are doing this or
> setting it up? Thanks :)
>
Connecting the search to the Db
Posted by Lawrence W Thorne <lt...@thornedigital.com>.
I hope I am emailing the correct address, if not, I apologize.
I am installing on Windows Server 2000 SP4 and am willing to produce a
detailed (with installation notes) installation/setup document for windows
(PDF even) in exchange for your help with this issue.
Windows Nutch testing server (do search to see error): the lang/docs all
work fine.
http://66.65.133.195/
I probably have my Nutch unpackaged in the wrong place I'm sure. No problems
running Nutch from command line, the Db and Segment builds go fine -
importing DMOZ XML feed -3000 - indexing seems fine as well - no errors.
I'm going step by step through your tutorial, no errors using cygwin, but
when I attempt to search from above link I get 500 server error.
!--BEGIN ERROR
javax.servlet.ServletException
org.apache.jasper.servlet.JspServlet.service(JspServlet.java:259)
javax.servlet.http.HttpServlet.service(HttpServlet.java:860)
sun.reflect.GeneratedMethodAccessor57.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl
.java:25)
java.lang.reflect.Method.invoke(Method.java:585)
org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:249)
java.security.AccessController.doPrivileged(Native Method)
javax.security.auth.Subject.doAsPrivileged(Subject.java:517)
org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:282)
org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:16
5)
root cause
java.lang.NoClassDefFoundError
org.apache.jsp.search_jsp.(search_jsp.java:29)
sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAcces
sorImpl.java:39)
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstruc
torAccessorImpl.java:27)
java.lang.reflect.Constructor.newInstance(Constructor.java:494)
java.lang.Class.newInstance0(Class.java:350)
java.lang.Class.newInstance(Class.java:303)
org.apache.jasper.servlet.JspServletWrapper.getServlet(JspServletWrapper.jav
a:143)
org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:3
19)
org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:301)
org.apache.jasper.servlet.JspServlet.service(JspServlet.java:251)
javax.servlet.http.HttpServlet.service(HttpServlet.java:860)
sun.reflect.GeneratedMethodAccessor57.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl
.java:25)
java.lang.reflect.Method.invoke(Method.java:585)
org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:249)
java.security.AccessController.doPrivileged(Native Method)
javax.security.auth.Subject.doAsPrivileged(Subject.java:517)
org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:282)
org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:16
5)
!--END ERROR
I am new to Java but have worked extensively with VB, VB.NET PHP/MySQL and
Perl - and am really excited about getting nutch up and running.
Back to CYGWIN session for bin/nutch commands in tutorial - to perhaps shed
some light. (see session below)
Administrator@nythorne02 /cygdrive/d/Nutch/nutch-0.7.1 $
/cygdrive/c/Program\ Files/Apache\ Group/Tomcat\ 4.1/bin/catalina.sh start
Using CATALINA_BASE: c:\Program Files\Apache Group\Tomcat 4.1
Using CATALINA_HOME: c:\Program Files\Apache Group\Tomcat 4.1
Using CATALINA_TMPDIR: c:\Program Files\Apache Group\Tomcat 4.1\temp
Using JAVA_HOME: c:\Program Files\Java\jdk1.5.0_06\
As you can see I have Tomcat installed default on C.
Sun/Nutch are installed/unpackaged to the larger drive at
D:\Sun\AppServer\domains\domain1\ and D:\Nutch\nutch-0.7.1\ respectively.
Do I need to have the nutch db and segments folders indexed inside
D:\Sun\AppServer\domains\domain1\applications\j2ee-modules\APPNAME ??
It is my understanding that as long as I initialize Catalina from the
directory above my segments and db dir, it will use that instance of the
indexed data?
This is the config I see from the Java Web Console:
${com.sun.aas.instanceRoot}/applications/j2ee-modules/nutch-0.7.1 which
translates to D:\Sun\AppServer\domains\domain1\ which leads me to believe I
need to run my nutch db and segments indices from inside that folder?
any help would be much appreciated.
-Lawrence Thorne
lthorne@thornedigital.com
Re: Updating the search index
Posted by Raghavendra Prabhu <rr...@gmail.com>.
I think the one that goes into memory are file handlers and other stuff
And everything is not loaded into memory i guess . Even hard disk access is
done during search
Can some one confirm this?
Rgds
Prabhu
On 1/28/06, Sunnyvale Fl <su...@gmail.com> wrote:
>
> We are facing the same problem here. For now, our solution is having the
> crawler notify the searcher whenever there is an updated segment, and the
> searcher gets a new bean when this happens. Now my question is, does the
> searcher load everything into memory, including segments, on startup?
> Because it seems that if I delete segments or replace them while the
> searcher is running, it doesn't affect the search result. But if it does
> load everything into memory, isn't that a LOT of memory we are talking
> about?
>
> On 1/27/06, Stefan Groschupf <sg...@media-style.com> wrote:
> >
> > This is difficult today.
> > We had somehow hack the nutch bean to get this done. If I would have
> > more time I would clean up things and publish, but you know time. :-O
> > Anyway we had a seperated thread that is scanning the folder and
> > compare the result to the last scan, if something changes we created
> > a new search and detailer, runs one query against them to warm up and
> > than replace the searcher and detailer in the nutch bean.
> > I notice that this is a common need and I will put this in my todo
> > list on a little higher position. :) Anyway for now there is no
> > solution.
> > Stefan
> >
> > Am 27.01.2006 um 05:03 schrieb Chun Wei Ho:
> >
> > > Hi,
> > >
> > > We are running a nutch crawler on one machine and a web search
> > > application on searching an index using NutchBean on another.
> > > Periodically we would like to copy the updated crawl index from the
> > > crawl machine to replace the one used by the search application,
> > > without resulting in any broken queries or search application
> > > downtime.
> > >
> > > Could anyone give us some pointers on how they are doing this or
> > > setting it up? Thanks :)
> > >
> >
> >
>
>
Re: Updating the search index
Posted by Sunnyvale Fl <su...@gmail.com>.
We are facing the same problem here. For now, our solution is having the
crawler notify the searcher whenever there is an updated segment, and the
searcher gets a new bean when this happens. Now my question is, does the
searcher load everything into memory, including segments, on startup?
Because it seems that if I delete segments or replace them while the
searcher is running, it doesn't affect the search result. But if it does
load everything into memory, isn't that a LOT of memory we are talking
about?
On 1/27/06, Stefan Groschupf <sg...@media-style.com> wrote:
>
> This is difficult today.
> We had somehow hack the nutch bean to get this done. If I would have
> more time I would clean up things and publish, but you know time. :-O
> Anyway we had a seperated thread that is scanning the folder and
> compare the result to the last scan, if something changes we created
> a new search and detailer, runs one query against them to warm up and
> than replace the searcher and detailer in the nutch bean.
> I notice that this is a common need and I will put this in my todo
> list on a little higher position. :) Anyway for now there is no
> solution.
> Stefan
>
> Am 27.01.2006 um 05:03 schrieb Chun Wei Ho:
>
> > Hi,
> >
> > We are running a nutch crawler on one machine and a web search
> > application on searching an index using NutchBean on another.
> > Periodically we would like to copy the updated crawl index from the
> > crawl machine to replace the one used by the search application,
> > without resulting in any broken queries or search application
> > downtime.
> >
> > Could anyone give us some pointers on how they are doing this or
> > setting it up? Thanks :)
> >
>
>
Re: Updating the search index
Posted by Stefan Groschupf <sg...@media-style.com>.
This is difficult today.
We had somehow hack the nutch bean to get this done. If I would have
more time I would clean up things and publish, but you know time. :-O
Anyway we had a seperated thread that is scanning the folder and
compare the result to the last scan, if something changes we created
a new search and detailer, runs one query against them to warm up and
than replace the searcher and detailer in the nutch bean.
I notice that this is a common need and I will put this in my todo
list on a little higher position. :) Anyway for now there is no
solution.
Stefan
Am 27.01.2006 um 05:03 schrieb Chun Wei Ho:
> Hi,
>
> We are running a nutch crawler on one machine and a web search
> application on searching an index using NutchBean on another.
> Periodically we would like to copy the updated crawl index from the
> crawl machine to replace the one used by the search application,
> without resulting in any broken queries or search application
> downtime.
>
> Could anyone give us some pointers on how they are doing this or
> setting it up? Thanks :)
>