You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@cocoon.apache.org by Upayavira <uv...@upaya.co.uk> on 2003/03/18 00:47:48 UTC

Lucene index building

I have built a site which I want to index with Lucene.

I am using the create-index.xsp file in the $COCOON-ROOT/search directory to 
build my index.

I have added the following to cocoon.xconf:

  <cocoon-crawler logger="core.search.crawler">
    <exclude>.*/search/.*</exclude>
    <link-view-query>cocoon-view=lucene-links</link-view-query>
  </cocoon-crawler>

  <lucene-xml-indexer logger="core.search.lucene">
  <store-fields>body</store-fields>
    <content-view-query>cocoon-view=lucene-content</content-view-query>
  </lucene-xml-indexer>

I've set up a view lucene-links which works, giving back just links from a page.
I've set up a view lucene-content just giving back the content. The content is like:
<page>
  <links>....list of links</links>
  <body>... the body content ...</body>
</page>

I have had it partially working (indexing both links and body), but now whenever I 
run create-index, it fails with a Cannot parse!: org.xml.sax.SAXParseException: 
Premature end of file.

Any ideas what I might be doing wrong?

Regards, Upayavira


---------------------------------------------------------------------
To unsubscribe, e-mail: cocoon-users-unsubscribe@xml.apache.org
For additional commands, e-mail: cocoon-users-help@xml.apache.org


Re: Lucene index building

Posted by Upayavira <uv...@upaya.co.uk>.
Jeremy,

Ages ago you gave me excellent advice on getting the Lucene engine to work (i.e. tell 
it to skip images and CSS, etc).

There was one gem in your message that I didn't spot at the time:

> <body>
>  <title>title gets stored, then displayed with hit</title>
>  <summary>summary gets stored, then displayed with hit</summary>
>  all of my body content with tags stripped out
> </body>

I have just made the spidered XML look exactly like this. How do I then get Lucene to 
return the <title> and <summary> elements along with the search hits, so that I can 
display them?

Regards, Upayavira


---------------------------------------------------------------------
To unsubscribe, e-mail: cocoon-users-unsubscribe@xml.apache.org
For additional commands, e-mail: cocoon-users-help@xml.apache.org


Re: Lucene index building

Posted by Upayavira <uv...@upaya.co.uk>.
Jeremy,

> This all looks fine
> My exclude string looks like this though :
> 
> <exclude>.*\.png$,.*\.js$,.*\.css$,.*\.gif$,.*\.jpg$,.*/search/.*,.*/
> easy/.*</exclude>
> 
> I believe as soon as you specify an exclude string, the default values
>  for images etc. are not used.
 
You're a hero. I added a number of excludes, then checked my logs. At
present, there are a few broken links that I haven't yet been able to fix, I spotted by
looking into the logs, which broke it. So I added them to my excludes, ran it and 
waited, and works! Without changing the heap-size, which, given the number of 
errors in my site, probably wouldn't have fixed it anyway.

I've just uploaded it onto my test server, and it works a treat!

Thanks for that.

Regards, Upayavira


---------------------------------------------------------------------
To unsubscribe, e-mail: cocoon-users-unsubscribe@xml.apache.org
For additional commands, e-mail: cocoon-users-help@xml.apache.org


Lookup of PortalManager failed

Posted by JD Daniels <jd...@datatrio.com>.
I copied the portal-fw folder up to the cocoon root. After adding the portal
pipelines to the root sitemap, I get this when I try to save a profile
change (ie, personalize(guest)-->Customize-->Save)

My root sitemap is here:
http://dev.datatrio.com/tmp-junk/sitemap.xmap

Uri:
http://localhost/portal-fw/sunspotdemo-portal?portalprofile=uprofile:portalh
andler|sunspotdemo:user_5_guest_guest&portalcmd=save


Error.log entry:

ERROR   (2003-03-18) 01:14.54:703   [access]
(/portal-fw/sunspotdemo-portlets) HttpProcessor[80][10]/CocoonServlet:
Internal Cocoon Problem
org.apache.cocoon.ProcessingException: Lookup of PortalManager failed.:
org.apache.avalon.framework.component.ComponentException: Could not find
component (role [org.apache.cocoon.webapps.portal.components.PortalManager])
 at
org.apache.cocoon.webapps.portal.generation.PortalGenerator.generate(PortalG
enerator.java:86)
 at
org.apache.cocoon.components.pipeline.AbstractProcessingPipeline.processXMLP
ipeline(AbstractProcessingPipeline.java:545)
 at
org.apache.cocoon.components.pipeline.impl.AbstractCachingProcessingPipeline
.processXMLPipeline(AbstractCachingProcessingPipeline.java:214)
 at
org.apache.cocoon.components.pipeline.AbstractProcessingPipeline.process(Abs
tractProcessingPipeline.java:489)
 at
org.apache.cocoon.components.treeprocessor.sitemap.SerializeNode.invoke(Seri
alizeNode.java:145)
 at
org.apache.cocoon.components.treeprocessor.AbstractParentProcessingNode.invo
keNodes(AbstractParentProcessingNode.java:84)
 at
org.apache.cocoon.components.treeprocessor.sitemap.PreparableMatchNode.invok
e(PreparableMatchNode.java:164)
 at
org.apache.cocoon.components.treeprocessor.AbstractParentProcessingNode.invo
keNodes(AbstractParentProcessingNode.java:84)
 at
org.apache.cocoon.components.treeprocessor.sitemap.ActTypeNode.invoke(ActTyp
eNode.java:158)
 at
org.apache.cocoon.components.treeprocessor.AbstractParentProcessingNode.invo
keNodes(AbstractParentProcessingNode.java:84)
 at
org.apache.cocoon.components.treeprocessor.sitemap.PreparableMatchNode.invok
e(PreparableMatchNode.java:164)
 at
org.apache.cocoon.components.treeprocessor.AbstractParentProcessingNode.invo
keNodes(AbstractParentProcessingNode.java:108)
 at
org.apache.cocoon.components.treeprocessor.sitemap.PipelineNode.invoke(Pipel
ineNode.java:153)
 at
org.apache.cocoon.components.treeprocessor.AbstractParentProcessingNode.invo
keNodes(AbstractParentProcessingNode.java:108)
 at
org.apache.cocoon.components.treeprocessor.sitemap.PipelinesNode.invoke(Pipe
linesNode.java:143)
 at
org.apache.cocoon.components.treeprocessor.TreeProcessor.process(TreeProcess
or.java:317)
 at
org.apache.cocoon.components.treeprocessor.TreeProcessor.process(TreeProcess
or.java:299)
 at org.apache.cocoon.Cocoon.process(Cocoon.java:639)
 at org.apache.cocoon.servlet.CocoonServlet.service(CocoonServlet.java:1074)
 at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)
 at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Application
FilterChain.java:247)
 at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterCh
ain.java:193)
 at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.ja
va:260)
 at
org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invok
eNext(StandardPipeline.java:643)
 at
org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)
 at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
 at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.ja
va:191)
 at
org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invok
eNext(StandardPipeline.java:643)
 at
org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)
 at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
 at
org.apache.catalina.core.StandardContext.invoke(StandardContext.java:2415)
 at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:180
)
 at
org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invok
eNext(StandardPipeline.java:643)
 at
org.apache.catalina.valves.ErrorDispatcherValve.invoke(ErrorDispatcherValve.
java:170)
 at
org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invok
eNext(StandardPipeline.java:641)
 at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:172
)
 at
org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invok
eNext(StandardPipeline.java:641)
 at
org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:509)
 at
org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invok
eNext(StandardPipeline.java:641)
 at
org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)
 at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
 at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java
:174)
 at
org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invok
eNext(StandardPipeline.java:643)
 at
org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)
 at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
 at
org.apache.catalina.connector.http.HttpProcessor.process(HttpProcessor.java:
1040)
 at
org.apache.catalina.connector.http.HttpProcessor.run(HttpProcessor.java:1151
)
 at java.lang.Thread.run(Thread.java:536)
Caused by: org.apache.avalon.framework.component.ComponentException: Could
not find component (role
[org.apache.cocoon.webapps.portal.components.PortalManager])
 at
org.apache.avalon.excalibur.component.ExcaliburComponentManager.lookup(Excal
iburComponentManager.java:255)
 at
org.apache.cocoon.components.CocoonComponentManager.lookup(CocoonComponentMa
nager.java:296)
 at
org.apache.avalon.excalibur.component.DefaultComponentFactory$ComponentManag
erProxy.lookup(DefaultComponentFactory.java:393)
 at
org.apache.avalon.excalibur.component.DefaultComponentFactory$ComponentManag
erProxy.lookup(DefaultComponentFactory.java:393)
 at
org.apache.cocoon.webapps.portal.generation.PortalGenerator.generate(PortalG
enerator.java:77)
 ... 47 more
org.apache.avalon.framework.component.ComponentException: Could not find
component (role [org.apache.cocoon.webapps.portal.components.PortalManager])
 at
org.apache.avalon.excalibur.component.ExcaliburComponentManager.lookup(Excal
iburComponentManager.java:255)
 at
org.apache.cocoon.components.CocoonComponentManager.lookup(CocoonComponentMa
nager.java:296)
 at
org.apache.avalon.excalibur.component.DefaultComponentFactory$ComponentManag
erProxy.lookup(DefaultComponentFactory.java:393)
 at
org.apache.avalon.excalibur.component.DefaultComponentFactory$ComponentManag
erProxy.lookup(DefaultComponentFactory.java:393)
 at
org.apache.cocoon.webapps.portal.generation.PortalGenerator.generate(PortalG
enerator.java:77)
 at
org.apache.cocoon.components.pipeline.AbstractProcessingPipeline.processXMLP
ipeline(AbstractProcessingPipeline.java:545)
 at
org.apache.cocoon.components.pipeline.impl.AbstractCachingProcessingPipeline
.processXMLPipeline(AbstractCachingProcessingPipeline.java:214)
 at
org.apache.cocoon.components.pipeline.AbstractProcessingPipeline.process(Abs
tractProcessingPipeline.java:489)
 at
org.apache.cocoon.components.treeprocessor.sitemap.SerializeNode.invoke(Seri
alizeNode.java:145)
 at
org.apache.cocoon.components.treeprocessor.AbstractParentProcessingNode.invo
keNodes(AbstractParentProcessingNode.java:84)
 at
org.apache.cocoon.components.treeprocessor.sitemap.PreparableMatchNode.invok
e(PreparableMatchNode.java:164)
 at
org.apache.cocoon.components.treeprocessor.AbstractParentProcessingNode.invo
keNodes(AbstractParentProcessingNode.java:84)
 at
org.apache.cocoon.components.treeprocessor.sitemap.ActTypeNode.invoke(ActTyp
eNode.java:158)
 at
org.apache.cocoon.components.treeprocessor.AbstractParentProcessingNode.invo
keNodes(AbstractParentProcessingNode.java:84)
 at
org.apache.cocoon.components.treeprocessor.sitemap.PreparableMatchNode.invok
e(PreparableMatchNode.java:164)
 at
org.apache.cocoon.components.treeprocessor.AbstractParentProcessingNode.invo
keNodes(AbstractParentProcessingNode.java:108)
 at
org.apache.cocoon.components.treeprocessor.sitemap.PipelineNode.invoke(Pipel
ineNode.java:153)
 at
org.apache.cocoon.components.treeprocessor.AbstractParentProcessingNode.invo
keNodes(AbstractParentProcessingNode.java:108)
 at
org.apache.cocoon.components.treeprocessor.sitemap.PipelinesNode.invoke(Pipe
linesNode.java:143)
 at
org.apache.cocoon.components.treeprocessor.TreeProcessor.process(TreeProcess
or.java:317)
 at
org.apache.cocoon.components.treeprocessor.TreeProcessor.process(TreeProcess
or.java:299)
 at org.apache.cocoon.Cocoon.process(Cocoon.java:639)
 at org.apache.cocoon.servlet.CocoonServlet.service(CocoonServlet.java:1074)
 at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)
 at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Application
FilterChain.java:247)
 at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterCh
ain.java:193)
 at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.ja
va:260)
 at
org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invok
eNext(StandardPipeline.java:643)
 at
org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)
 at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
 at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.ja
va:191)
 at
org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invok
eNext(StandardPipeline.java:643)
 at
org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)
 at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
 at
org.apache.catalina.core.StandardContext.invoke(StandardContext.java:2415)
 at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:180
)
 at
org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invok
eNext(StandardPipeline.java:643)
 at
org.apache.catalina.valves.ErrorDispatcherValve.invoke(ErrorDispatcherValve.
java:170)
 at
org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invok
eNext(StandardPipeline.java:641)
 at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:172
)
 at
org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invok
eNext(StandardPipeline.java:641)
 at
org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:509)
 at
org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invok
eNext(StandardPipeline.java:641)
 at
org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)
 at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
 at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java
:174)
 at
org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invok
eNext(StandardPipeline.java:643)
 at
org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)
 at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
 at
org.apache.catalina.connector.http.HttpProcessor.process(HttpProcessor.java:
1040)
 at
org.apache.catalina.connector.http.HttpProcessor.run(HttpProcessor.java:1151
)
 at java.lang.Thread.run(Thread.java:536)


---------------------------------------------------------------------
To unsubscribe, e-mail: cocoon-users-unsubscribe@xml.apache.org
For additional commands, e-mail: cocoon-users-help@xml.apache.org


Re: Lucene index building

Posted by Olivier Billard <ob...@rennes.jouve.fr>.
Hi Upayavira,

It sounds like an error I also got. It seems that Lucene doesn't release
some Objects and keeps increasing memory consumption. And the end a parsing
error occurs.
My solution was to raise the memory heap size to the JVM with the
CATALINA_OPTS variable : "CATALINA_OPTS=-Xmx128m" (set 128 Mo of heap size,
the default value is 64 Mo).
Then to raise the default heap size in cocoon.xconf :

  <store-janitor class="org.apache.cocoon.components.store.StoreJanitorImpl"
logger="core.store.janitor">
    ...
     <!-- Indicates the limit of the jvm memory consumption. The default max
     heapsize for Sun's JVM is 64Mb -->
     <parameter name="heapsize" value="128000000"/>
<---------
    ...
  </store-janitor>

And it worked...

This turn-around worked also :

  <!-- Memory Storing: -->
  <cache-transient class="org.apache.cocoon.components.store.MRUMemoryStore"
logger="core.store.transient">
     <!-- Indicates how many objects will be hold in the cache.
     When the number of maxobjects has been reached. The last object in the
     cache will be thrown out.
     <parameter name="maxobjects" value="100"/>
     -->
     <parameter name="maxobjects" value="50"/>
    ...
  </cache-transient>

Or an other value of maxobjects. But there is a real decrease of
performance...


But is's just a "turn-around"... Not a solution.

Olivier


----- Original Message -----
From: "Upayavira" <uv...@upaya.co.uk>
To: <co...@xml.apache.org>
Sent: Tuesday, March 18, 2003 12:47 AM
Subject: Lucene index building


> I have built a site which I want to index with Lucene.
>
> I am using the create-index.xsp file in the $COCOON-ROOT/search directory
to
> build my index.
>
> I have added the following to cocoon.xconf:
>
>   <cocoon-crawler logger="core.search.crawler">
>     <exclude>.*/search/.*</exclude>
>     <link-view-query>cocoon-view=lucene-links</link-view-query>
>   </cocoon-crawler>
>
>   <lucene-xml-indexer logger="core.search.lucene">
>   <store-fields>body</store-fields>
>     <content-view-query>cocoon-view=lucene-content</content-view-query>
>   </lucene-xml-indexer>
>
> I've set up a view lucene-links which works, giving back just links from a
page.
> I've set up a view lucene-content just giving back the content. The
content is like:
> <page>
>   <links>....list of links</links>
>   <body>... the body content ...</body>
> </page>
>
> I have had it partially working (indexing both links and body), but now
whenever I
> run create-index, it fails with a Cannot parse!:
org.xml.sax.SAXParseException:
> Premature end of file.
>
> Any ideas what I might be doing wrong?
>
> Regards, Upayavira
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: cocoon-users-unsubscribe@xml.apache.org
> For additional commands, e-mail: cocoon-users-help@xml.apache.org
>
>


---------------------------------------------------------------------
To unsubscribe, e-mail: cocoon-users-unsubscribe@xml.apache.org
For additional commands, e-mail: cocoon-users-help@xml.apache.org


Re: Lucene index building

Posted by Jeremy Quinn <je...@media.demon.co.uk>.
On Monday, March 17, 2003, at 11:47 PM, Upayavira wrote:

> I have built a site which I want to index with Lucene.
>
> I am using the create-index.xsp file in the $COCOON-ROOT/search  
> directory to
> build my index.
>
> I have added the following to cocoon.xconf:
>
>   <cocoon-crawler logger="core.search.crawler">
>     <exclude>.*/search/.*</exclude>
>     <link-view-query>cocoon-view=lucene-links</link-view-query>
>   </cocoon-crawler>
>
>   <lucene-xml-indexer logger="core.search.lucene">
>   <store-fields>body</store-fields>
>     <content-view-query>cocoon-view=lucene-content</content-view-query>
>   </lucene-xml-indexer>
>

This all looks fine
My exclude string looks like this though :

<exclude>.*\.png$,.*\.js$,.*\.css$,.*\.gif$,.*\.jpg$,.*/search/.*,.*/ 
easy/.*</exclude>

I believe as soon as you specify an exclude string, the default values  
for images etc. are not used.

> I've set up a view lucene-links which works, giving back just links  
> from a page.
> I've set up a view lucene-content just giving back the content. The  
> content is like:
> <page>
>   <links>....list of links</links>
>   <body>... the body content ...</body>
> </page>
>
> I have had it partially working (indexing both links and body), but  
> now whenever I
> run create-index, it fails with a Cannot parse!:  
> org.xml.sax.SAXParseException:
> Premature end of file.
>
> Any ideas what I might be doing wrong?

I got problems like this, it turned out to be pages that did not return  
valid xml. Look in your logs to see if indexing stops on a particular  
url.

I also found that I could overcome the need to provide more memory by  
stripping un-needed tags from my 'content' xml being indexed.

My content for indexing looks like this:

<body>
	<title>title gets stored, then displayed with hit</title>
	<summary>summary gets stored, then displayed with hit</summary>
	all of my body content with tags stripped out
</body>

Hope this helps

regards Jeremy


---------------------------------------------------------------------
To unsubscribe, e-mail: cocoon-users-unsubscribe@xml.apache.org
For additional commands, e-mail: cocoon-users-help@xml.apache.org