[Lenya Wiki] Update of "Large number of documents HOWTO" by DominiqueDeMunck

[Apache Wiki <wi...@apache.org>]

Dear Wiki user,

You have subscribed to a wiki page or wiki category on "Lenya Wiki" for change notification.

The following page has been changed by DominiqueDeMunck:
http://wiki.apache.org/lenya/Large_number_of_documents_HOWTO

The comment on the change is:
first draft

New page:
* WORKING ON THIS RIGHT NOW *

It is possible to include 30 000+ documents in a Lenya publication.

Yet some problems make Lenya slow down a lot with that number of
documents. I used following steps in my Lenya 1.2.4 installation to speed Lenya
enormously:

1) Java XML processing supports XML documents of 10's of thousands of nodes,
yet therefore the default heap space needs to be enlarged. 
- In Linux/tomcat (jetty probably also)
	set the environment variable :

JAVA_OPTS="-Xmx256m"

- In Windows/tomcat, there is  a GUI tool called "tomcat5w" where you can set
this option for the tomcat service (Maximum Memory Pool option) .

2) The main slow-down problem is that the whole sitetree is passed through a 
number of navigation pipes. This is not really necessary, because generally, one only
wants to see the current document, its parents and its direct descendants.
This is why navigation in "Site" mode goes fast: it uses the SiteTreeFragmentGenerator.
A similar solution can be applied for the left navigation menu in authoring mode.
I created a Cocoon generator which only generates the minimal tree to navigate from the 
currently selected  document (see Attachement). 

You need to include that file into the lenya folder:

src/java/org/apache/lenya/cms/cocoon/generation/

And then we will make that generator our menu generator.

in src/webapp/sitemap.xmap add to the generators list:

<map:generator name="sitetree-selective" label="content,data" logger="sitemap.generator.sitetree-selective" pool-grow="2" pool-max="16" pool-min="2" src="org.apache.lenya.cms.cocoon.generation.SelectiveSitetreeGenerator"/>

and then we use it in our navigation:

src/webapp/lenya/navigation.xmap

replace our normal sitetree generator:

<map:generate type="sitetree">
            <map:parameter name="area" value="{2}"/>
</map:generate>

with our selective one:

<map:generate type="sitetree-selective">
          <map:parameter name="area" value="{page-envelope:area}"/>
          <map:parameter name="documentid" value="{page-envelope:document-id}"/>
          </map:generate>





3)

OK, we know only generate the sitetree that we need. 
Still, the sitetreeselective generator takes on a Pentium 4 / 2.4 Ghz about 200ms to fetch the right
components out of the sitetree.xml (30 000 nodes). Unfortunatly, in the current sitemap, this fetch occurs 4 times 
(breadcrumb, menu, tabs, search), which results in slowing down every click to a new document which 
wasn't cached to about 1 second. This is because the "map:aggregate" construct does not cache the first
fetch of this sitetree-selective. The "cinclude" does, which speeds up the rendering of a page with about 600ms
for every page you visit  (when sitetree was not cached after eg. move)! 
Therefore, in your publicaton-sitemap.xml  replace the part:

  <map:aggregate element="cmsbody">
          <map:part src="cocoon://navigation/{2}/{3}/breadcrumb/{5}.xml"/>
          <map:part src="cocoon://navigation/{2}/{3}/tabs/{5}.xml"/>
          <map:part src="cocoon://navigation/{2}/{3}/menu/{5}.xml"/>
          <map:part src="cocoon://navigation/{2}/{3}/search/{5}.xml"/>
          <map:part src="cocoon:/lenya-document-{1}/{3}/{4}/{page-envelope:document-path}"/>
        </map:aggregate>


with:


		<map:generate src="../../content/util/empty.xml" />
				<map:transform src="xslt/custom/lenyaBodyCincludes.xsl">
					<map:parameter name="rendertype" value="{1}" />
					<map:parameter name="publication-id" value="{2}" />
					<map:parameter name="area" value="{3}" />
					<map:parameter name="doctype" value="{4}" />
					<map:parameter name="url" value="{5}" />
					<map:parameter name="document-path" value="{page-envelope:document-path}" />
				</map:transform>
				<map:transform type="cinclude" />
				
				
You should add the attached XSL file lenyaBodyCincludes.xsl in "yourPUB/xslt/custom/" 





Note that for this caching to take effect you should make sure that the sitetree-selective pipe (in navigation.xmap) is cached !
On a normal build this should be ok since the default pipe was defined to be caching (in sitemap.xmap).

4)

the DocumentReferencesHelper looks through all the documents to find links to a document.
If don't want to slow down the delete usecase (which looks for broken links in all documents !),

You should uncomment the part with the DocumentReferencesHelper statements in 

src/webapp/lenya/content/info/delete.xsp

(insert comments after </task-id> and before </info> )


Other xsp's that use the reference helper and which you may want to disable (called via usecase.xmap).

src/webapp/lenya/content/info/archive.xsp
src/webapp/lenya/content/info/deactivate.xsp
src/webapp/lenya/content/info/delete.xsp
src/webapp/lenya/content/publishing/referenced-documents.xsp
src/webapp/lenya/content/publishing/screen.xsp
src/webapp/lenya/content/scheduler/screen.xsp

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lenya.apache.org
For additional commands, e-mail: dev-help@lenya.apache.org