You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@manifoldcf.apache.org by kw...@apache.org on 2010/11/12 22:33:21 UTC
svn commit: r1034577 [1/2] - in /incubator/lcf/site: publish/
src/documentation/content/xdocs/
Author: kwright
Date: Fri Nov 12 21:33:20 2010
New Revision: 1034577
URL: http://svn.apache.org/viewvc?rev=1034577&view=rev
Log:
Create a native site build-and-deploy page
Added:
incubator/lcf/site/publish/how-to-build-and-deploy.html
incubator/lcf/site/publish/how-to-build-and-deploy.pdf (with props)
incubator/lcf/site/src/documentation/content/xdocs/how-to-build-and-deploy.xml (with props)
Modified:
incubator/lcf/site/publish/developer-resources.html
incubator/lcf/site/publish/developer-resources.pdf
incubator/lcf/site/publish/end-user-documentation.pdf
incubator/lcf/site/publish/faq.pdf
incubator/lcf/site/publish/index.pdf
incubator/lcf/site/publish/linkmap.pdf
incubator/lcf/site/publish/mail.pdf
incubator/lcf/site/publish/who.pdf
incubator/lcf/site/src/documentation/content/xdocs/developer-resources.xml
Modified: incubator/lcf/site/publish/developer-resources.html
URL: http://svn.apache.org/viewvc/incubator/lcf/site/publish/developer-resources.html?rev=1034577&r1=1034576&r2=1034577&view=diff
==============================================================================
--- incubator/lcf/site/publish/developer-resources.html (original)
+++ incubator/lcf/site/publish/developer-resources.html Fri Nov 12 21:33:20 2010
@@ -250,7 +250,7 @@ document.write("Last Published: " + docu
<a name="N1002B"></a><a name="howtobuilddeploy"></a>
<h2 class="h3">How to Build and Deploy</h2>
<div class="section">
-<p>Building ManifoldCF requires ant 1.7 or greater. Instructions for building and deploying can be found <a href="http://cwiki.apache.org/confluence/display/CONNECTORS/How+to+Build+and+Deploy+ManifoldCF">here</a>.
+<p>Building ManifoldCF requires ant 1.7 or greater. Instructions for building and deploying can be found <a href="how-to-build-and-deploy.html">here</a>.
</p>
</div>
Modified: incubator/lcf/site/publish/developer-resources.pdf
URL: http://svn.apache.org/viewvc/incubator/lcf/site/publish/developer-resources.pdf?rev=1034577&r1=1034576&r2=1034577&view=diff
==============================================================================
Binary files - no diff available.
Modified: incubator/lcf/site/publish/end-user-documentation.pdf
URL: http://svn.apache.org/viewvc/incubator/lcf/site/publish/end-user-documentation.pdf?rev=1034577&r1=1034576&r2=1034577&view=diff
==============================================================================
Files incubator/lcf/site/publish/end-user-documentation.pdf (original) and incubator/lcf/site/publish/end-user-documentation.pdf Fri Nov 12 21:33:20 2010 differ
Modified: incubator/lcf/site/publish/faq.pdf
URL: http://svn.apache.org/viewvc/incubator/lcf/site/publish/faq.pdf?rev=1034577&r1=1034576&r2=1034577&view=diff
==============================================================================
Files incubator/lcf/site/publish/faq.pdf (original) and incubator/lcf/site/publish/faq.pdf Fri Nov 12 21:33:20 2010 differ
Added: incubator/lcf/site/publish/how-to-build-and-deploy.html
URL: http://svn.apache.org/viewvc/incubator/lcf/site/publish/how-to-build-and-deploy.html?rev=1034577&view=auto
==============================================================================
--- incubator/lcf/site/publish/how-to-build-and-deploy.html (added)
+++ incubator/lcf/site/publish/how-to-build-and-deploy.html Fri Nov 12 21:33:20 2010
@@ -0,0 +1,1301 @@
+<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
+<html>
+<head>
+<META http-equiv="Content-Type" content="text/html; charset=UTF-8">
+<meta content="Apache Forrest" name="Generator">
+<meta name="Forrest-version" content="0.9-dev">
+<meta name="Forrest-skin-name" content="lucene">
+<title>Building ManifoldCF</title>
+<link type="text/css" href="skin/basic.css" rel="stylesheet">
+<link media="screen" type="text/css" href="skin/screen.css" rel="stylesheet">
+<link media="print" type="text/css" href="skin/print.css" rel="stylesheet">
+<link type="text/css" href="skin/profile.css" rel="stylesheet">
+<script src="skin/getBlank.js" language="javascript" type="text/javascript"></script><script src="skin/getMenu.js" language="javascript" type="text/javascript"></script><script src="skin/fontsize.js" language="javascript" type="text/javascript"></script>
+<link rel="shortcut icon" href="images/favicon.ico">
+</head>
+<body onload="init()">
+<script type="text/javascript">ndeSetTextSize();</script>
+<div id="top">
+<!--+
+ |breadtrail
+ +-->
+<div class="breadtrail">
+<a href="http://www.apache.org/">Apache</a><script src="skin/breadcrumbs.js" language="JavaScript" type="text/javascript"></script>
+</div>
+<!--+
+ |header
+ +-->
+<div class="header">
+<!--+
+ |start group logo
+ +-->
+<div class="grouplogo">
+<a href="http://www.apache.org"><img class="logoImage" alt="Apache" src="images/apache_feather.gif" title="Apache Software Foundation"></a>
+</div>
+<!--+
+ |end group logo
+ +-->
+<!--+
+ |start Project Logo
+ +-->
+<div class="projectlogo">
+<a href="http://incubator.apache.org/lcf"><img class="logoImage" alt="Apache ManifoldCF" src="images/ManifoldCF-logo.PNG" title="ManifoldCF"></a>
+</div>
+<!--+
+ |end Project Logo
+ +-->
+<!--+
+ |start Search
+ +-->
+<div class="searchbox">
+<form action="http://www.lucidimagination.com/search/" method="get" class="roundtopsmall">
+<input onFocus="getBlank (this, 'Search the site with Solr');" size="25" name="q" id="query" type="text" value="Search the site with Solr">
+ <input name="Search" value="Search" type="submit">
+</form>
+<div style="position: relative; top: -5px; left: -10px">Powered by <a href="http://www.lucidimagination.com" style="color: #033268">Lucid Imagination</a>
+</div>
+</div>
+<!--+
+ |end search
+ +-->
+<!--+
+ |start Tabs
+ +-->
+<ul id="tabs">
+<li class="current">
+<a class="selected" href="index.html">Main</a>
+</li>
+<li>
+<a class="unselected" href="http://cwiki.apache.org/confluence/display/CONNECTORS/Index">Wiki</a>
+</li>
+</ul>
+<!--+
+ |end Tabs
+ +-->
+</div>
+</div>
+<div id="main">
+<div id="publishedStrip">
+<!--+
+ |start Subtabs
+ +-->
+<div id="level2tabs"></div>
+<!--+
+ |end Endtabs
+ +-->
+<script type="text/javascript"><!--
+document.write("Last Published: " + document.lastModified);
+// --></script>
+</div>
+<!--+
+ |breadtrail
+ +-->
+<div class="breadtrail">
+
+
+ </div>
+<!--+
+ |start Menu, mainarea
+ +-->
+<!--+
+ |start Menu
+ +-->
+<div id="menu">
+<div onclick="SwitchMenu('menu_1.1', 'skin/')" id="menu_1.1Title" class="menutitle">About</div>
+<div id="menu_1.1" class="menuitemgroup">
+<div class="menuitem">
+<a href="index.html">Welcome</a>
+</div>
+<div class="menuitem">
+<a href="who.html">Who We Are</a>
+</div>
+<div class="menuitem">
+<a href="mail.html">Mailing Lists</a>
+</div>
+<div class="menuitem">
+<a href="http://www.cafepress.com/lucene/">Buy Stuff</a>
+</div>
+<div class="menuitem">
+<a href="http://www.apache.org/foundation/sponsorship.html">Sponsor Apache</a>
+</div>
+<div class="menuitem">
+<a href="http://www.apache.org/foundation/thanks.html">Sponsors of Apache</a>
+</div>
+</div>
+<div onclick="SwitchMenu('menu_1.2', 'skin/')" id="menu_1.2Title" class="menutitle">Documentation</div>
+<div id="menu_1.2" class="menuitemgroup">
+<div class="menuitem">
+<a href="faq.html">Frequently Asked Questions</a>
+</div>
+<div class="menuitem">
+<a href="developer-resources.html">Developer/Integrator Resources</a>
+</div>
+<div class="menuitem">
+<a href="end-user-documentation.html">End-user Documentation</a>
+</div>
+</div>
+<div onclick="SwitchMenu('menu_1.3', 'skin/')" id="menu_1.3Title" class="menutitle">Related-Projects</div>
+<div id="menu_1.3" class="menuitemgroup">
+<div class="menuitem">
+<a href="http://incubator.apache.org/droids/">Droids</a>
+</div>
+<div class="menuitem">
+<a href="http://lucene.apache.org/java/">Java</a>
+</div>
+<div class="menuitem">
+<a href="http://incubator.apache.org/lucene.net/">Lucene.Net</a>
+</div>
+<div class="menuitem">
+<a href="http://lucene.apache.org/lucy/">Lucy</a>
+</div>
+<div class="menuitem">
+<a href="http://lucene.apache.org/mahout/">Mahout</a>
+</div>
+<div class="menuitem">
+<a href="http://lucene.apache.org/nutch/">Nutch</a>
+</div>
+<div class="menuitem">
+<a href="http://lucene.apache.org/openrelevance/">Open Relevance</a>
+</div>
+<div class="menuitem">
+<a href="http://lucene.apache.org/pylucene/">PyLucene</a>
+</div>
+<div class="menuitem">
+<a href="http://lucene.apache.org/solr/">Solr</a>
+</div>
+<div class="menuitem">
+<a href="http://lucene.apache.org/tika/">Tika</a>
+</div>
+</div>
+<div id="credit"></div>
+<div id="roundbottom">
+<img style="display: none" class="corner" height="15" width="15" alt="" src="skin/images/rc-b-l-15-1body-2menu-3menu.png"></div>
+<!--+
+ |alternative credits
+ +-->
+<div id="credit2"></div>
+</div>
+<!--+
+ |end Menu
+ +-->
+<!--+
+ |start content
+ +-->
+<div id="content">
+<div title="Portable Document Format" class="pdflink">
+<a class="dida" href="how-to-build-and-deploy.pdf"><img alt="PDF -icon" src="skin/images/pdfdoc.gif" class="skin"><br>
+ PDF</a>
+</div>
+<h1>Building ManifoldCF</h1>
+<div id="minitoc-area">
+<ul class="minitoc">
+<li>
+<a href="#Building+ManifoldCF">Building ManifoldCF</a>
+<ul class="minitoc">
+<li>
+<a href="#Building+the+framework+and+the+connectors">Building the framework and the connectors</a>
+<ul class="minitoc">
+<li>
+<a href="#Building+the+Documentum+connector">Building the Documentum connector</a>
+</li>
+<li>
+<a href="#Building+the+FileNet+connector">Building the FileNet connector</a>
+</li>
+<li>
+<a href="#Building+the+JDBC+connector%2C+including+Oracle%2C+SQLServer%2C+or+Sybase+JDBC+drivers">Building the JDBC connector, including Oracle, SQLServer, or Sybase JDBC drivers</a>
+</li>
+<li>
+<a href="#Building+the+jCIFS+connector">Building the jCIFS connector</a>
+</li>
+<li>
+<a href="#Building+the+LiveLink+connector">Building the LiveLink connector</a>
+</li>
+<li>
+<a href="#Building+the+Memex+connector">Building the Memex connector</a>
+</li>
+<li>
+<a href="#Building+the+Meridio+connector">Building the Meridio connector</a>
+</li>
+<li>
+<a href="#Building+the+SharePoint+connector">Building the SharePoint connector</a>
+</li>
+<li>
+<a href="#Building+ManifoldCF%27s+Apache2+plugin">Building ManifoldCF's Apache2 plugin</a>
+</li>
+</ul>
+</li>
+</ul>
+</li>
+<li>
+<a href="#Running+ManifoldCF">Running ManifoldCF</a>
+<ul class="minitoc">
+<li>
+<a href="#Quick+start">Quick start</a>
+<ul class="minitoc">
+<li>
+<a href="#The+quick-start+connectors.xml+configuration+file">The quick-start connectors.xml configuration file</a>
+</li>
+</ul>
+</li>
+<li>
+<a href="#Framework+and+connectors">Framework and connectors</a>
+<ul class="minitoc">
+<li>
+<a href="#Configuring+the+Postgresql+database">Configuring the Postgresql database</a>
+</li>
+<li>
+<a href="#A+note+about+maintenance">A note about maintenance</a>
+</li>
+<li>
+<a href="#The+ManifoldCF+configuration+file">The ManifoldCF configuration file</a>
+</li>
+<li>
+<a href="#Properties">Properties</a>
+</li>
+<li>
+<a href="#Class+path+libraries">Class path libraries</a>
+</li>
+<li>
+<a href="#Examples">Examples</a>
+</li>
+<li>
+<a href="#Commands">Commands</a>
+</li>
+<li>
+<a href="#Initializing+the+database">Initializing the database</a>
+</li>
+<li>
+<a href="#Deploying+the">Deploying the mcf-crawler-ui, mcf-authority-service, and mcf-api-service web applications</a>
+</li>
+<li>
+<a href="#Running+the">Running the agents process</a>
+</li>
+<li>
+<a href="#Running+connector-specific+processes">Running connector-specific processes</a>
+</li>
+</ul>
+</li>
+<li>
+<a href="#Running+the+ManifoldCF+Apache2+plug+in">Running the ManifoldCF Apache2 plug in</a>
+<ul class="minitoc">
+<li>
+<a href="#Configuring+the+ManifoldCF+Apache2+plug+in">Configuring the ManifoldCF Apache2 plug in</a>
+</li>
+</ul>
+</li>
+</ul>
+</li>
+</ul>
+</div>
+
+<a name="N1000D"></a><a name="Building+ManifoldCF"></a>
+<h2 class="h3">Building ManifoldCF</h2>
+<div class="section">
+<p></p>
+<p>ManifoldCF consists of the framework itself, a set of connectors, and an Apache2 plug-in module. These can be built as follows.</p>
+<p></p>
+<a name="N1001A"></a><a name="Building+the+framework+and+the+connectors"></a>
+<h3 class="h4">Building the framework and the connectors</h3>
+<p></p>
+<p>To build the ManifoldCF framework code, and the particular connectors you are interested in, you currently need to do the following:</p>
+<p></p>
+<ol>
+
+<li>Check out https://svn.apache.org/repos/asf/incubator/lcf/trunk.</li>
+
+<li>cd to "modules".</li>
+
+<li>Install desired dependent LGPL and proprietary libraries, wsdls, and xsds. See below for details.</li>
+
+<li>Run ant.</li>
+
+</ol>
+<p></p>
+<p>If you supply <strong>no</strong> LGPL or proprietary libraries, the framework itself and only the following repository connectors will be built:</p>
+<p></p>
+<ul>
+
+<li>Active Directory authority</li>
+
+<li>Filesystem connector</li>
+
+<li>JDBC connector, with just the postgresql jdbc driver</li>
+
+<li>RSS connector</li>
+
+<li>Webcrawler connector</li>
+
+</ul>
+<p></p>
+<p>In addition, the following output connectors will be built:</p>
+<p></p>
+<ul>
+
+<li>MetaCarta GTS output connector</li>
+
+<li>Apache Solr output connector</li>
+
+<li>Null output connector</li>
+
+<li>Null authority connector</li>
+
+</ul>
+<p></p>
+<p>The LGPL and proprietary connector dependencies are described in separate sections below.</p>
+<p></p>
+<p>The output of the ant build is produced in the <em>modules/dist</em> directory, which is further broken down by process. The number of produced process directories may vary, because optional individual connectors do sometimes supply processes that must be run to support the connector. See the table below for a description of the <em>modules/dist</em> folder.</p>
+<p></p>
+<table class="ForrestTable" cellspacing="1" cellpadding="4">
+
+<caption>Distribution directories</caption>
+
+<tr>
+<th colspan="1" rowspan="1"><em>modules/dist</em> directory</th><th colspan="1" rowspan="1">Meaning</th>
+</tr>
+
+<tr>
+<td colspan="1" rowspan="1"><em>web</em></td><td colspan="1" rowspan="1">Web applications that should be deployed on tomcat or the equivalent, plus recommended application server -D switch names and values</td>
+</tr>
+
+<tr>
+<td colspan="1" rowspan="1"><em>processes</em></td><td colspan="1" rowspan="1">classpath jars that should be included in the class path for all non-connector-specific processes, along with -D switches, using the same convention as described for tomcat, above</td>
+</tr>
+
+<tr>
+<td colspan="1" rowspan="1"><em>lib</em></td><td colspan="1" rowspan="1">jars for all the connector plugins, which should be referenced by the appropriate clause in the ManifoldCF configuration file</td>
+</tr>
+
+<tr>
+<td colspan="1" rowspan="1"><em>wsdd</em></td><td colspan="1" rowspan="1">wsdd files that are needed by the included connectors in order to function</td>
+</tr>
+
+<tr>
+<td colspan="1" rowspan="1"><em>xxx-process</em></td><td colspan="1" rowspan="1">classpath jars and -D switches needed for a required connector-specific process</td>
+</tr>
+
+<tr>
+<td colspan="1" rowspan="1"><em>example</em></td><td colspan="1" rowspan="1">a jetty-based example that runs in a single process (except for any connector-specific processes)</td>
+</tr>
+
+<tr>
+<td colspan="1" rowspan="1"><em>doc</em></td><td colspan="1" rowspan="1">javadocs for framework and all included connectors</td>
+</tr>
+
+</table>
+<p></p>
+<p>For all of the <em>dist</em> subdirectories above (except for <em>wsdd</em>, which does not correspond to a process), any scripts resulting from the build that pertain to that process will be placed in a <em>script</em> subdirectory. Thus, the command for executing a command under Windows for the <em>processes</em> subdirectory will be found in <em>dist/processes/script/executecommand.bat</em>. (This script requires two variables to be set before execution: JAVA_HOME, and MCF_HOME, which should point to ManifoldCF's home execution directory, described below.) Indeed, everything you need to run an ManifoldCF process can be found under <em>dist/processes</em> when the ant build completes: a <em>define</em> subdirectory containing -D switch description files, a <em>jar</em> subdirectory where jars are placed, and a <em>war</em> subdirectory where war files are output. </p>
+<p></p>
+<p>The supplied scripts in the <em>script</em> directory for a process generally take care of building an appropriate classpath and set of -D switches. If you need to construct a classpath by hand, it is important to remember that "more" is not necessarily "better". The process deployment strategy implied by the build structure has been carefully thought out to avoid jar conflicts. Indeed, several connectors are structured using multiple processes precisely for that reason.</p>
+<p></p>
+<a name="N10103"></a><a name="Building+the+Documentum+connector"></a>
+<h4>Building the Documentum connector</h4>
+<p></p>
+<p>The Documentum connector requires EMC's DFC product in order to be built. Install DFC on the build system, and locate the jars it installs. You will need to copy at least dfc.jar, dfcbase.jar, and dctm.jar into the directory "modules/connectors/documentum/dfc".</p>
+<p></p>
+<a name="N10111"></a><a name="Building+the+FileNet+connector"></a>
+<h4>Building the FileNet connector</h4>
+<p></p>
+<p>The FileNet connector requires IBM's FileNet P8 API jar in order to be build. Install the FileNet P8 API on the build system, and copy at least "Jace.jar" from that installation into "modules/connectors/filenet/filenet-api".</p>
+<p></p>
+<a name="N1011F"></a><a name="Building+the+JDBC+connector%2C+including+Oracle%2C+SQLServer%2C+or+Sybase+JDBC+drivers"></a>
+<h4>Building the JDBC connector, including Oracle, SQLServer, or Sybase JDBC drivers</h4>
+<p></p>
+<p>The JDBC connector also knows how to work with Oracle, SQLServer, and Sybase JDBC drivers. For Oracle, download the appropriate Oracle JDBC jar from the Oracle site, and copy it into the directory "modules/connectors/jdbc/jdbc-drivers". For SQLServer and Sybase, download jtds.jar, and copy it into the same directory.</p>
+<p></p>
+<a name="N1012D"></a><a name="Building+the+jCIFS+connector"></a>
+<h4>Building the jCIFS connector</h4>
+<p></p>
+<p>To build this connector, you need to download jcifs.jar from <a href="http://samba.jcifs.org">http://samba.jcifs.org</a>, and copy it into the "modules/connectors/jcifs/jcifs" directory.</p>
+<p></p>
+<a name="N1013F"></a><a name="Building+the+LiveLink+connector"></a>
+<h4>Building the LiveLink connector</h4>
+<p></p>
+<p>This connector needs LAPI, which is a proprietary java library that allows access to OpenText's LiveLink server. Copy the lapi.jar into the "modules/connectors/livelink/lapi" directory.</p>
+<p></p>
+<a name="N1014D"></a><a name="Building+the+Memex+connector"></a>
+<h4>Building the Memex connector</h4>
+<p></p>
+<p>This connector needs the Memex API jar, usually called JavaMXIELIB.jar. Copy this jar into the "modules/connectors/memex/mxie-java" directory.</p>
+<p></p>
+<a name="N1015B"></a><a name="Building+the+Meridio+connector"></a>
+<h4>Building the Meridio connector</h4>
+<p></p>
+<p>The Meridio connector needs wsdls and xsds downloaded from an installed Meridio instance using <strong>disco.exe</strong>, which is installed as part of Microsoft Visual Studio, typically under "c:\Program Files\Microsoft SDKs\Windows\V6.x\bin". Obtain the preliminary wsdls and xsds by interrogating the following Meridio web services:</p>
+<p></p>
+<ul>
+
+<li>http[s]://<meridio_server>/DMWS/MeridioDMWS.asmx</li>
+
+<li>http[s]://<meridio_server>/RMWS/MeridioRMWS.asmx</li>
+
+</ul>
+<p></p>
+<p>You should have obtained the following files in this step:</p>
+<p></p>
+<ul>
+
+<li>MeridioDMWS.wsdl</li>
+
+<li>MeridioRMWS.wsdl</li>
+
+<li>DMDataSet.xsd</li>
+
+<li>RMDataSet.xsd</li>
+
+<li>RMClassificationDataSet.xsd</li>
+
+</ul>
+<p></p>
+<p>Next, patch these using Microsoft's <strong>xmldiffpatch</strong> utility suite, downloadable for Windows from <a href="http://msdn.microsoft.com/en-us/library/aa302294.aspx">http://msdn.microsoft.com/en-us/library/aa302294.aspx</a>. The appropriate diff files to apply as patches can be found in "modules/connectors/meridio/upstream-diffs". After the patching, rename so that you have the files:</p>
+<p></p>
+<ul>
+
+<li>MeridioDMWS_axis.wsdl</li>
+
+<li>MeridioRMWS_axis.wsdl</li>
+
+<li>DMDataSet_castor.xsd</li>
+
+<li>RMDataSet_castor.xsd</li>
+
+<li>RMClassificationDataSet_castor.xsd</li>
+
+</ul>
+<p></p>
+<p>Finally, copy all of these to: "modules/connectors/meridio/wsdls".</p>
+<p></p>
+<a name="N101B5"></a><a name="Building+the+SharePoint+connector"></a>
+<h4>Building the SharePoint connector</h4>
+<p></p>
+<p>In order to build this connector, you need to download wsdls from an installed SharePoint instance. The wsdls in question are:</p>
+<p></p>
+<ul>
+
+<li>Permissions.wsdl</li>
+
+<li>Lists.wsdl</li>
+
+<li>Dspsts.wsdl</li>
+
+<li>usergroup.wsdl</li>
+
+<li>versions.wsdl</li>
+
+<li>webs.wsdl</li>
+
+</ul>
+<p></p>
+<p>To download a wsdl, use Microsoft's <strong>disco.exe</strong> tool, which is part of Visual Studio, typically under "c:\Program Files\Microsoft SDKs\Windows\V6.x\bin". You'd want to interrogate the following urls:</p>
+<p></p>
+<ul>
+
+<li>http[s]://<server_name>/_vti_bin/Permissions.asmx</li>
+
+<li>http[s]://<server_name>/_vti_bin/Lists.asmx</li>
+
+<li>http[s]://<server_name>/_vti_bin/Dspsts.asmx</li>
+
+<li>http[s]://<server_name>/_vti_bin/usergroup.asmx</li>
+
+<li>http[s]://<server_name>/_vti_bin/versions.asmx</li>
+
+<li>http[s]://<server_name>/_vti_bin/webs.asmx</li>
+
+</ul>
+<p></p>
+<p>When the wsdl files have been downloaded, copy them to: "modules/connectors/sharepoint/wsdls".</p>
+<p></p>
+<p>Note well: For SharePoint instances version 3.0 or higher, in order to support file and folder level security, you also must deploy a custom SharePoint web service on the SharePoint instance you intend to connect to. This is because Microsoft apparently overlooked support for web-service-based access to such security information when SharePoint 3.0 was released.</p>
+<p></p>
+<p>In order to build the service, you need to have access to a Windows machine that has a reasonably current version of Microsoft Visual Studio available, with .NET installed and (at least) SharePoint 2.0 installed as well. The fastest way to build the service is to do the following <strong>after</strong> building everything else:</p>
+<p></p>
+<pre class="code">
+cd connectors/sharepoint
+ant build-webservice
+cd webservice/Package
+ </pre>
+<p></p>
+<p>Then, follow the directions in the file "Installation Readme.txt", found in that directory.</p>
+<p></p>
+<p></p>
+<a name="N10218"></a><a name="Building+ManifoldCF%27s+Apache2+plugin"></a>
+<h4>Building ManifoldCF's Apache2 plugin</h4>
+<p></p>
+<p>To build the mod-authz-annotate plugin, you need to start with a Unix system that has the apache2 development tools installed on it, plus the curl development package (from <a href="http://curl.haxx.se">http://curl.haxx.se</a> or elsewhere). Then, cd to modules/mod-authz-annotate, and type "make". The build will produce a file called mod-authz-annotate.so, which should be copied to the appropriate Apache2 directory so it can be used as a plugin.</p>
+<p></p>
+<p></p>
+</div>
+
+
+<a name="N1022E"></a><a name="Running+ManifoldCF"></a>
+<h2 class="h3">Running ManifoldCF</h2>
+<div class="section">
+<p></p>
+<a name="N10236"></a><a name="Quick+start"></a>
+<h3 class="h4">Quick start</h3>
+<p></p>
+<p>You can run most of ManifoldCF in a single process, for evaluation and convenience. This single-process version uses Jetty to handle its web applications, and Derby as an embedded database. All you need to do to run this version of ManifoldCF is to follow the build instructions above, and then:</p>
+<p></p>
+<pre class="code">
+cd dist/example
+<java> -jar start.jar
+ </pre>
+<p></p>
+<p>In this jetty setup, all database initialization and connector registration takes place automatically (at the cost of some startup delay). The crawler UI can be found at http://>host<:8345/mcf-crawler-ui. The authority service can be found at http://>host<:8345/mcf-authority-service. The programmatic API is at http://>host<:8345/mcf-api.</p>
+<p></p>
+<p>You can stop ManifoldCF at any time using ^C.</p>
+<p></p>
+<p>Bear in mind that Derby is not as full-featured a database as is Postgresql. This means that any performance testing you may do against the quick start example may not be applicable to a full installation. Furthermore, Derby only permits one process at a time to be connected to its databases, so you <strong>cannot</strong> use any of the ManifoldCF commands (as described below) while the quick-start ManifoldCF is running.</p>
+<p></p>
+<p>Another caveat that you will need to be aware of with the quick-start version of ManifoldCF is that it in no way removes the need for you to run any separate processes that individual connectors require. Specifically, the Documentum and FileNet connectors require processes to be independently started in order to function. You will need to read about these connector-specific processes below in order to use the corresponding connectors.</p>
+<p></p>
+<a name="N10260"></a><a name="The+quick-start+connectors.xml+configuration+file"></a>
+<h4>The quick-start connectors.xml configuration file</h4>
+<p></p>
+<p>The quick-start version of ManifoldCF reads its own configuration file, called <em>connectors.xml</em>, in order to register the available connectors in the database. The file has this basic format:</p>
+<p></p>
+<pre class="code">
+<?xml version="1.0" encoding="UTF-8" ?>
+<connectors>
+ (clauses)
+</connectors>
+ </pre>
+<p></p>
+<p>The following tags are available to specify your connectors:</p>
+<p></p>
+<p><repositoryconnector name="<em>pretty_name</em>" class="<em>connector_class</em>"/></p>
+<p><authorityconnector name="<em>pretty_name</em>" class="<em>connector_class</em>"/></p>
+<p><outputconnector name="<em>pretty_name</em>" class="<em>connector_class</em>"/></p>
+<p></p>
+<a name="N1029A"></a><a name="Framework+and+connectors"></a>
+<h3 class="h4">Framework and connectors</h3>
+<p></p>
+<p>The core part of ManifoldCF consists of several pieces. These basic pieces are enumerated below:</p>
+<p></p>
+<ul>
+
+<li>A database, which is where ManifoldCF keeps all of its configuration and state information, usually Postgresql</li>
+
+<li>A synchronization directory, which how ManifoldCF coordinates activity among its various processes</li>
+
+<li>An <strong>agents</strong> process, which is the process that actually crawls documents and ingests them</li>
+
+<li>A <strong>crawler-ui</strong> web application, which presents the UI users interact with to configure and control the crawler</li>
+
+<li>An <strong>authority-service</strong> web application, which responds to requests for authorization tokens, given a user name</li>
+
+<li>An <strong>api-service</strong> web application, which responds to REST API requests</li>
+
+</ul>
+<p></p>
+<p>In addition, there are a number of java classes in ManifoldCF that are intended to be called directly, to perform specific actions in the environment or in the database. These classes are usually invoked from the command line, with appropriate arguments supplied, and are thus considered to be ManifoldCF <strong>commands</strong>. Basic functionality supplied by these command classes are as follows:</p>
+<p></p>
+<ul>
+
+<li>Create/Destroy the ManifoldCF database instance</li>
+
+<li>Start/Stop the <strong>agents</strong> process</li>
+
+<li>Register/Unregister an agent class (there's currently only one included)</li>
+
+<li>Register/Unregister an output connector</li>
+
+<li>Register/Unregister a repository connector</li>
+
+<li>Register/Unregister an authority connector</li>
+
+<li>Clean up synchronization directory garbage resulting from an ungraceful interruption of an ManifoldCF process</li>
+
+<li>Query for certain kinds of job-related information</li>
+
+</ul>
+<p></p>
+<p>Individual connectors may contribute additional command classes and processes to this picture. A properly built connector typically consists of:</p>
+<p></p>
+<ul>
+
+<li>One or more jar files meant to be included in the library area meant for connector jars and their dependencies.</li>
+
+<li>Possibly some java commands, which are meant to support or configure the connector in some way.</li>
+
+<li>Possibly a connector-specific process or two, each requiring a distinct classpath, which usually serves to isolate the <strong>crawler-ui</strong> web application, <strong>authority-service</strong> web application, <strong>agents</strong> process, and any commands from problematic aspects of the client environment</li>
+
+<li>A recommended set of java "define" variables, which should be used consistently with all involved processes, e.g. the <strong>agents</strong> process, the application server running the <strong>authority-service</strong> and <strong>crawler-ui</strong>, and any commands. (This is historical, and no connectors as of this writing have any of these any longer).</li>
+
+</ul>
+<p></p>
+<p>An individual connector package will typically supply an output connector, or a repository connector, or both a repository connector and an authority connector. The ant build script under <em>modules</em> automatically forms each individual connector's contribution to the overall system into the overall package.</p>
+<p></p>
+<p>The basic steps required to set up and run ManifoldCF are as follows:</p>
+<p></p>
+<ul>
+
+<li>Check out and build, using "ant". The default target builds everything.</li>
+
+<li>Install postgresql. The postgresql JDBC driver included with ManifoldCF is known to work with version 8.3.x, so that version is the currently recommended one. Configure postgresql for your environment; the default configuration is acceptable for testing and experimentation.</li>
+
+<li>Install a Java application server, such as Tomcat.</li>
+
+<li>Create a home directory for ManifoldCF. To do this, make a copy of the contents of <em>modules/dist</em> from the build. In this directory, create properties.ini and logging.ini, as described above. Note that you will also need to create a synchronization directory, also detailed above, and refer to this directory within your properties.xml.</li>
+
+<li>Deploy the war files in <em><MCF_HOME>/web/war</em> to your application server.</li>
+
+<li>Set the starting environment variables for your app server to include the -D commands found in <em><MCF_HOME>/web/define</em>. The -D commands should be of the form, "-D<file name>=<file contents>".</li>
+
+<li>Use the <em><MCF_HOME>/processes/script/executecommand.bat</em> command from execute the appropriate commands from the next section below, being sure to first set the JAVA_HOME and MCF_HOME environment variables properly.</li>
+
+<li>Start any supporting processes that result from your build. (Some connectors such as Documentum and FileNet have auxiliary processes you need to run to make these connectors functional.)</li>
+
+<li>Start your application server.</li>
+
+<li>Start the ManifoldCF agents process.</li>
+
+<li>At this point, you should be able to interact with the ManifoldCF UI, which can be accessed via the mcf-crawler-ui web application</li>
+
+</ul>
+<p></p>
+<p>For each of the described steps, details are furnished in the steps below.</p>
+<p></p>
+<p></p>
+<a name="N10360"></a><a name="Configuring+the+Postgresql+database"></a>
+<h4>Configuring the Postgresql database</h4>
+<p></p>
+<p>Despite having an internal architecture that cleanly abstracts from specific database details, ManifoldCF is currently fairly specific to Postgresql at this time. There are a number of reasons for this.</p>
+<p></p>
+<ul>
+
+<li>ManifoldCF uses the database for its document queue, which places a significant load on it. The back-end database is thus a significant factor in ManifoldCF's performance. But, in exchange, ManifoldCF benefits enormously from the underlying ACID properties of the database.</li>
+
+<li>The strategy for getting optimal query plans from the database is not abstracted. For example, Postgresql 8.3+ is very sensitive to certain statistics about a database table, and will not generate a performant plan if the statistics are inaccurate by even a little, in some cases. So, for Postgresql, the database table must be analyzed very frequently, to avoid catastrophically bad plans. But luckily, Postgresql is pretty good at doing analysis quickly. Oracle, on the other hand, takes a very long time to perform analysis, but its plans are much less sensitive.</li>
+
+<li>Postgresql always does a sequential scan in order to count the number of rows in a table, while other databases return this efficiently. This has affected the design of the ManifoldCF UI.</li>
+
+<li>The choice of query form influences the query plan. Ideally, this is not true, but for both Postgresql and for (say) Oracle, it is.</li>
+
+<li>Postgresql has a high degree of parallelism and lack of internal single-threadedness.</li>
+
+</ul>
+<p></p>
+<p>ManifoldCF has been tested against Postgresql 8.3.7. We recommend the following configuration parameter settings to work optimally with ManifoldCF:</p>
+<p></p>
+<ul>
+
+<li>A default database encoding of UTF-8</li>
+
+<li>
+<em>postgresql.conf</em> settings as described in the table below</li>
+
+<li>
+<em>pg_hba.conf</em> settings to allow password access for TCP/IP connections from ManifoldCF</li>
+
+<li>A maintenance strategy involving cronjob-style vacuuming, rather than Postgresql autovacuum</li>
+
+</ul>
+<p></p>
+<table class="ForrestTable" cellspacing="1" cellpadding="4">
+
+<caption>Postgresql.conf parameters</caption>
+
+<tr>
+<th colspan="1" rowspan="1"><em>postgresql.conf</em> parameter</th><th colspan="1" rowspan="1">Tested value</th>
+</tr>
+
+<tr>
+<td colspan="1" rowspan="1">shared_buffers</td><td colspan="1" rowspan="1">1024MB</td>
+</tr>
+
+<tr>
+<td colspan="1" rowspan="1">checkpoint_segments</td><td colspan="1" rowspan="1">300</td>
+</tr>
+
+<tr>
+<td colspan="1" rowspan="1">maintenance<em>work</em>mem</td><td colspan="1" rowspan="1">2MB</td>
+</tr>
+
+<tr>
+<td colspan="1" rowspan="1">tcpip_socket</td><td colspan="1" rowspan="1">true</td>
+</tr>
+
+<tr>
+<td colspan="1" rowspan="1">max_connections</td><td colspan="1" rowspan="1">400</td>
+</tr>
+
+<tr>
+<td colspan="1" rowspan="1">checkpoint_timeout</td><td colspan="1" rowspan="1">900</td>
+</tr>
+
+<tr>
+<td colspan="1" rowspan="1">datastyle</td><td colspan="1" rowspan="1">ISO,European</td>
+</tr>
+
+<tr>
+<td colspan="1" rowspan="1">autovacuum</td><td colspan="1" rowspan="1">off</td>
+</tr>
+
+</table>
+<p></p>
+<a name="N10403"></a><a name="A+note+about+maintenance"></a>
+<h4>A note about maintenance</h4>
+<p></p>
+<p>Postgresql's architecture causes it to accumulate dead tuples in its data files, which do not interfere with its performance but do bloat the database over time. The usage pattern of ManifoldCF is such that it can cause significant bloat to occur to the underlying Postgresql database in only a few days, under sufficient load. Postgresql has a feature to address this bloat, called <strong>vacuuming</strong>. This comes in three varieties: autovacuum, manual vacuum, and manual full vacuum.</p>
+<p></p>
+<p>We have found that Postgresql's autovacuum feature is inadequate under such conditions, because it not only fights for database resources pretty much all the time, but it falls further and further behind as well. Postgresql's in-place manual vacuum functionality is a bit better, but is still much, much slower than actually making a new copy of the database files, which is what happens when a manual full vacuum is performed.</p>
+<p></p>
+<p>Dead-tuple bloat also occurs in indexes in Postgresql, so tables that have had a lot of activity may benefit from being reindexed at the time of maintenance. </p>
+<p>We therefore recommend periodic, scheduled maintenance operations instead, consisting of the following:</p>
+<p></p>
+<ul>
+
+<li>VACUUM FULL VERBOSE;</li>
+
+<li>REINDEX DATABASE <the_db_name>;</li>
+
+</ul>
+<p>
+</p>
+<p>During maintenance, Postgresql locks tables one at a time. Nevertheless, the crawler ui may become unresponsive for some operations, such as when counting outstanding documents on the job status page. ManifoldCF thus has the ability to check for the existence of a file prior to such sensitive operations, and will display a useful "maintenance in progress" message if that file is found. This allows a user to set up a maintenance system that provides adequate feedback for an ManifoldCF user of the overall status of the system.</p>
+<p></p>
+<a name="N10432"></a><a name="The+ManifoldCF+configuration+file"></a>
+<h4>The ManifoldCF configuration file</h4>
+<p></p>
+<p>Currently, ManifoldCF requires two configuration files: the main configuration property file, and the logging configuration file.</p>
+<p></p>
+<p>The property file path can be specified by the system property "org.apache.manifoldcf.configfile". If not specified through a -D operation, its name is presumed to be <em><user_home>/lcf/properties.xml</em>. The form of the property file is XML, of the following basic form:</p>
+<p></p>
+<pre class="code">
+<?xml version="1.0" encoding="UTF-8" ?>
+<configuration>
+ (clauses)
+</configuration>
+ </pre>
+<p></p>
+<a name="N1044E"></a><a name="Properties"></a>
+<h4>Properties</h4>
+<p></p>
+<p>The configuration file allows properties to be specified. A property clause has the form:</p>
+<p></p>
+<p><property name="<em>property_name</em>" value="<em>property_value</em>"/></p>
+<p></p>
+<p>One of the optional properties is the name of the logging configuration file. This property's name is "org.apache.manifoldcf.logconfigfile". If not present, the logging configuration file will be assumed to be <em><user_home>/manifoldcf/logging.ini</em>. The logging configuration file is a standard commons-logging property file, and should be formatted accordingly.</p>
+<p></p>
+<p>Note that all properties described below can also be specified on the command line, via a -D switch. If both methods of setting the property are used, the -D switch value will override the property file value.</p>
+<p></p>
+<p>The following table describes the configuration property file properties, and what they do:</p>
+<p></p>
+<table class="ForrestTable" cellspacing="1" cellpadding="4">
+
+<caption>Property.xml properties</caption>
+
+<tr>
+<th colspan="1" rowspan="1">Property</th><th colspan="1" rowspan="1">Required?</th><th colspan="1" rowspan="1">Function</th>
+</tr>
+
+<tr>
+<td colspan="1" rowspan="1">org.apache.manifoldcf.lockmanagerclass</td><td colspan="1" rowspan="1">No</td><td colspan="1" rowspan="1">Specifies the class to use to implement synchronization. Default is a built-in file-based synchronization class.</td>
+</tr>
+
+<tr>
+<td colspan="1" rowspan="1">org.apache.manifoldcf.databaseimplementationclass</td><td colspan="1" rowspan="1">No</td><td colspan="1" rowspan="1">Specifies the class to use to implement database access. Default is a built-in Postgresql implementation.</td>
+</tr>
+
+<tr>
+<td colspan="1" rowspan="1">org.apache.manifoldcf.synchdirectory</td><td colspan="1" rowspan="1">Yes, if file-based synchronization class is used</td><td colspan="1" rowspan="1">Specifies the path of a synchronization directory. All ManifoldCF process owners <strong>must</strong> have read/write privileges to this directory.</td>
+</tr>
+
+<tr>
+<td colspan="1" rowspan="1">org.apache.manifoldcf.database.maxhandles</td><td colspan="1" rowspan="1">No</td><td colspan="1" rowspan="1">Specifies the maximum number of database connection handles that will by pooled. Recommended value is 200.</td>
+</tr>
+
+<tr>
+<td colspan="1" rowspan="1">org.apache.manifoldcf.database.handletimeout</td><td colspan="1" rowspan="1">No</td><td colspan="1" rowspan="1">Specifies the maximum time a handle is to live before it is presumed dead. Recommend a value of 604800, which is the maximum allowable.</td>
+</tr>
+
+<tr>
+<td colspan="1" rowspan="1">org.apache.manifoldcf.logconfigfile</td><td colspan="1" rowspan="1">No</td><td colspan="1" rowspan="1">Specifies location of logging configuration file.</td>
+</tr>
+
+<tr>
+<td colspan="1" rowspan="1">org.apache.manifoldcf.database.name</td><td colspan="1" rowspan="1">No</td><td colspan="1" rowspan="1">Describes database name for ManifoldCF; defaults to "dbname" if not specified.</td>
+</tr>
+
+<tr>
+<td colspan="1" rowspan="1">org.apache.manifoldcf.database.username</td><td colspan="1" rowspan="1">No</td><td colspan="1" rowspan="1">Describes database user name for ManifoldCF; defaults to "manifoldcf" if not specified.</td>
+</tr>
+
+<tr>
+<td colspan="1" rowspan="1">org.apache.manifoldcf.database.password</td><td colspan="1" rowspan="1">No</td><td colspan="1" rowspan="1">Describes database user's password for ManifoldCF; defaults to "local_pg_password" if not specified.</td>
+</tr>
+
+<tr>
+<td colspan="1" rowspan="1">org.apache.manifoldcf.crawler.threads</td><td colspan="1" rowspan="1">No</td><td colspan="1" rowspan="1">Number of crawler worker threads created. Suggest a value of 30.</td>
+</tr>
+
+<tr>
+<td colspan="1" rowspan="1">org.apache.manifoldcf.crawler.deletethreads</td><td colspan="1" rowspan="1">No</td><td colspan="1" rowspan="1">Number of crawler delete threads created. Suggest a value of 10.</td>
+</tr>
+
+<tr>
+<td colspan="1" rowspan="1">org.apache.manifoldcf.misc</td><td colspan="1" rowspan="1">No</td><td colspan="1" rowspan="1">Miscellaneous debugging output. Legal values INFO, WARN, or DEBUG.</td>
+</tr>
+
+<tr>
+<td colspan="1" rowspan="1">org.apache.manifoldcf.db</td><td colspan="1" rowspan="1">No</td><td colspan="1" rowspan="1">Database debugging output. Legal values INFO, WARN, or DEBUG.</td>
+</tr>
+
+<tr>
+<td colspan="1" rowspan="1">org.apache.manifoldcf.lock</td><td colspan="1" rowspan="1">No</td><td colspan="1" rowspan="1">Lock management debugging output. Legal values INFO, WARN, or DEBUG.</td>
+</tr>
+
+<tr>
+<td colspan="1" rowspan="1">org.apache.manifoldcf.cache</td><td colspan="1" rowspan="1">No</td><td colspan="1" rowspan="1">Cache management debugging output. Legal values INFO, WARN, or DEBUG.</td>
+</tr>
+
+<tr>
+<td colspan="1" rowspan="1">org.apache.manifoldcf.agents</td><td colspan="1" rowspan="1">No</td><td colspan="1" rowspan="1">Agent management debugging output. Legal values INFO, WARN, or DEBUG.</td>
+</tr>
+
+<tr>
+<td colspan="1" rowspan="1">org.apache.manifoldcf.perf</td><td colspan="1" rowspan="1">No</td><td colspan="1" rowspan="1">Performance logging debugging output. Legal values INFO, WARN, or DEBUG.</td>
+</tr>
+
+<tr>
+<td colspan="1" rowspan="1">org.apache.manifoldcf.crawlerthreads</td><td colspan="1" rowspan="1">No</td><td colspan="1" rowspan="1">Log crawler thread activity. Legal values INFO, WARN, or DEBUG.</td>
+</tr>
+
+<tr>
+<td colspan="1" rowspan="1">org.apache.manifoldcf.hopcount</td><td colspan="1" rowspan="1">No</td><td colspan="1" rowspan="1">Log hopcount tracking activity. Legal values INFO, WARN, or DEBUG.</td>
+</tr>
+
+<tr>
+<td colspan="1" rowspan="1">org.apache.manifoldcf.jobs</td><td colspan="1" rowspan="1">No</td><td colspan="1" rowspan="1">Log job activity. Legal values INFO, WARN, or DEBUG.</td>
+</tr>
+
+<tr>
+<td colspan="1" rowspan="1">org.apache.manifoldcf.connectors</td><td colspan="1" rowspan="1">No</td><td colspan="1" rowspan="1">Log connector activity. Legal values INFO, WARN, or DEBUG.</td>
+</tr>
+
+<tr>
+<td colspan="1" rowspan="1">org.apache.manifoldcf.scheduling</td><td colspan="1" rowspan="1">No</td><td colspan="1" rowspan="1">Log document scheduling activity. Legal values INFO, WARN, or DEBUG.</td>
+</tr>
+
+<tr>
+<td colspan="1" rowspan="1">org.apache.manifoldcf.authorityconnectors</td><td colspan="1" rowspan="1">No</td><td colspan="1" rowspan="1">Log authority connector activity. Legal values INFO, WARN, or DEBUG.</td>
+</tr>
+
+<tr>
+<td colspan="1" rowspan="1">org.apache.manifoldcf.authorityservice</td><td colspan="1" rowspan="1">No</td><td colspan="1" rowspan="1">Log authority service activity. Legal values are INFO, WARN, or DEBUG.</td>
+</tr>
+
+<tr>
+<td colspan="1" rowspan="1">org.apache.manifoldcf.sharepoint.wsddpath</td><td colspan="1" rowspan="1">Yes, for SharePoint Connector</td><td colspan="1" rowspan="1">Path to the SharePoint Connector wsdd file.</td>
+</tr>
+
+<tr>
+<td colspan="1" rowspan="1">org.apache.manifoldcf.meridio.wsddpath</td><td colspan="1" rowspan="1">Yes, for Meridio Connector</td><td colspan="1" rowspan="1">Path to the Meridio Connector wsdd file.</td>
+</tr>
+
+</table>
+<p></p>
+<a name="N105FE"></a><a name="Class+path+libraries"></a>
+<h4>Class path libraries</h4>
+<p></p>
+<p>The configuration file can also specify a set of directories which will be searched for connector jars. The directive that adds to the class path is:</p>
+<p></p>
+<p><libdir path="<em>path</em>"/></p>
+<p></p>
+<p>Note that the path can be relative. For the purposes of path resolution, "." means the directory in which the properties.xml file is located.</p>
+<p></p>
+<a name="N10619"></a><a name="Examples"></a>
+<h4>Examples</h4>
+<p></p>
+<p>An example properties file might be:</p>
+<p></p>
+<pre class="code">
+<?xml version="1.0" encoding="UTF-8" ?>
+<configuration>
+ <property name="org.apache.manifoldcf.synchdirectory" value="c:/mysynchdir"/>
+ <property name="org.apache.manifoldcf.logconfigfile" value="c:/conf/logging.ini"/>
+ <libdir path="./lib"/>
+</configuration>
+ </pre>
+<p></p>
+<p>An example simple logging configuration file might be:</p>
+<p></p>
+<pre class="code">
+# Set the default log level and parameters
+# This gets inherited by all child loggers
+log4j.rootLogger=WARN, MAIN
+
+log4j.additivity.org.apache=false
+
+log4j.appender.MAIN=org.apache.log4j.RollingFileAppender
+log4j.appender.MAIN.File=c:/dataarea/manifoldcf.log
+log4j.appender.MAIN.MaxFileSize=50MB
+log4j.appender.MAIN.MaxBackupIndex=10
+log4j.appender.MAIN.layout=org.apache.log4j.PatternLayout
+log4j.appender.MAIN.layout.ConversionPattern=[%d]%-5p %m%n
+ </pre>
+<p></p>
+<p></p>
+<p></p>
+<a name="N1063C"></a><a name="Commands"></a>
+<h4>Commands</h4>
+<p></p>
+<p>After you have created the necessary configuration files, you will need to initialize the database, register the "pull-agent" agent, and then register your individual connectors. ManifoldCF provides a set of commands for performing these actions, and others as well. The classes implementing these commands are specified below.</p>
+<p></p>
+<table class="ForrestTable" cellspacing="1" cellpadding="4">
+
+<tr>
+<th colspan="1" rowspan="1">Core Command Class</th><th colspan="1" rowspan="1">Arguments</th><th colspan="1" rowspan="1">Function</th>
+</tr>
+
+<tr>
+<td colspan="1" rowspan="1">org.apache.manifoldcf.core.DBCreate</td><td colspan="1" rowspan="1"><em>dbuser</em> [<em>dbpassword</em>]</td><td colspan="1" rowspan="1">Create ManifoldCF database instance</td>
+</tr>
+
+<tr>
+<td colspan="1" rowspan="1">org.apache.manifoldcf.core.DBDrop</td><td colspan="1" rowspan="1"><em>dbuser</em> [<em>dbpassword</em>]</td><td colspan="1" rowspan="1">Drop ManifoldCF database instance</td>
+</tr>
+
+<tr>
+<td colspan="1" rowspan="1">org.apache.manifoldcf.core.LockClean</td><td colspan="1" rowspan="1">None</td><td colspan="1" rowspan="1">Clean out synchronization directory</td>
+</tr>
+
+</table>
+<p></p>
+<table class="ForrestTable" cellspacing="1" cellpadding="4">
+
+<tr>
+<th colspan="1" rowspan="1">Agents Command Class</th><th colspan="1" rowspan="1">Arguments</th><th colspan="1" rowspan="1">Function</th>
+</tr>
+
+<tr>
+<td colspan="1" rowspan="1">org.apache.manifoldcf.agents.Install</td><td colspan="1" rowspan="1">None</td><td colspan="1" rowspan="1">Create ManifoldCF agents tables</td>
+</tr>
+
+<tr>
+<td colspan="1" rowspan="1">org.apache.manifoldcf.agents.Uninstall</td><td colspan="1" rowspan="1">None</td><td colspan="1" rowspan="1">Remove ManifoldCF agents tables</td>
+</tr>
+
+<tr>
+<td colspan="1" rowspan="1">org.apache.manifoldcf.agents.Register</td><td colspan="1" rowspan="1"><em>classname</em></td><td colspan="1" rowspan="1">Register an agent class</td>
+</tr>
+
+<tr>
+<td colspan="1" rowspan="1">org.apache.manifoldcf.agents.UnRegister</td><td colspan="1" rowspan="1"><em>classname</em></td><td colspan="1" rowspan="1">Un-register an agent class</td>
+</tr>
+
+<tr>
+<td colspan="1" rowspan="1">org.apache.manifoldcf.agents.UnRegisterAll</td><td colspan="1" rowspan="1">None</td><td colspan="1" rowspan="1">Un-register all current agent classes</td>
+</tr>
+
+<tr>
+<td colspan="1" rowspan="1">org.apache.manifoldcf.agents.SynchronizeAll</td><td colspan="1" rowspan="1">None</td><td colspan="1" rowspan="1">Un-register all registered agent classes that can't be found</td>
+</tr>
+
+<tr>
+<td colspan="1" rowspan="1">org.apache.manifoldcf.agents.RegisterOutput</td><td colspan="1" rowspan="1"><em>classname</em> <em>description</em></td><td colspan="1" rowspan="1">Register an output connector class</td>
+</tr>
+
+<tr>
+<td colspan="1" rowspan="1">org.apache.manifoldcf.agents.UnRegisterOutput</td><td colspan="1" rowspan="1"><em>classname</em></td><td colspan="1" rowspan="1">Un-register an output connector class</td>
+</tr>
+
+<tr>
+<td colspan="1" rowspan="1">org.apache.manifoldcf.agents.UnRegisterAllOutputs</td><td colspan="1" rowspan="1">None</td><td colspan="1" rowspan="1">Un-register all current output connector classes</td>
+</tr>
+
+<tr>
+<td colspan="1" rowspan="1">org.apache.manifoldcf.agents.SynchronizeOutputs</td><td colspan="1" rowspan="1">None</td><td colspan="1" rowspan="1">Un-register all registered output connector classes that can't be found</td>
+</tr>
+
+<tr>
+<td colspan="1" rowspan="1">org.apache.manifoldcf.agents.AgentRun</td><td colspan="1" rowspan="1">None</td><td colspan="1" rowspan="1">Main <strong>agents</strong> process class</td>
+</tr>
+
+<tr>
+<td colspan="1" rowspan="1">org.apache.manifoldcf.agents.AgentStop</td><td colspan="1" rowspan="1">None</td><td colspan="1" rowspan="1">Stops the running <strong>agents</strong> process</td>
+</tr>
+
+</table>
+<p></p>
+<table class="ForrestTable" cellspacing="1" cellpadding="4">
+
+<tr>
+<th colspan="1" rowspan="1">Crawler Command Class</th><th colspan="1" rowspan="1">Arguments</th><th colspan="1" rowspan="1">Function</th>
+</tr>
+
+<tr>
+<td colspan="1" rowspan="1">org.apache.manifoldcf.crawler.Register</td><td colspan="1" rowspan="1"><em>classname</em> <em>description</em></td><td colspan="1" rowspan="1">Register a repository connector class</td>
+</tr>
+
+<tr>
+<td colspan="1" rowspan="1">org.apache.manifoldcf.crawler.UnRegister</td><td colspan="1" rowspan="1"><em>classname</em></td><td colspan="1" rowspan="1">Un-register a repository connector class</td>
+</tr>
+
+<tr>
+<td colspan="1" rowspan="1">org.apache.manifoldcf.crawler.UnRegisterAll</td><td colspan="1" rowspan="1">None</td><td colspan="1" rowspan="1">Un-register all repository connector classes</td>
+</tr>
+
+<tr>
+<td colspan="1" rowspan="1">org.apache.manifoldcf.crawler.SynchronizeConnectors</td><td colspan="1" rowspan="1">None</td><td colspan="1" rowspan="1">Un-register all registered repository connector classes that can't be found</td>
+</tr>
+
+<tr>
+<td colspan="1" rowspan="1">org.apache.manifoldcf.crawler.ExportConfiguration</td><td colspan="1" rowspan="1"><em>filename</em></td><td colspan="1" rowspan="1">Export crawler configuration to a file</td>
+</tr>
+
+<tr>
+<td colspan="1" rowspan="1">org.apache.manifoldcf.crawler.ImportConfiguration</td><td colspan="1" rowspan="1"><em>filename</em></td><td colspan="1" rowspan="1">Import crawler configuration from a file</td>
+</tr>
+
+</table>
+<p></p>
+<table class="ForrestTable" cellspacing="1" cellpadding="4">
+
+<tr>
+<th colspan="1" rowspan="1">Authority Command Class</th><th colspan="1" rowspan="1">Arguments</th><th colspan="1" rowspan="1">Function</th>
+</tr>
+
+<tr>
+<td colspan="1" rowspan="1">org.apache.manifoldcf.authorities.RegisterAuthority</td><td colspan="1" rowspan="1"><em>classname</em> <em>description</em></td><td colspan="1" rowspan="1">Register an authority connector class</td>
+</tr>
+
+<tr>
+<td colspan="1" rowspan="1">org.apache.manifoldcf.authorities.UnRegisterAuthority</td><td colspan="1" rowspan="1"><em>classname</em></td><td colspan="1" rowspan="1">Un-register an authority connector class</td>
+</tr>
+
+<tr>
+<td colspan="1" rowspan="1">org.apache.manifoldcf.authorities.UnRegisterAllAuthorities</td><td colspan="1" rowspan="1">None</td><td colspan="1" rowspan="1">Un-register all authority connector classes</td>
+</tr>
+
+<tr>
+<td colspan="1" rowspan="1">org.apache.manifoldcf.authorities.SynchronizeAuthorities</td><td colspan="1" rowspan="1">None</td><td colspan="1" rowspan="1">Un-register all registered authority connector classes that can't be found</td>
+</tr>
+
+</table>
+<p></p>
+<p>Remember that you need to include all the jars under <em>module/dist/processes</em> in the classpath whenever you run one of these commands! You also must include the corresponding -D switches, as described earlier.</p>
+<p></p>
+<a name="N1081F"></a><a name="Initializing+the+database"></a>
+<h4>Initializing the database</h4>
+<p></p>
+<p>These are some of the commands you will need to use to create the database instance, initialize the schema, and register all of the appropriate components:</p>
+<p></p>
+<table class="ForrestTable" cellspacing="1" cellpadding="4">
+
+<tr>
+<th colspan="1" rowspan="1">Command</th><th colspan="1" rowspan="1">Arguments</th>
+</tr>
+
+<tr>
+<td colspan="1" rowspan="1">org.apache.manifoldcf.core.DBCreate</td><td colspan="1" rowspan="1">postgres postgres</td>
+</tr>
+
+<tr>
+<td colspan="1" rowspan="1">org.apache.manifoldcf.agents.Install</td><td colspan="1" rowspan="1"></td>
+</tr>
+
+<tr>
+<td colspan="1" rowspan="1">org.apache.manifoldcf.agents.Register</td><td colspan="1" rowspan="1">org.apache.manifoldcf.crawler.system.CrawlerAgent</td>
+</tr>
+
+<tr>
+<td colspan="1" rowspan="1">org.apache.manifoldcf.agents.RegisterOutput</td><td colspan="1" rowspan="1">org.apache.manifoldcf.agents.output.gts.GTSConnector "GTS Connector"</td>
+</tr>
+
+<tr>
+<td colspan="1" rowspan="1">org.apache.manifoldcf.agents.RegisterOutput</td><td colspan="1" rowspan="1">org.apache.manifoldcf.agents.output.solr.SolrConnector "SOLR Connector"</td>
+</tr>
+
+<tr>
+<td colspan="1" rowspan="1">org.apache.manifoldcf.agents.RegisterOutput</td><td colspan="1" rowspan="1">org.apache.manifoldcf.agents.output.nullconnector.NullConnector "Null Connector"</td>
+</tr>
+
+<tr>
+<td colspan="1" rowspan="1">org.apache.manifoldcf.authorities.RegisterAuthority</td><td colspan="1" rowspan="1">org.apache.manifoldcf.authorities.authorities.activedirectory.ActiveDirectoryAuthority "Active Directory Authority"</td>
+</tr>
+
+<tr>
+<td colspan="1" rowspan="1">org.apache.manifoldcf.crawler.Register</td><td colspan="1" rowspan="1">org.apache.manifoldcf.crawler.connectors.DCTM.DCTM "Documentum Connector"</td>
+</tr>
+
+<tr>
+<td colspan="1" rowspan="1">org.apache.manifoldcf.authorities.RegisterAuthority</td><td colspan="1" rowspan="1">org.apache.manifoldcf.crawler.authorities.DCTM.AuthorityConnector "Documentum Authority"</td>
+</tr>
+
+<tr>
+<td colspan="1" rowspan="1">org.apache.manifoldcf.crawler.Register</td><td colspan="1" rowspan="1">org.apache.manifoldcf.crawler.connectors.filenet.FilenetConnector "FileNet Connector"</td>
+</tr>
+
+<tr>
+<td colspan="1" rowspan="1">org.apache.manifoldcf.crawler.Register</td><td colspan="1" rowspan="1">org.apache.manifoldcf.crawler.connectors.filesystem.FileConnector "Filesystem Connector"</td>
+</tr>
+
+<tr>
+<td colspan="1" rowspan="1">org.apache.manifoldcf.crawler.Register</td><td colspan="1" rowspan="1">org.apache.manifoldcf.crawler.connectors.jdbc.JDBCConnector "Database Connector"</td>
+</tr>
+
+<tr>
+<td colspan="1" rowspan="1">org.apache.manifoldcf.crawler.Register</td><td colspan="1" rowspan="1">org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector "Windows Share Connector"</td>
+</tr>
+
+<tr>
+<td colspan="1" rowspan="1">org.apache.manifoldcf.crawler.Register</td><td colspan="1" rowspan="1">org.apache.manifoldcf.crawler.connectors.livelink.LivelinkConnector "LiveLink Connector"</td>
+</tr>
+
+<tr>
+<td colspan="1" rowspan="1">org.apache.manifoldcf.authorities.RegisterAuthority</td><td colspan="1" rowspan="1">org.apache.manifoldcf.crawler.connectors.livelink.LivelinkAuthority "LiveLink Authority"</td>
+</tr>
+
+<tr>
+<td colspan="1" rowspan="1">org.apache.manifoldcf.crawler.Register</td><td colspan="1" rowspan="1">org.apache.manifoldcf.crawler.connectors.memex.MemexConnector "Memex Connector"</td>
+</tr>
+
+<tr>
+<td colspan="1" rowspan="1">org.apache.manifoldcf.authorities.RegisterAuthority</td><td colspan="1" rowspan="1">org.apache.manifoldcf.crawler.connectors.memex.MemexAuthority "Memex Authority"</td>
+</tr>
+
+<tr>
+<td colspan="1" rowspan="1">org.apache.manifoldcf.crawler.Register</td><td colspan="1" rowspan="1">org.apache.manifoldcf.crawler.connectors.meridio.MeridioConnector "Meridio Connector"</td>
+</tr>
+
+<tr>
+<td colspan="1" rowspan="1">org.apache.manifoldcf.authorities.RegisterAuthority</td><td colspan="1" rowspan="1">org.apache.manifoldcf.crawler.connectors.meridio.MemexAuthority "Meridio Authority"</td>
+</tr>
+
+<tr>
+<td colspan="1" rowspan="1">org.apache.manifoldcf.crawler.Register</td><td colspan="1" rowspan="1">org.apache.manifoldcf.crawler.connectors.rss.RSSConnector "RSS Connector"</td>
+</tr>
+
+<tr>
+<td colspan="1" rowspan="1">org.apache.manifoldcf.crawler.Register</td><td colspan="1" rowspan="1">org.apache.manifoldcf.crawler.connectors.sharepoint.SharePointRepository "SharePoint Connector"</td>
+</tr>
+
+<tr>
+<td colspan="1" rowspan="1">org.apache.manifoldcf.crawler.Register</td><td colspan="1" rowspan="1">org.apache.manifoldcf.crawler.connectors.webcrawler.WebcrawlerConnector "Web Connector"</td>
+</tr>
+
+</table>
+<p></p>
+<a name="N10917"></a><a name="Deploying+the"></a>
+<h4>Deploying the mcf-crawler-ui, mcf-authority-service, and mcf-api-service web applications</h4>
+<p></p>
+<p>If you built ManifoldCF using ant under the <em>modules</em> directory, then the ant build will have constructed three war files for you under <em>modules/dist/web</em>. Take these war files and deploy them as web applications under one or more instances of your application server. There is no requirement that the <strong>mcf-crawler-ui</strong>, <strong>mcf-authority-service</strong>, and <strong>mcf-api-service</strong> web applications be deployed on the same instance of the application server. With the current architecture of ManifoldCF, they must be deployed on the same physical server, however.</p>
+<p></p>
+<p>Under <em>modules/dist/web</em>, you may also see files that are not war files. These files are meant to be used as command-line -D switches for the application server process. The switches may or may not be identical for the two web applications, but they will never conflict. You may need to alter environment variables or your application server startup scripts in order to provide these switches. (More about this in the future...)</p>
+<p></p>
+<a name="N10945"></a><a name="Running+the"></a>
+<h4>Running the agents process</h4>
+<p></p>
+<p>The <strong>agents</strong> process is the process that actually performs the crawling for ManifoldCF. Start this process by running the command "org.apache.manifoldcf.agents.AgentRun". This class will run until stopped by invoking the command "org.apache.manifoldcf.agents.AgentStop". It is highly recommended that you stop the process in this way. You may also stop the process using a SIGTERM signal, but "kill -9" or the equivalent is NOT recommended, because that may result in dangling locks in the ManifoldCF synchronization directory. (If you have to, clean up these locks by shutting down all ManifoldCF processes, including the application server instances that are running the web applications, and invoking the command "org.apache.manifoldcf.core.LockClean".)</p>
+<p></p>
+<a name="N10959"></a><a name="Running+connector-specific+processes"></a>
+<h4>Running connector-specific processes</h4>
+<p></p>
+<p>Connector-specific processes require the classpath for their invocation to include all the jars that are in the corresponding <em>modules/dist/<process_name>-process</em> directory. The Documentum and FileNet connectors are the only two connectors that currently require additional processes. Start these processes using the commands listed below, and stop them with SIGTERM.</p>
+<p></p>
+<table class="ForrestTable" cellspacing="1" cellpadding="4">
+
+<tr>
+<th colspan="1" rowspan="1">Connector</th><th colspan="1" rowspan="1">Process</th><th colspan="1" rowspan="1">Start class</th>
+</tr>
+
+<tr>
+<td colspan="1" rowspan="1">Documentum</td><td colspan="1" rowspan="1">documentum-server-process</td><td colspan="1" rowspan="1">org.apache.manifoldcf.crawler.server.DCTM.DCTM</td>
+</tr>
+
+<tr>
+<td colspan="1" rowspan="1">Documentum</td><td colspan="1" rowspan="1">documentum-registry-process</td><td colspan="1" rowspan="1">org.apache.manifoldcf.crawler.registry.DCTM.DCTM</td>
+</tr>
+
+<tr>
+<td colspan="1" rowspan="1">FileNet</td><td colspan="1" rowspan="1">filenet-server-process</td><td colspan="1" rowspan="1">org.apache.manifoldcf.crawler.server.filenet.Filenet</td>
+</tr>
+
+<tr>
+<td colspan="1" rowspan="1">FileNet</td><td colspan="1" rowspan="1">filenet-registry-process</td><td colspan="1" rowspan="1">org.apache.manifoldcf.crawler.registry.filenet.Filenet</td>
+</tr>
+
+</table>
+<p></p>
+<a name="N109B6"></a><a name="Running+the+ManifoldCF+Apache2+plug+in"></a>
+<h3 class="h4">Running the ManifoldCF Apache2 plug in</h3>
+<p></p>
+<p>The ManifoldCF Apache2 plugin, mod-authz-annotate, is designed to convert an authenticated principle (e.g. from mod-auth-kerb), and query a set of authority services for access tokens using an HTTP request. These access tokens are then passed to a (not included) search engine UI, which can use them to help compose a search that properly excludes content that the user is not supposed to see.</p>
+<p></p>
+<p>The list of authority services so queried is configured in Apache's httpd.conf file. This project includes only one such service: the java authority service, which uses authority connections defined in the crawler UI to obtain appropriate access tokens.</p>
+<p></p>
+<p>In order for mod-authz-annotate to be used, it must be placed into Apache2's extensions directory, and configured appropriately in the httpd.conf file.</p>
+<p></p>
+<p>Note: The ManifoldCF project now contains support for converting a Kerberos principal to a list of Active Directory SIDs. This functionality is contained in the Active Directory Authority. The following connectors are expected to make use of this authority:</p>
+<p></p>
+<ul>
+
+<li>FileNet</li>
+
+<li>Meridio</li>
+
+<li>SharePoint</li>
+
+</ul>
+<p></p>
+<a name="N109E0"></a><a name="Configuring+the+ManifoldCF+Apache2+plug+in"></a>
+<h4>Configuring the ManifoldCF Apache2 plug in</h4>
+<p></p>
+<p>mod-authz-annotate understands the following httpd.conf commands:</p>
+<p></p>
+<table class="ForrestTable" cellspacing="1" cellpadding="4">
+
+<tr>
+<th colspan="1" rowspan="1">Command</th><th colspan="1" rowspan="1">Meaning</th><th colspan="1" rowspan="1">Values</th>
+</tr>
+
+<tr>
+<td colspan="1" rowspan="1">AuthzAnnotateEnable</td><td colspan="1" rowspan="1">Turn on/off the plugin</td><td colspan="1" rowspan="1">"On", "Off"</td>
+</tr>
+
+<tr>
+<td colspan="1" rowspan="1">AuthzAnnotateAuthority</td><td colspan="1" rowspan="1">Point to an authority service that supports ACL queries, but not ID queries</td><td colspan="1" rowspan="1">The authority URL</td>
+</tr>
+
+<tr>
+<td colspan="1" rowspan="1">AuthzAnnotateACLAuthority</td><td colspan="1" rowspan="1">Point to an authority service that supports ACL queries, but not ID queries</td><td colspan="1" rowspan="1">The authority URL</td>
+</tr>
+
+<tr>
+<td colspan="1" rowspan="1">AuthzAnnotateIDAuthority</td><td colspan="1" rowspan="1">Point to an authority service that supports ID queries, but not ACL queries</td><td colspan="1" rowspan="1">The authority URL</td>
+</tr>
+
+<tr>
+<td colspan="1" rowspan="1">AuthzAnnotateIDACLAuthority</td><td colspan="1" rowspan="1">Point to an authority service that supports both ACL queries and ID queries</td><td colspan="1" rowspan="1">The authority URL</td>
+</tr>
+
+</table>
+</div>
+
+</div>
+<!--+
+ |end content
+ +-->
+<div class="clearboth"> </div>
+</div>
+<div id="footer">
+<!--+
+ |start bottomstrip
+ +-->
+<div class="lastmodified">
+<script type="text/javascript"><!--
+document.write("Last Published: " + document.lastModified);
+// --></script>
+</div>
+<div class="copyright">
+ Copyright ©
+ 2009, 2010 <a href="http://www.apache.org/licenses/">The Apache Software Foundation.</a>
+</div>
+<!--+
+ |end bottomstrip
+ +-->
+</div>
+</body>
+</html>
Added: incubator/lcf/site/publish/how-to-build-and-deploy.pdf
URL: http://svn.apache.org/viewvc/incubator/lcf/site/publish/how-to-build-and-deploy.pdf?rev=1034577&view=auto
==============================================================================
Binary file - no diff available.
Propchange: incubator/lcf/site/publish/how-to-build-and-deploy.pdf
------------------------------------------------------------------------------
svn:mime-type = application/octet-stream
Modified: incubator/lcf/site/publish/index.pdf
URL: http://svn.apache.org/viewvc/incubator/lcf/site/publish/index.pdf?rev=1034577&r1=1034576&r2=1034577&view=diff
==============================================================================
Binary files - no diff available.
Modified: incubator/lcf/site/publish/linkmap.pdf
URL: http://svn.apache.org/viewvc/incubator/lcf/site/publish/linkmap.pdf?rev=1034577&r1=1034576&r2=1034577&view=diff
==============================================================================
Binary files - no diff available.
Modified: incubator/lcf/site/publish/mail.pdf
URL: http://svn.apache.org/viewvc/incubator/lcf/site/publish/mail.pdf?rev=1034577&r1=1034576&r2=1034577&view=diff
==============================================================================
Binary files - no diff available.
Modified: incubator/lcf/site/publish/who.pdf
URL: http://svn.apache.org/viewvc/incubator/lcf/site/publish/who.pdf?rev=1034577&r1=1034576&r2=1034577&view=diff
==============================================================================
Binary files - no diff available.
Modified: incubator/lcf/site/src/documentation/content/xdocs/developer-resources.xml
URL: http://svn.apache.org/viewvc/incubator/lcf/site/src/documentation/content/xdocs/developer-resources.xml?rev=1034577&r1=1034576&r2=1034577&view=diff
==============================================================================
--- incubator/lcf/site/src/documentation/content/xdocs/developer-resources.xml (original)
+++ incubator/lcf/site/src/documentation/content/xdocs/developer-resources.xml Fri Nov 12 21:33:20 2010
@@ -22,7 +22,7 @@
<section id="howtobuilddeploy">
<title>How to Build and Deploy</title>
- <p>Building ManifoldCF requires ant 1.7 or greater. Instructions for building and deploying can be found <a href="http://cwiki.apache.org/confluence/display/CONNECTORS/How+to+Build+and+Deploy+ManifoldCF">here</a>.
+ <p>Building ManifoldCF requires ant 1.7 or greater. Instructions for building and deploying can be found <a href="how-to-build-and-deploy.html">here</a>.
</p>
</section>