You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@manifoldcf.apache.org by kw...@apache.org on 2013/05/02 21:35:37 UTC
svn commit: r1478500 - in /manifoldcf/trunk: ./
site/src/documentation/content/xdocs/en_US/
site/src/documentation/resources/images/en_US/
tests/webcrawler/src/test/java/org/apache/manifoldcf/webcrawler_tests/
Author: kwright
Date: Thu May 2 19:35:37 2013
New Revision: 1478500
URL: http://svn.apache.org/r1478500
Log:
Fix for CONNECTORS-683. Add DropBox connector documentation.
Added:
manifoldcf/trunk/site/src/documentation/resources/images/en_US/dropbox-repository-application-secret-passwords.PNG (with props)
manifoldcf/trunk/site/src/documentation/resources/images/en_US/dropbox-repository-connection-configuration-save.PNG (with props)
manifoldcf/trunk/site/src/documentation/resources/images/en_US/dropbox-repository-connection-configuration.PNG (with props)
manifoldcf/trunk/site/src/documentation/resources/images/en_US/dropbox-repository-connection-job-dropbox-folder-to-index.PNG (with props)
manifoldcf/trunk/site/src/documentation/resources/images/en_US/dropbox-repository-connection-job-save.PNG (with props)
manifoldcf/trunk/site/src/documentation/resources/images/en_US/dropbox-repository-create-application.PNG (with props)
Modified:
manifoldcf/trunk/CHANGES.txt
manifoldcf/trunk/site/src/documentation/content/xdocs/en_US/end-user-documentation.xml
manifoldcf/trunk/tests/webcrawler/src/test/java/org/apache/manifoldcf/webcrawler_tests/ThrottlingTester.java
Modified: manifoldcf/trunk/CHANGES.txt
URL: http://svn.apache.org/viewvc/manifoldcf/trunk/CHANGES.txt?rev=1478500&r1=1478499&r2=1478500&view=diff
==============================================================================
--- manifoldcf/trunk/CHANGES.txt (original)
+++ manifoldcf/trunk/CHANGES.txt Thu May 2 19:35:37 2013
@@ -3,6 +3,9 @@ $Id$
======================= 1.3-dev =====================
+CONNECTORS-683: Add Dropbox connector end-user documentation.
+(Andrew Janowczyk)
+
======================= Release 1.2 =====================
CONNECTORS-586: Fix native2ascii incompatibility with Java 7.
Modified: manifoldcf/trunk/site/src/documentation/content/xdocs/en_US/end-user-documentation.xml
URL: http://svn.apache.org/viewvc/manifoldcf/trunk/site/src/documentation/content/xdocs/en_US/end-user-documentation.xml?rev=1478500&r1=1478499&r2=1478500&view=diff
==============================================================================
--- manifoldcf/trunk/site/src/documentation/content/xdocs/en_US/end-user-documentation.xml (original)
+++ manifoldcf/trunk/site/src/documentation/content/xdocs/en_US/end-user-documentation.xml Thu May 2 19:35:37 2013
@@ -1628,6 +1628,54 @@
<p>If more than one rule is present, the rules are all executed in sequence. That is, the output of the first rule is modified by the second rule, etc.</p>
</section>
+ <section id="dropboxrepository">
+ <title>Dropbox Repository Connection</title>
+ <p>The Dropbox Repository Connection type allows you to index content from <a href="https://www.dropbox.com/home">Dropbox</a>.</p>
+ <p>Each Dropbox Connection manages access to a single dropbox repository. This means that if you have multiple dropbox repositories (i.e. different users),
+ you need to create a specific connection for each dropbox repository and provide the associated authentication information.</p>
+ <br/>
+ <p>A Dropbox connection has the following configuration parameters on the repository connection editing screen:</p>
+ <br/><br/>
+ <figure src="images/en_US/dropbox-repository-connection-configuration.PNG" alt="Dropbox Repository Connection, configuration parameters" width="80%"/>
+ <br/><br/>
+ <p>As we can see there are 4 pieces of information which are needed to create a succesful connection. The application key and secret are given by dropbox
+ when you register your application for a development license. This is typically done through the application developer <a href="https://www.dropbox.com/developers/apps">Dropbox website</a>.</p>
+ <br/><br/>
+ <figure src="images/en_US/dropbox-repository-create-application.PNG" alt="Dropbox create application" width="80%"/>
+ <br/><br/>
+ <p>For our purposes, we need to select "Core" as the application type as we'll be using REST services to communicate with dropbox. Also we select
+ "full access". This merits a small discussion. Typically an application which wants to store and retreive information does so from an application specific
+ folder. In this case, we assume that the user wants to have access to their files as is, and not copy them into a manifoldcf specific folder. As a result,
+ we have selected full access instead of "App folder".</p>
+ <br/><br/>
+ <figure src="images/en_US/dropbox-repository-application-secret-passwords.PNG" alt="Dropbox get key and secret passwords" width="80%"/>
+ <br/><br/>
+ <p> Afterwards we can see the app key and app secret which are the two pieces of information requested by the connector.</p>
+ <p>Now each user must confirm their acceptance of allowing your application to access their dropbox. This is done through a run-of-the-mill OAUTH
+ approach. After providing your application key and secret, the user is directed to a dropbox website asking them if they wish to grant permission to your
+ application. After they accept the request, dropbox provides a client key and secret. These are the last two pieces of information needed for the dropbox
+ connector. This process is covered in depth at the <a href="https://www.dropbox.com/developers/core/authentication">dropbox website</a>
+ which shows examples of how to generate the two needed client tokens. </p>
+ <br/>
+ <br/><br/>
+ <p>After you click the "Save" button, you will see a connection summary screen, which might look something like this:</p>
+ <br/><br/>
+ <figure src="images/en_US/dropbox-repository-connection-configuration-save.PNG" alt="Dropbox Repository Connection, saving configuration" width="80%"/>
+ <br/><br/>
+ <p>When you configure a job to use the Dropbox repository connection an additional tab is presented. This is the "Dropbox Folder to Index" tab:</p>
+ <br/><br/>
+ <figure src="images/en_US/dropbox-repository-connection-job-dropbox-folder-to-index.PNG" alt="Dropbox Repository Connection, Dropbox Folder to Index" width="80%"/>
+ <br/><br/>
+ <p>The Dropbox Folder to Index tab allows you to specify the directory which the dropbox connector will index. Dropbox uses unix style paths, with
+ "/" indicating the root path (and thus the entire dropbox). For example if you want to just index the Photos directory, you would specify "/Photos". </p>
+ <p>Note that the Dropbox Connector during the ingestion process, for each result, when it find a folder node, it will ingest all the children of the folder node;
+ otherwise it will directly ingest the document.</p>
+ <p>When you are done, and you click the "Save" button, you will see a summary page looking something like this:</p>
+ <br/><br/>
+ <figure src="images/en_US/dropbox-repository-connection-job-save.PNG" alt="CMIS Repository Connection, saving job" width="80%"/>
+ <br/><br/>
+ </section>
+
<section id="livelinkrepository">
<title>OpenText LiveLink Repository Connection</title>
<p>The LiveLink connection type allows you to index content from LiveLink repositories. LiveLink has a rich variety of different document types and metadata,
Added: manifoldcf/trunk/site/src/documentation/resources/images/en_US/dropbox-repository-application-secret-passwords.PNG
URL: http://svn.apache.org/viewvc/manifoldcf/trunk/site/src/documentation/resources/images/en_US/dropbox-repository-application-secret-passwords.PNG?rev=1478500&view=auto
==============================================================================
Binary file - no diff available.
Propchange: manifoldcf/trunk/site/src/documentation/resources/images/en_US/dropbox-repository-application-secret-passwords.PNG
------------------------------------------------------------------------------
svn:mime-type = application/octet-stream
Added: manifoldcf/trunk/site/src/documentation/resources/images/en_US/dropbox-repository-connection-configuration-save.PNG
URL: http://svn.apache.org/viewvc/manifoldcf/trunk/site/src/documentation/resources/images/en_US/dropbox-repository-connection-configuration-save.PNG?rev=1478500&view=auto
==============================================================================
Binary file - no diff available.
Propchange: manifoldcf/trunk/site/src/documentation/resources/images/en_US/dropbox-repository-connection-configuration-save.PNG
------------------------------------------------------------------------------
svn:mime-type = application/octet-stream
Added: manifoldcf/trunk/site/src/documentation/resources/images/en_US/dropbox-repository-connection-configuration.PNG
URL: http://svn.apache.org/viewvc/manifoldcf/trunk/site/src/documentation/resources/images/en_US/dropbox-repository-connection-configuration.PNG?rev=1478500&view=auto
==============================================================================
Binary file - no diff available.
Propchange: manifoldcf/trunk/site/src/documentation/resources/images/en_US/dropbox-repository-connection-configuration.PNG
------------------------------------------------------------------------------
svn:mime-type = application/octet-stream
Added: manifoldcf/trunk/site/src/documentation/resources/images/en_US/dropbox-repository-connection-job-dropbox-folder-to-index.PNG
URL: http://svn.apache.org/viewvc/manifoldcf/trunk/site/src/documentation/resources/images/en_US/dropbox-repository-connection-job-dropbox-folder-to-index.PNG?rev=1478500&view=auto
==============================================================================
Binary file - no diff available.
Propchange: manifoldcf/trunk/site/src/documentation/resources/images/en_US/dropbox-repository-connection-job-dropbox-folder-to-index.PNG
------------------------------------------------------------------------------
svn:mime-type = application/octet-stream
Added: manifoldcf/trunk/site/src/documentation/resources/images/en_US/dropbox-repository-connection-job-save.PNG
URL: http://svn.apache.org/viewvc/manifoldcf/trunk/site/src/documentation/resources/images/en_US/dropbox-repository-connection-job-save.PNG?rev=1478500&view=auto
==============================================================================
Binary file - no diff available.
Propchange: manifoldcf/trunk/site/src/documentation/resources/images/en_US/dropbox-repository-connection-job-save.PNG
------------------------------------------------------------------------------
svn:mime-type = application/octet-stream
Added: manifoldcf/trunk/site/src/documentation/resources/images/en_US/dropbox-repository-create-application.PNG
URL: http://svn.apache.org/viewvc/manifoldcf/trunk/site/src/documentation/resources/images/en_US/dropbox-repository-create-application.PNG?rev=1478500&view=auto
==============================================================================
Binary file - no diff available.
Propchange: manifoldcf/trunk/site/src/documentation/resources/images/en_US/dropbox-repository-create-application.PNG
------------------------------------------------------------------------------
svn:mime-type = application/octet-stream
Modified: manifoldcf/trunk/tests/webcrawler/src/test/java/org/apache/manifoldcf/webcrawler_tests/ThrottlingTester.java
URL: http://svn.apache.org/viewvc/manifoldcf/trunk/tests/webcrawler/src/test/java/org/apache/manifoldcf/webcrawler_tests/ThrottlingTester.java?rev=1478500&r1=1478499&r2=1478500&view=diff
==============================================================================
--- manifoldcf/trunk/tests/webcrawler/src/test/java/org/apache/manifoldcf/webcrawler_tests/ThrottlingTester.java (original)
+++ manifoldcf/trunk/tests/webcrawler/src/test/java/org/apache/manifoldcf/webcrawler_tests/ThrottlingTester.java Thu May 2 19:35:37 2013
@@ -133,7 +133,16 @@ public class ThrottlingTester
// Now, start the job, and wait until it completes.
long startTime = System.currentTimeMillis();
jobManager.manualStart(job.getID());
- instance.waitJobInactiveNative(jobManager,job.getID(),900000L);
+ try
+ {
+ instance.waitJobInactiveNative(jobManager,job.getID(),900000L);
+ }
+ catch (ManifoldCFException e)
+ {
+ System.err.println("Halting for inspection");
+ Thread.sleep(9000000L);
+ throw e;
+ }
System.err.println(" Crawl required "+new Long(System.currentTimeMillis()-startTime).toString()+" milliseconds");
// Check to be sure we actually processed the right number of documents.