You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@manifoldcf.apache.org by kw...@apache.org on 2013/06/24 08:55:18 UTC
svn commit: r1495934 - in /manifoldcf/trunk: ./ connectors/ connectors/jira/
connectors/livelink/connector/src/main/java/org/apache/manifoldcf/crawler/connectors/livelink/
connectors/rss/
connectors/rss/connector/src/main/java/org/apache/manifoldcf/cra...
Author: kwright
Date: Mon Jun 24 06:55:17 2013
New Revision: 1495934
URL: http://svn.apache.org/r1495934
Log:
Add Jira connector. Finishes CONNECTORS-723.
Added:
manifoldcf/trunk/connectors/jira/ (props changed)
- copied from r1495933, manifoldcf/branches/CONNECTORS-723/connectors/jira/
manifoldcf/trunk/framework/core/src/main/java/org/apache/manifoldcf/core/common/InterruptibleSocketFactory.java
- copied unchanged from r1495933, manifoldcf/branches/CONNECTORS-723/framework/core/src/main/java/org/apache/manifoldcf/core/common/InterruptibleSocketFactory.java
manifoldcf/trunk/site/src/documentation/resources/images/en_US/jira-repository-connection-configuration-save.PNG
- copied unchanged from r1495933, manifoldcf/branches/CONNECTORS-723/site/src/documentation/resources/images/en_US/jira-repository-connection-configuration-save.PNG
manifoldcf/trunk/site/src/documentation/resources/images/en_US/jira-repository-connection-configuration.PNG
- copied unchanged from r1495933, manifoldcf/branches/CONNECTORS-723/site/src/documentation/resources/images/en_US/jira-repository-connection-configuration.PNG
manifoldcf/trunk/site/src/documentation/resources/images/en_US/jira-repository-connection-job-jira-seed-query.PNG
- copied unchanged from r1495933, manifoldcf/branches/CONNECTORS-723/site/src/documentation/resources/images/en_US/jira-repository-connection-job-jira-seed-query.PNG
Modified:
manifoldcf/trunk/ (props changed)
manifoldcf/trunk/CHANGES.txt
manifoldcf/trunk/build.xml
manifoldcf/trunk/connectors/livelink/connector/src/main/java/org/apache/manifoldcf/crawler/connectors/livelink/LivelinkConnector.java
manifoldcf/trunk/connectors/pom.xml
manifoldcf/trunk/connectors/rss/ (props changed)
manifoldcf/trunk/connectors/rss/connector/src/main/java/org/apache/manifoldcf/crawler/connectors/rss/ThrottledFetcher.java
manifoldcf/trunk/connectors/webcrawler/connector/src/main/java/org/apache/manifoldcf/crawler/connectors/webcrawler/ThrottledFetcher.java
manifoldcf/trunk/connectors/wiki/ (props changed)
manifoldcf/trunk/connectors/wiki/connector/src/main/java/org/apache/manifoldcf/crawler/connectors/wiki/WikiConnector.java
manifoldcf/trunk/site/src/documentation/content/xdocs/en_US/end-user-documentation.xml
manifoldcf/trunk/site/src/documentation/skinconf.xml
Propchange: manifoldcf/trunk/
------------------------------------------------------------------------------
Merged /manifoldcf/branches/CONNECTORS-723:r1494650-1495933
Modified: manifoldcf/trunk/CHANGES.txt
URL: http://svn.apache.org/viewvc/manifoldcf/trunk/CHANGES.txt?rev=1495934&r1=1495933&r2=1495934&view=diff
==============================================================================
--- manifoldcf/trunk/CHANGES.txt (original)
+++ manifoldcf/trunk/CHANGES.txt Mon Jun 24 06:55:17 2013
@@ -3,6 +3,11 @@ $Id$
======================= 1.3-dev =====================
+CONNECTORS-723: Add a Jira connector. This connector does not yet
+handle security or comments, but may be extended in the future to do
+so.
+(Andrew Janowczyk, Karl Wright)
+
CONNECTORS-725: Update to SolrJ 4.3.1.
(Karl Wright)
Modified: manifoldcf/trunk/build.xml
URL: http://svn.apache.org/viewvc/manifoldcf/trunk/build.xml?rev=1495934&r1=1495933&r2=1495934&view=diff
==============================================================================
--- manifoldcf/trunk/build.xml (original)
+++ manifoldcf/trunk/build.xml Mon Jun 24 06:55:17 2013
@@ -60,6 +60,7 @@
<ant dir="connectors/cmis" target="clean"/>
<ant dir="connectors/dropbox" target="clean"/>
<ant dir="connectors/googledrive" target="clean"/>
+ <ant dir="connectors/jira" target="clean"/>
<ant dir="connectors/activedirectory" target="clean"/>
<ant dir="connectors/ldap" target="clean"/>
<ant dir="connectors/documentum" target="clean"/>
@@ -112,6 +113,7 @@
<ant dir="connectors/cmis" target="clean"/>
<ant dir="connectors/dropbox" target="clean"/>
<ant dir="connectors/googledrive" target="clean"/>
+ <ant dir="connectors/jira" target="clean"/>
<ant dir="connectors/activedirectory" target="clean"/>
<ant dir="connectors/ldap" target="clean"/>
<ant dir="connectors/documentum" target="clean"/>
@@ -285,6 +287,7 @@
<target name="setup-dropbox-connector" depends="build-framework" if="downloaded"/>
<target name="setup-googledrive-connector" depends="build-framework" if="downloaded"/>
+ <target name="setup-jira-connector" depends="build-framework" if="downloaded"/>
<target name="setup-alfresco-connector-tests" depends="build-tests-framework" if="downloaded"/>
@@ -311,6 +314,10 @@
<ant dir="connectors/googledrive" target="build"/>
</target>
+ <target name="build-jira-connector" depends="setup-jira-connector" if="downloaded">
+ <ant dir="connectors/jira" target="build"/>
+ </target>
+
<target name="doc-dropbox-connector" depends="setup-dropbox-connector" if="downloaded">
<ant dir="connectors/dropbox" target="doc"/>
@@ -320,7 +327,10 @@
<ant dir="connectors/googledrive" target="doc"/>
</target>
-
+ <target name="doc-jira-connector" depends="setup-jira-connector" if="downloaded">
+ <ant dir="connectors/googledrive" target="doc"/>
+ </target>
+
<target name="doc-alfresco-connector" depends="setup-alfresco-connector" if="downloaded">
<ant dir="connectors/alfresco" target="doc"/>
</target>
@@ -1431,6 +1441,26 @@
</condition>
</target>
+<target name="calculate-jira-condition" depends="build-jira-connector">
+ <available file="connectors/jira/dist/lib" type="dir" property="jira.exists"/>
+ <condition property="jira.include">
+ <and>
+ <isset property="jira.exists"/>
+ <isset property="downloaded"/>
+ </and>
+ </condition>
+ </target>
+
+ <target name="calculate-jira-doc-condition" depends="doc-jira-connector">
+ <available file="connectors/jira/dist/doc" type="dir" property="jira-doc.exists"/>
+ <condition property="jira-doc.include">
+ <and>
+ <isset property="jira-doc.exists"/>
+ <isset property="downloaded"/>
+ </and>
+ </condition>
+ </target>
+
<target name="calculate-cmis-doc-condition" depends="doc-cmis-connector">
@@ -1493,7 +1523,24 @@
</antcall>
</target>
-
+ <target name="deliver-jira-connector" depends="calculate-jira-condition" if="jira.include">
+ <antcall target="general-connector-delivery">
+ <param name="connector-name" value="jira"/>
+ </antcall>
+ <antcall target="general-add-repository-connector">
+ <param name="connector-name" value="jira"/>
+ <param name="connector-label" value="Jira"/>
+ <param name="connector-class" value="org.apache.manifoldcf.crawler.connectors.jira.JiraRepositoryConnector"/>
+ </antcall>
+ </target>
+
+ <target name="deliver-jira-connector-doc" depends="calculate-jira-doc-condition" if="jira-doc.include">
+ <antcall target="general-connector-doc-delivery">
+ <param name="connector-name" value="jira"/>
+ </antcall>
+ </target>
+
+
<target name="deliver-cmis-connector-doc" depends="calculate-cmis-doc-condition" if="cmis-doc.include">
<antcall target="general-connector-doc-delivery">
<param name="connector-name" value="cmis"/>
@@ -2651,8 +2698,8 @@
<target name="end-to-end-loadtests-HSQLDB" depends="run-filesystem-loadtests-HSQLDB,run-rss-loadtests-HSQLDB,run-wiki-loadtests-HSQLDB,run-alfresco-loadtests-HSQLDB,run-cmis-loadtests-HSQLDB,run-sharepoint-loadtests-HSQLDB"/>
- <target name="deliver-open-connectors" depends="deliver-googledrive-connector,deliver-dropbox-connector,deliver-nullauthority-connector,deliver-activedirectory-connector,deliver-ldap-connector,deliver-alfresco-connector,deliver-cmis-connector,deliver-filesystem-connector,deliver-rss-connector,deliver-webcrawler-connector,deliver-wiki-connector,deliver-jdbc-connector"/>
- <target name="deliver-open-connectors-doc" depends="deliver-googledrive-connector-doc,deliver-dropbox-connector-doc,deliver-nullauthority-connector-doc,deliver-activedirectory-connector-doc,deliver-ldap-connector-doc,deliver-alfresco-connector-doc,deliver-cmis-connector-doc,deliver-filesystem-connector-doc,deliver-rss-connector-doc,deliver-webcrawler-connector-doc,deliver-wiki-connector-doc,deliver-jdbc-connector-doc"/>
+ <target name="deliver-open-connectors" depends="deliver-jira-connector,deliver-googledrive-connector,deliver-dropbox-connector,deliver-nullauthority-connector,deliver-activedirectory-connector,deliver-ldap-connector,deliver-alfresco-connector,deliver-cmis-connector,deliver-filesystem-connector,deliver-rss-connector,deliver-webcrawler-connector,deliver-wiki-connector,deliver-jdbc-connector"/>
+ <target name="deliver-open-connectors-doc" depends="deliver-googledrive-connector-doc,deliver-jira-connector-doc,deliver-dropbox-connector-doc,deliver-nullauthority-connector-doc,deliver-activedirectory-connector-doc,deliver-ldap-connector-doc,deliver-alfresco-connector-doc,deliver-cmis-connector-doc,deliver-filesystem-connector-doc,deliver-rss-connector-doc,deliver-webcrawler-connector-doc,deliver-wiki-connector-doc,deliver-jdbc-connector-doc"/>
<target name="deliver-output-connectors" depends="deliver-gts-connector,deliver-solr-connector,deliver-nulloutput-connector,deliver-opensearchserver-connector,deliver-elasticsearch-connector"/>
<target name="deliver-output-connectors-doc" depends="deliver-gts-connector-doc,deliver-solr-connector-doc,deliver-nulloutput-connector-doc,deliver-opensearchserver-connector-doc,deliver-elasticsearch-connector-doc"/>
@@ -3660,6 +3707,24 @@ Use Apache Forrest version forrest-0.9-d
</antcall>
</target>
+
+ <target name="download-jira-client">
+ <mkdir dir="lib"/>
+ <antcall target="download-via-maven">
+ <param name="target" value="lib"/>
+ <param name="project-path" value="com/googlecode/json-simple"/>
+ <param name="artifact-version" value="1.1"/>
+ <param name="artifact-name" value="json-simple"/>
+ <param name="artifact-type" value="jar"/>
+ </antcall>
+ <antcall target="download-via-maven">
+ <param name="target" value="lib"/>
+ <param name="project-path" value="commons-codec"/>
+ <param name="artifact-version" value="1.8"/>
+ <param name="artifact-name" value="commons-codec"/>
+ <param name="artifact-type" value="jar"/>
+ </antcall>
+ </target>
<target name="download-google-api-client">
<mkdir dir="lib"/>
@@ -3761,7 +3826,7 @@ Use Apache Forrest version forrest-0.9-d
</antcall>
</target>
- <target name="make-core-deps" depends="download-google-api-client,download-dropbox-client,download-solrj,download-httpcomponents,download-json,download-hsqldb,download-xerces,download-commons,download-elasticsearch-plugin,download-solr-plugins,download-sharepoint-plugins,download-jstl,download-xmlgraphics-commons,download-wstx-asl,download-xmlsec,download-xml-apis,download-wss4j,download-velocity,download-streambuffer,download-stax,download-servlet-api,download-xml-resolver,download-osgi,download-opensaml,download-mimepull,download-mail,download-log4j,download-junit,download-jaxws,download-glassfish,download-jaxb,download-tomcat,download-h2,download-h2-support,download-geronimo-specs,download-fop,download-derby,download-postgresql,download-axis,download-saaj,download-wsdl4j,download-castor,download-jetty,download-slf4j,download-xalan,download-activation,download-avalon-framework,download-poi,download-chemistry,download-ecj">
+ <target name="make-core-deps" depends="download-jira-client,download-google-api-client,download-dropbox-client,download-solrj,download-httpcomponents,download-json,download-hsqldb,download-xerces,download-commons,download-elasticsearch-plugin,download-solr-plugins,download-sharepoint-plugins,download-jstl,download-xmlgraphics-commons,download-wstx-asl,download-xmlsec,download-xml-apis,download-wss4j,download-velocity,download-streambuffer,download-stax,download-servlet-api,download-xml-resolver,download-osgi,download-opensaml,download-mimepull,download-mail,download-log4j,download-junit,download-jaxws,download-glassfish,download-jaxb,download-tomcat,download-h2,download-h2-support,download-geronimo-specs,download-fop,download-derby,download-postgresql,download-axis,download-saaj,download-wsdl4j,download-castor,download-jetty,download-slf4j,download-xalan,download-activation,download-avalon-framework,download-poi,download-chemistry,download-ecj">
<copy todir="lib">
<fileset dir="lib-license" includes="*.txt"/>
</copy>
@@ -3792,6 +3857,7 @@ Use Apache Forrest version forrest-0.9-d
<ant dir="connectors/cmis" target="download-dependencies"/>
<ant dir="connectors/dropbox" target="download-dependencies"/>
<ant dir="connectors/googledrive" target="download-dependencies"/>
+ <ant dir="connectors/jira" target="download-dependencies"/>
<ant dir="connectors/activedirectory" target="download-dependencies"/>
<ant dir="connectors/ldap" target="download-dependencies"/>
<ant dir="connectors/documentum" target="download-dependencies"/>
@@ -3830,6 +3896,7 @@ Use Apache Forrest version forrest-0.9-d
<ant dir="connectors/cmis" target="download-cleanup"/>
<ant dir="connectors/dropbox" target="download-cleanup"/>
<ant dir="connectors/googledrive" target="download-cleanup"/>
+ <ant dir="connectors/jira" target="download-cleanup"/>
<ant dir="connectors/activedirectory" target="download-cleanup"/>
<ant dir="connectors/ldap" target="download-cleanup"/>
<ant dir="connectors/documentum" target="download-cleanup"/>
Propchange: manifoldcf/trunk/connectors/jira/
------------------------------------------------------------------------------
--- svn:ignore (added)
+++ svn:ignore Mon Jun 24 06:55:17 2013
@@ -0,0 +1,10 @@
+build
+dist
+doc
+target
+test-output
+test-derby-output
+test-postgresql-output
+test-HSQLDB-output
+test-HSQLDBext-output
+test-mysql-output
Modified: manifoldcf/trunk/connectors/livelink/connector/src/main/java/org/apache/manifoldcf/crawler/connectors/livelink/LivelinkConnector.java
URL: http://svn.apache.org/viewvc/manifoldcf/trunk/connectors/livelink/connector/src/main/java/org/apache/manifoldcf/crawler/connectors/livelink/LivelinkConnector.java?rev=1495934&r1=1495933&r2=1495934&view=diff
==============================================================================
--- manifoldcf/trunk/connectors/livelink/connector/src/main/java/org/apache/manifoldcf/crawler/connectors/livelink/LivelinkConnector.java (original)
+++ manifoldcf/trunk/connectors/livelink/connector/src/main/java/org/apache/manifoldcf/crawler/connectors/livelink/LivelinkConnector.java Mon Jun 24 06:55:17 2013
@@ -25,6 +25,7 @@ import org.apache.manifoldcf.crawler.sys
import org.apache.manifoldcf.crawler.system.ManifoldCF;
import org.apache.manifoldcf.core.common.XThreadInputStream;
import org.apache.manifoldcf.core.common.XThreadOutputStream;
+import org.apache.manifoldcf.core.common.InterruptibleSocketFactory;
import java.io.*;
import java.util.*;
@@ -473,6 +474,9 @@ public class LivelinkConnector extends o
getSessionParameters();
if (hasConnected == false)
{
+ int socketTimeout = 900000;
+ int connectionTimeout = 300000;
+
// Set up connection manager
PoolingClientConnectionManager localConnectionManager = new PoolingClientConnectionManager();
localConnectionManager.setMaxTotal(1);
@@ -480,7 +484,7 @@ public class LivelinkConnector extends o
// Set up ingest ssl if indicated
if (ingestKeystoreManager != null)
{
- SSLSocketFactory myFactory = new SSLSocketFactory(ingestKeystoreManager.getSecureSocketFactory(),
+ SSLSocketFactory myFactory = new SSLSocketFactory(new InterruptibleSocketFactory(ingestKeystoreManager.getSecureSocketFactory(), connectionTimeout),
new BrowserCompatHostnameVerifier());
Scheme myHttpsProtocol = new Scheme("https", 443, myFactory);
localConnectionManager.getSchemeRegistry().register(myHttpsProtocol);
Modified: manifoldcf/trunk/connectors/pom.xml
URL: http://svn.apache.org/viewvc/manifoldcf/trunk/connectors/pom.xml?rev=1495934&r1=1495933&r2=1495934&view=diff
==============================================================================
--- manifoldcf/trunk/connectors/pom.xml (original)
+++ manifoldcf/trunk/connectors/pom.xml Mon Jun 24 06:55:17 2013
@@ -52,6 +52,7 @@
<module>elasticsearch</module>
<module>dropbox</module>
<module>googledrive</module>
+ <module>jira</module>
</modules>
</project>
Propchange: manifoldcf/trunk/connectors/rss/
------------------------------------------------------------------------------
Merged /manifoldcf/branches/CONNECTORS-723/connectors/rss:r1494650-1495933
Modified: manifoldcf/trunk/connectors/rss/connector/src/main/java/org/apache/manifoldcf/crawler/connectors/rss/ThrottledFetcher.java
URL: http://svn.apache.org/viewvc/manifoldcf/trunk/connectors/rss/connector/src/main/java/org/apache/manifoldcf/crawler/connectors/rss/ThrottledFetcher.java?rev=1495934&r1=1495933&r2=1495934&view=diff
==============================================================================
--- manifoldcf/trunk/connectors/rss/connector/src/main/java/org/apache/manifoldcf/crawler/connectors/rss/ThrottledFetcher.java (original)
+++ manifoldcf/trunk/connectors/rss/connector/src/main/java/org/apache/manifoldcf/crawler/connectors/rss/ThrottledFetcher.java Mon Jun 24 06:55:17 2013
@@ -20,6 +20,7 @@ package org.apache.manifoldcf.crawler.co
import org.apache.manifoldcf.core.interfaces.*;
import org.apache.manifoldcf.core.common.XThreadInputStream;
+import org.apache.manifoldcf.core.common.InterruptibleSocketFactory;
import org.apache.manifoldcf.agents.interfaces.*;
import org.apache.manifoldcf.crawler.interfaces.*;
import org.apache.manifoldcf.crawler.system.Logging;
@@ -1485,176 +1486,5 @@ public class ThrottledFetcher
}
- /** SSL Socket factory which wraps another socket factory but allows timeout on socket
- * creation.
- */
- protected static class InterruptibleSocketFactory extends javax.net.ssl.SSLSocketFactory
- {
- protected final javax.net.ssl.SSLSocketFactory wrappedFactory;
- protected final long connectTimeoutMilliseconds;
-
- public InterruptibleSocketFactory(javax.net.ssl.SSLSocketFactory wrappedFactory, long connectTimeoutMilliseconds)
- {
- this.wrappedFactory = wrappedFactory;
- this.connectTimeoutMilliseconds = connectTimeoutMilliseconds;
- }
-
- @Override
- public Socket createSocket()
- throws IOException
- {
- // Socket isn't open
- return wrappedFactory.createSocket();
- }
-
- @Override
- public Socket createSocket(String host, int port)
- throws IOException, UnknownHostException
- {
- return fireOffThread(InetAddress.getByName(host),port,null,-1);
- }
-
- @Override
- public Socket createSocket(InetAddress host, int port)
- throws IOException
- {
- return fireOffThread(host,port,null,-1);
- }
-
- @Override
- public Socket createSocket(String host, int port, InetAddress localHost, int localPort)
- throws IOException, UnknownHostException
- {
- return fireOffThread(InetAddress.getByName(host),port,localHost,localPort);
- }
-
- @Override
- public Socket createSocket(InetAddress address, int port, InetAddress localAddress, int localPort)
- throws IOException
- {
- return fireOffThread(address,port,localAddress,localPort);
- }
-
- @Override
- public Socket createSocket(Socket s, String host, int port, boolean autoClose)
- throws IOException
- {
- // Socket's already open
- return wrappedFactory.createSocket(s,host,port,autoClose);
- }
-
- @Override
- public String[] getDefaultCipherSuites()
- {
- return wrappedFactory.getDefaultCipherSuites();
- }
-
- @Override
- public String[] getSupportedCipherSuites()
- {
- return wrappedFactory.getSupportedCipherSuites();
- }
-
- protected Socket fireOffThread(InetAddress address, int port, InetAddress localHost, int localPort)
- throws IOException
- {
- SocketCreateThread thread = new SocketCreateThread(wrappedFactory,address,port,localHost,localPort);
- thread.start();
- try
- {
- // Wait for thread to complete for only a certain amount of time!
- thread.join(connectTimeoutMilliseconds);
- // If join() times out, then the thread is going to still be alive.
- if (thread.isAlive())
- {
- // Kill the thread - not that this will necessarily work, but we need to try
- thread.interrupt();
- throw new ConnectTimeoutException("Secure connection timed out");
- }
- // The thread terminated. Throw an error if there is one, otherwise return the result.
- Throwable t = thread.getException();
- if (t != null)
- {
- if (t instanceof java.net.SocketTimeoutException)
- throw (java.net.SocketTimeoutException)t;
- else if (t instanceof ConnectTimeoutException)
- throw (ConnectTimeoutException)t;
- else if (t instanceof InterruptedIOException)
- throw (InterruptedIOException)t;
- else if (t instanceof IOException)
- throw (IOException)t;
- else if (t instanceof Error)
- throw (Error)t;
- else if (t instanceof RuntimeException)
- throw (RuntimeException)t;
- throw new Error("Received an unexpected exception: "+t.getMessage(),t);
- }
- return thread.getResult();
- }
- catch (InterruptedException e)
- {
- throw new InterruptedIOException("Interrupted: "+e.getMessage());
- }
-
- }
-
- }
-
- /** Create a secure socket in a thread, so that we can "give up" after a while if the socket fails to connect.
- */
- protected static class SocketCreateThread extends Thread
- {
- // Socket factory
- protected javax.net.ssl.SSLSocketFactory socketFactory;
- protected InetAddress host;
- protected int port;
- protected InetAddress clientHost;
- protected int clientPort;
-
- // The return socket
- protected Socket rval = null;
- // The return error
- protected Throwable throwable = null;
-
- /** Create the thread */
- public SocketCreateThread(javax.net.ssl.SSLSocketFactory socketFactory,
- InetAddress host,
- int port,
- InetAddress clientHost,
- int clientPort)
- {
- this.socketFactory = socketFactory;
- this.host = host;
- this.port = port;
- this.clientHost = clientHost;
- this.clientPort = clientPort;
- setDaemon(true);
- }
-
- public void run()
- {
- try
- {
- if (clientHost == null)
- rval = socketFactory.createSocket(host,port);
- else
- rval = socketFactory.createSocket(host,port,clientHost,clientPort);
- }
- catch (Throwable e)
- {
- throwable = e;
- }
- }
-
- public Throwable getException()
- {
- return throwable;
- }
-
- public Socket getResult()
- {
- return rval;
- }
- }
}
Modified: manifoldcf/trunk/connectors/webcrawler/connector/src/main/java/org/apache/manifoldcf/crawler/connectors/webcrawler/ThrottledFetcher.java
URL: http://svn.apache.org/viewvc/manifoldcf/trunk/connectors/webcrawler/connector/src/main/java/org/apache/manifoldcf/crawler/connectors/webcrawler/ThrottledFetcher.java?rev=1495934&r1=1495933&r2=1495934&view=diff
==============================================================================
--- manifoldcf/trunk/connectors/webcrawler/connector/src/main/java/org/apache/manifoldcf/crawler/connectors/webcrawler/ThrottledFetcher.java (original)
+++ manifoldcf/trunk/connectors/webcrawler/connector/src/main/java/org/apache/manifoldcf/crawler/connectors/webcrawler/ThrottledFetcher.java Mon Jun 24 06:55:17 2013
@@ -21,6 +21,7 @@ package org.apache.manifoldcf.crawler.co
import org.apache.manifoldcf.core.interfaces.*;
import org.apache.manifoldcf.core.common.DeflateInputStream;
import org.apache.manifoldcf.core.common.XThreadInputStream;
+import org.apache.manifoldcf.core.common.InterruptibleSocketFactory;
import org.apache.manifoldcf.agents.interfaces.*;
import org.apache.manifoldcf.crawler.interfaces.*;
import org.apache.manifoldcf.crawler.system.Logging;
@@ -2259,178 +2260,6 @@ public class ThrottledFetcher
}
}
- /** SSL Socket factory which wraps another socket factory but allows timeout on socket
- * creation.
- */
- protected static class InterruptibleSocketFactory extends javax.net.ssl.SSLSocketFactory
- {
- protected final javax.net.ssl.SSLSocketFactory wrappedFactory;
- protected final long connectTimeoutMilliseconds;
-
- public InterruptibleSocketFactory(javax.net.ssl.SSLSocketFactory wrappedFactory, long connectTimeoutMilliseconds)
- {
- this.wrappedFactory = wrappedFactory;
- this.connectTimeoutMilliseconds = connectTimeoutMilliseconds;
- }
-
- @Override
- public Socket createSocket()
- throws IOException
- {
- // Socket isn't open
- return wrappedFactory.createSocket();
- }
-
- @Override
- public Socket createSocket(String host, int port)
- throws IOException, UnknownHostException
- {
- return fireOffThread(InetAddress.getByName(host),port,null,-1);
- }
-
- @Override
- public Socket createSocket(InetAddress host, int port)
- throws IOException
- {
- return fireOffThread(host,port,null,-1);
- }
-
- @Override
- public Socket createSocket(String host, int port, InetAddress localHost, int localPort)
- throws IOException, UnknownHostException
- {
- return fireOffThread(InetAddress.getByName(host),port,localHost,localPort);
- }
-
- @Override
- public Socket createSocket(InetAddress address, int port, InetAddress localAddress, int localPort)
- throws IOException
- {
- return fireOffThread(address,port,localAddress,localPort);
- }
-
- @Override
- public Socket createSocket(Socket s, String host, int port, boolean autoClose)
- throws IOException
- {
- // Socket's already open
- return wrappedFactory.createSocket(s,host,port,autoClose);
- }
-
- @Override
- public String[] getDefaultCipherSuites()
- {
- return wrappedFactory.getDefaultCipherSuites();
- }
-
- @Override
- public String[] getSupportedCipherSuites()
- {
- return wrappedFactory.getSupportedCipherSuites();
- }
-
- protected Socket fireOffThread(InetAddress address, int port, InetAddress localHost, int localPort)
- throws IOException
- {
- SocketCreateThread thread = new SocketCreateThread(wrappedFactory,address,port,localHost,localPort);
- thread.start();
- try
- {
- // Wait for thread to complete for only a certain amount of time!
- thread.join(connectTimeoutMilliseconds);
- // If join() times out, then the thread is going to still be alive.
- if (thread.isAlive())
- {
- // Kill the thread - not that this will necessarily work, but we need to try
- thread.interrupt();
- throw new ConnectTimeoutException("Secure connection timed out");
- }
- // The thread terminated. Throw an error if there is one, otherwise return the result.
- Throwable t = thread.getException();
- if (t != null)
- {
- if (t instanceof java.net.SocketTimeoutException)
- throw (java.net.SocketTimeoutException)t;
- else if (t instanceof ConnectTimeoutException)
- throw (ConnectTimeoutException)t;
- else if (t instanceof InterruptedIOException)
- throw (InterruptedIOException)t;
- else if (t instanceof IOException)
- throw (IOException)t;
- else if (t instanceof Error)
- throw (Error)t;
- else if (t instanceof RuntimeException)
- throw (RuntimeException)t;
- throw new Error("Received an unexpected exception: "+t.getMessage(),t);
- }
- return thread.getResult();
- }
- catch (InterruptedException e)
- {
- throw new InterruptedIOException("Interrupted: "+e.getMessage());
- }
-
- }
-
- }
-
- /** Create a secure socket in a thread, so that we can "give up" after a while if the socket fails to connect.
- */
- protected static class SocketCreateThread extends Thread
- {
- // Socket factory
- protected javax.net.ssl.SSLSocketFactory socketFactory;
- protected InetAddress host;
- protected int port;
- protected InetAddress clientHost;
- protected int clientPort;
-
- // The return socket
- protected Socket rval = null;
- // The return error
- protected Throwable throwable = null;
-
- /** Create the thread */
- public SocketCreateThread(javax.net.ssl.SSLSocketFactory socketFactory,
- InetAddress host,
- int port,
- InetAddress clientHost,
- int clientPort)
- {
- this.socketFactory = socketFactory;
- this.host = host;
- this.port = port;
- this.clientHost = clientHost;
- this.clientPort = clientPort;
- setDaemon(true);
- }
-
- public void run()
- {
- try
- {
- if (clientHost == null)
- rval = socketFactory.createSocket(host,port);
- else
- rval = socketFactory.createSocket(host,port,clientHost,clientPort);
- }
- catch (Throwable e)
- {
- throwable = e;
- }
- }
-
- public Throwable getException()
- {
- return throwable;
- }
-
- public Socket getResult()
- {
- return rval;
- }
- }
-
/** Class to override browser compatibility to make it not check cookie paths. See CONNECTORS-97.
*/
protected static class LaxBrowserCompatSpec extends BrowserCompatSpec
Propchange: manifoldcf/trunk/connectors/wiki/
------------------------------------------------------------------------------
Merged /manifoldcf/branches/CONNECTORS-723/connectors/wiki:r1494650-1495933
Modified: manifoldcf/trunk/connectors/wiki/connector/src/main/java/org/apache/manifoldcf/crawler/connectors/wiki/WikiConnector.java
URL: http://svn.apache.org/viewvc/manifoldcf/trunk/connectors/wiki/connector/src/main/java/org/apache/manifoldcf/crawler/connectors/wiki/WikiConnector.java?rev=1495934&r1=1495933&r2=1495934&view=diff
==============================================================================
--- manifoldcf/trunk/connectors/wiki/connector/src/main/java/org/apache/manifoldcf/crawler/connectors/wiki/WikiConnector.java (original)
+++ manifoldcf/trunk/connectors/wiki/connector/src/main/java/org/apache/manifoldcf/crawler/connectors/wiki/WikiConnector.java Mon Jun 24 06:55:17 2013
@@ -57,6 +57,9 @@ import org.apache.http.message.BasicHead
import org.apache.http.client.params.ClientPNames;
import org.apache.http.client.HttpRequestRetryHandler;
import org.apache.http.protocol.HttpContext;
+import org.apache.http.conn.scheme.Scheme;
+import org.apache.http.conn.ssl.SSLSocketFactory;
+import org.apache.http.conn.ssl.AllowAllHostnameVerifier;
import org.apache.http.conn.ConnectTimeoutException;
import org.apache.http.client.CircularRedirectException;
@@ -181,13 +184,20 @@ public class WikiConnector extends org.a
baseURL = protocol + "://" + server + ((portString!=null)?":" + portString:"") + path + "/api.php?format=xml&";
+ int socketTimeout = 900000;
+ int connectionTimeout = 300000;
+
+ javax.net.ssl.SSLSocketFactory httpsSocketFactory = KeystoreManagerFactory.getTrustingSecureSocketFactory();
+ SSLSocketFactory myFactory = new SSLSocketFactory(new InterruptibleSocketFactory(httpsSocketFactory,connectionTimeout),
+ new AllowAllHostnameVerifier());
+ Scheme myHttpsProtocol = new Scheme("https", 443, myFactory);
+
// Set up connection manager
PoolingClientConnectionManager localConnectionManager = new PoolingClientConnectionManager();
localConnectionManager.setMaxTotal(1);
connectionManager = localConnectionManager;
-
- int socketTimeout = 900000;
- int connectionTimeout = 300000;
+ // Set up protocol registry
+ connectionManager.getSchemeRegistry().register(myHttpsProtocol);
BasicHttpParams params = new BasicHttpParams();
params.setBooleanParameter(CoreProtocolPNames.USE_EXPECT_CONTINUE,true);
Modified: manifoldcf/trunk/site/src/documentation/content/xdocs/en_US/end-user-documentation.xml
URL: http://svn.apache.org/viewvc/manifoldcf/trunk/site/src/documentation/content/xdocs/en_US/end-user-documentation.xml?rev=1495934&r1=1495933&r2=1495934&view=diff
==============================================================================
--- manifoldcf/trunk/site/src/documentation/content/xdocs/en_US/end-user-documentation.xml (original)
+++ manifoldcf/trunk/site/src/documentation/content/xdocs/en_US/end-user-documentation.xml Mon Jun 24 06:55:17 2013
@@ -1762,11 +1762,35 @@ curl -XGET http://localhost:9200/index/_
<figure src="images/en_US/googledrive-repository-connection-job-googledrive-seed-query.PNG" alt="googledrive Repository Connection, seed query" width="80%"/>
<br/><br/>
<p>This tab allows you to specify the query which will be used to seed documents for the indexing process. The query language is
- specified on the <a href="https://developers.google.com/drive/search-parameters">Drive Search Paramters</a> site. Directories
+ specified on the <a href="https://developers.google.com/drive/search-parameters">Drive Search Parameters</a> site. Directories
which meet the seed query are fully crawled as the query on applies to seeds. The default query indexes the entire drive. Lastly, native Google
documents such as spreadsheets and word documents are exported to PDF and then ingested.</p>
</section>
+ <section id="jirarepository">
+ <title>Jira Repository Connection</title>
+ <p>The Jira Repository Connection type allows you to index tickets from <a href="http://www.atlassian.com/â">Atlassian</a>'s <a href="http://www.atlassian.com/software/jira">Jira</a>.</p>
+ <p>A Jira connection has the following configuration parameters on the repository connection editing screen:</p>
+ <br/><br/>
+ <figure src="images/en_US/jira-repository-connection-configuration.PNG" alt="jira Repository Connection, configuration parameters" width="80%"/>
+ <br/><br/>
+ <p>As we can see there are 3 pieces of information which are needed to create a successful connection. The Client ID and Client Secret
+ are the username and password of a Jira account. The JiraUrl is the base endpoint of the particular Jira Instance, for example:
+ https://searchbox.atlassian.net is the Jira version for searchbox which is hosted at Atlassian.</p>
+ <br/><br/>
+ <p>After you click the "Save" button, you will see a connection summary screen, which might look something like this:</p>
+ <br/><br/>
+ <figure src="images/en_US/jira-repository-connection-configuration-save.PNG" alt="jira Repository Connection, saving configuration" width="80%"/>
+ <br/><br/>
+ <p>When you configure a job to use the Jira repository connection an additional tab is presented. This is the "Seed Query" tab:</p>
+ <br/><br/>
+ <figure src="images/en_US/jira-repository-connection-job-jira-seed-query.PNG" alt="jira Repository Connection, seed query" width="80%"/>
+ <br/><br/>
+ <p>This tab allows you to specify the query which will be used to find documents for the indexing process. The query language is
+ specified on the <a href="https://confluence.atlassian.com/display/JIRA/Advanced+Searching">Jira Advanced Searching</a> site. Directories
+ which meet the seed query are fully crawled as the query on applies to seeds.</p>
+ </section>
+
<section id="livelinkrepository">
<title>OpenText LiveLink Repository Connection</title>
<p>The LiveLink connection type allows you to index content from LiveLink repositories. LiveLink has a rich variety of different document types and metadata,
Modified: manifoldcf/trunk/site/src/documentation/skinconf.xml
URL: http://svn.apache.org/viewvc/manifoldcf/trunk/site/src/documentation/skinconf.xml?rev=1495934&r1=1495933&r2=1495934&view=diff
==============================================================================
--- manifoldcf/trunk/site/src/documentation/skinconf.xml (original)
+++ manifoldcf/trunk/site/src/documentation/skinconf.xml Mon Jun 24 06:55:17 2013
@@ -104,6 +104,7 @@ which will be used to configure the chos
QBase, MetaCarta, and GTS are trademarks of QBase, Inc.
Meridio and Autonomy are trademarks of Hewlett Packard, Inc.
Alfresco is a trademark of Alfresco Software, Inc.
+ Jira is a trademark of Atlassian, Inc.
</trademark-statement>
<!-- Some skins use this to form a 'breadcrumb trail' of links.