You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@manifoldcf.apache.org by kw...@apache.org on 2014/12/17 20:41:44 UTC
svn commit: r1646321 - in /manifoldcf/branches/dev_1x: ./ framework/
framework/agents/src/main/java/org/apache/manifoldcf/agents/incrementalingest/
framework/pull-agent/src/main/java/org/apache/manifoldcf/crawler/system/
site/src/documentation/content/...
Author: kwright
Date: Wed Dec 17 19:41:44 2014
New Revision: 1646321
URL: http://svn.apache.org/r1646321
Log:
Undo unintentional commit
Modified:
manifoldcf/branches/dev_1x/ (props changed)
manifoldcf/branches/dev_1x/CHANGES.txt
manifoldcf/branches/dev_1x/framework/ (props changed)
manifoldcf/branches/dev_1x/framework/agents/src/main/java/org/apache/manifoldcf/agents/incrementalingest/IncrementalIngester.java
manifoldcf/branches/dev_1x/framework/pull-agent/src/main/java/org/apache/manifoldcf/crawler/system/JobResetThread.java
manifoldcf/branches/dev_1x/framework/pull-agent/src/main/java/org/apache/manifoldcf/crawler/system/StartupThread.java
manifoldcf/branches/dev_1x/site/src/documentation/content/xdocs/en_US/end-user-documentation.xml
Propchange: manifoldcf/branches/dev_1x/
------------------------------------------------------------------------------
--- svn:mergeinfo (original)
+++ svn:mergeinfo Wed Dec 17 19:41:44 2014
@@ -118,4 +118,4 @@
/manifoldcf/branches/CONNECTORS-981:1605049-1605773
/manifoldcf/branches/CONNECTORS-989:1611600-1612101
/manifoldcf/branches/CONNECTORS-990:1610284-1610707
-/manifoldcf/trunk:1620703,1620748,1620812,1620862,1621449,1621613,1621855,1622213,1622740,1622850,1622853-1622854,1623249,1623251,1623314,1623599,1623951,1623953-1623954,1623956,1623972,1624058,1624085,1624174,1624236,1624377,1624384,1624399,1624449,1624464,1624504,1624729-1624731,1624906,1624909-1624910,1625023,1625095,1625103,1625108,1625264,1625270,1625394,1625400,1625910,1626090,1626097,1626102,1626638-1626639,1626973,1627687,1627690,1627959,1628066,1628106,1628168,1628188,1628699,1628798,1628808,1628845,1628905,1629122,1629374-1629375,1629379,1629541,1629994,1630188,1630535,1630623,1630671,1630812,1630885,1631039,1631162,1631164,1631252,1631750,1631953,1632013,1632225,1632289,1632562,1632844,1632847,1632854,1633062-1633063,1633108,1633193,1633202,1633282,1633284,1633295,1633336,1633339,1633345,1633348,1633364,1633378,1633383,1633432,1633546,1633590,1633634,1633668,1633727,1633760,1633764,1633786,1633910,1633923,1634021,1634028,1634067,1634132,1634145,1634148,1634155,1634188,163
4202,1634264,1634373,1634530,1634688,1634850,1634857,1635103,1635116,1635421,1635438,1635478,1635481,1635484,1635490,1635809,1635939,1636146,1636167,1636180,1636207,1636215,1636232,1636334,1636519,1636570,1636684,1636940,1637011,1637310,1637350,1637364,1637373,1637378,1639259,1639593,1639600,1640018,1640101,1640199,1640204,1640208,1640314,1640319,1640749,1640772,1640805,1640888,1640925,1640941-1640942,1641222,1641328,1641557,1641559,1641629,1641633,1641724,1641754,1641911,1642163,1642255,1642318,1642531,1642650,1642658,1642673,1642716,1644197,1644399,1644538,1644920,1644931,1646317
+/manifoldcf/trunk:1620703,1620748,1620812,1620862,1621449,1621613,1621855,1622213,1622740,1622850,1622853-1622854,1623249,1623251,1623314,1623599,1623951,1623953-1623954,1623956,1623972,1624058,1624085,1624174,1624236,1624377,1624384,1624399,1624449,1624464,1624504,1624729-1624731,1624906,1624909-1624910,1625023,1625095,1625103,1625108,1625264,1625270,1625394,1625400,1625910,1626090,1626097,1626102,1626638-1626639,1626973,1627687,1627690,1627959,1628066,1628106,1628168,1628188,1628699,1628798,1628808,1628845,1628905,1629122,1629374-1629375,1629379,1629541,1629994,1630188,1630535,1630623,1630671,1630812,1630885,1631039,1631162,1631164,1631252,1631750,1631953,1632013,1632225,1632289,1632562,1632844,1632847,1632854,1633062-1633063,1633108,1633193,1633202,1633282,1633284,1633295,1633336,1633339,1633345,1633348,1633364,1633378,1633383,1633432,1633546,1633590,1633634,1633668,1633727,1633760,1633764,1633786,1633910,1633923,1634021,1634028,1634067,1634132,1634145,1634148,1634155,1634188,163
4202,1634264,1634373,1634530,1634688,1634850,1634857,1635103,1635116,1635421,1635438,1635478,1635481,1635484,1635490,1635809,1635939,1636146,1636167,1636180,1636207,1636215,1636232,1636334,1636519,1636570,1636684,1636940,1637011,1637310,1637350,1637364,1637373,1637378,1639259,1639593,1639600,1640018,1640101,1640199,1640204,1640208,1640314,1640319,1640749,1640772,1640805,1640888,1640925,1640941-1640942,1641222,1641328,1641557,1641559,1641629,1641633,1641724,1641754,1641911,1642163,1642255,1642318,1642531,1642650,1642658,1642673,1642716,1644197,1644399,1644538,1644920,1644931
Modified: manifoldcf/branches/dev_1x/CHANGES.txt
URL: http://svn.apache.org/viewvc/manifoldcf/branches/dev_1x/CHANGES.txt?rev=1646321&r1=1646320&r2=1646321&view=diff
==============================================================================
--- manifoldcf/branches/dev_1x/CHANGES.txt (original)
+++ manifoldcf/branches/dev_1x/CHANGES.txt Wed Dec 17 19:41:44 2014
@@ -6,10 +6,6 @@ $Id$
======================= Release 1.8 =====================
-CONNECTORS-1123: Reduce the maximum number of zookeeper lock
-nodes, by hashing the lock names that correspond to documents.
-(Aeham Abushwashi, Karl Wright)
-
CONNECTORS-1035: Introduce connector-common jar, to get
legacy Alfresco connector working again.
(Karl Wright)
Propchange: manifoldcf/branches/dev_1x/framework/
------------------------------------------------------------------------------
--- svn:mergeinfo (original)
+++ svn:mergeinfo Wed Dec 17 19:41:44 2014
@@ -113,4 +113,4 @@
/manifoldcf/branches/CONNECTORS-989/framework:1611600-1612101
/manifoldcf/branches/CONNECTORS-990/framework:1610284-1610707
/manifoldcf/trunk:1629122
-/manifoldcf/trunk/framework:1620703,1620748,1620812,1620862,1621449,1621613,1621855,1622213,1622740,1622850,1622853-1622854,1623249,1623251,1623314,1623599,1623951,1623953-1623954,1623956,1623972,1624058,1624085,1624174,1624236,1624377,1624384,1624399,1624449,1624464,1624504,1624729-1624731,1624906,1624909-1624910,1625023,1625095,1625103,1625108,1625264,1625270,1625394,1625400,1625910,1626090,1626097,1626102,1626638-1626639,1626973,1627687,1627690,1627959,1628066,1628106,1628168,1628188,1628699,1628798,1628808,1628845,1628905,1629122,1629374-1629375,1629379,1629541,1629994,1630188,1630535,1630623,1630671,1630812,1630885,1631039,1631162,1631164,1631252,1632013,1632289,1632844,1633108,1633193,1633202,1633348,1633364,1634145,1634148,1634155,1634264,1634373,1634530,1635438,1635809,1636146,1636180,1636207,1636232,1636334,1636519,1636570,1636684,1636940,1637011,1639593,1639600,1640018,1640101,1640199,1640314,1640319,1640749,1640772,1640925,1640941,1641222,1641557,1641559,1641724,1641911,1
642163,1642255,1642318,1644197,1644399,1646317
+/manifoldcf/trunk/framework:1620703,1620748,1620812,1620862,1621449,1621613,1621855,1622213,1622740,1622850,1622853-1622854,1623249,1623251,1623314,1623599,1623951,1623953-1623954,1623956,1623972,1624058,1624085,1624174,1624236,1624377,1624384,1624399,1624449,1624464,1624504,1624729-1624731,1624906,1624909-1624910,1625023,1625095,1625103,1625108,1625264,1625270,1625394,1625400,1625910,1626090,1626097,1626102,1626638-1626639,1626973,1627687,1627690,1627959,1628066,1628106,1628168,1628188,1628699,1628798,1628808,1628845,1628905,1629122,1629374-1629375,1629379,1629541,1629994,1630188,1630535,1630623,1630671,1630812,1630885,1631039,1631162,1631164,1631252,1632013,1632289,1632844,1633108,1633193,1633202,1633348,1633364,1634145,1634148,1634155,1634264,1634373,1634530,1635438,1635809,1636146,1636180,1636207,1636232,1636334,1636519,1636570,1636684,1636940,1637011,1639593,1639600,1640018,1640101,1640199,1640314,1640319,1640749,1640772,1640925,1640941,1641222,1641557,1641559,1641724,1641911,1
642163,1642255,1642318,1644197,1644399
Modified: manifoldcf/branches/dev_1x/framework/agents/src/main/java/org/apache/manifoldcf/agents/incrementalingest/IncrementalIngester.java
URL: http://svn.apache.org/viewvc/manifoldcf/branches/dev_1x/framework/agents/src/main/java/org/apache/manifoldcf/agents/incrementalingest/IncrementalIngester.java?rev=1646321&r1=1646320&r2=1646321&view=diff
==============================================================================
--- manifoldcf/branches/dev_1x/framework/agents/src/main/java/org/apache/manifoldcf/agents/incrementalingest/IncrementalIngester.java (original)
+++ manifoldcf/branches/dev_1x/framework/agents/src/main/java/org/apache/manifoldcf/agents/incrementalingest/IncrementalIngester.java Wed Dec 17 19:41:44 2014
@@ -999,15 +999,6 @@ public class IncrementalIngester extends
}
}
- protected static String createURILockName(String outputConnectionName, String uriHash)
- {
- // The lock name needs to be constrained to some acceptably small number in order to avoid
- // a lot of zookeeper locks. See CONNECTORS-1123.
- int hashCode = outputConnectionName.hashCode() + uriHash.hashCode();
- hashCode &= 0xffff;
- return "URILOCK-"+hashCode;
- }
-
/** Delete multiple documents from the search engine index.
*@param pipelineConnections is the pipeline specification.
*@param identifierClasses are the names of the spaces in which the identifier hashes should be interpreted.
@@ -1063,7 +1054,7 @@ public class IncrementalIngester extends
{
validURIArray[validURIcount] = uri.getURI();
validURIHashArray[validURIcount] = uri.getURIHash();
- lockArray[validURIcount] = createURILockName(outputConnectionName,validURIHashArray[validURIcount]);
+ lockArray[validURIcount] = outputConnectionName+":"+validURIHashArray[validURIcount];
validURIcount++;
}
}
@@ -1278,7 +1269,7 @@ public class IncrementalIngester extends
{
validURIArray[validURIcount] = uri.getURI();
validURIHashArray[validURIcount] = uri.getURIHash();
- lockArray[validURIcount] = createURILockName(outputConnectionName,validURIHashArray[validURIcount]);
+ lockArray[validURIcount] = outputConnectionName+":"+validURIHashArray[validURIcount];
validURIcount++;
}
}
@@ -3464,19 +3455,19 @@ public class IncrementalIngester extends
}
}
- protected static String[] computeLockArray(String documentURIHash, String oldURIHash, String outputConnectionName)
+ protected static String[] computeLockArray(String documentURI, String oldURI, String outputConnectionName)
{
int uriCount = 0;
- if (documentURIHash != null)
+ if (documentURI != null)
uriCount++;
- if (oldURIHash != null && (documentURIHash == null || !documentURIHash.equals(oldURIHash)))
+ if (oldURI != null && (documentURI == null || !documentURI.equals(oldURI)))
uriCount++;
String[] lockArray = new String[uriCount];
uriCount = 0;
- if (documentURIHash != null)
- lockArray[uriCount++] = createURILockName(outputConnectionName,documentURIHash);
- if (oldURIHash != null && (documentURIHash == null || !documentURIHash.equals(oldURIHash)))
- lockArray[uriCount++] = createURILockName(outputConnectionName,oldURIHash);
+ if (documentURI != null)
+ lockArray[uriCount++] = outputConnectionName+":"+documentURI;
+ if (oldURI != null && (documentURI == null || !documentURI.equals(oldURI)))
+ lockArray[uriCount++] = outputConnectionName+":"+oldURI;
return lockArray;
}
Modified: manifoldcf/branches/dev_1x/framework/pull-agent/src/main/java/org/apache/manifoldcf/crawler/system/JobResetThread.java
URL: http://svn.apache.org/viewvc/manifoldcf/branches/dev_1x/framework/pull-agent/src/main/java/org/apache/manifoldcf/crawler/system/JobResetThread.java?rev=1646321&r1=1646320&r2=1646321&view=diff
==============================================================================
--- manifoldcf/branches/dev_1x/framework/pull-agent/src/main/java/org/apache/manifoldcf/crawler/system/JobResetThread.java (original)
+++ manifoldcf/branches/dev_1x/framework/pull-agent/src/main/java/org/apache/manifoldcf/crawler/system/JobResetThread.java Wed Dec 17 19:41:44 2014
@@ -102,7 +102,7 @@ public class JobResetThread extends Thre
// not predicted by the algorithm that assigned those priorities. This is, of course, quite expensive,
// but it cannot be helped (at least, I cannot find a way to avoid it).
//
- if (jobStops.size() > 0 /*|| jobResumes.size() > 0 */)
+ if (jobStops.size() > 0 || jobResumes.size() > 0)
{
Logging.threads.debug("Job reset thread reprioritizing documents...");
Modified: manifoldcf/branches/dev_1x/framework/pull-agent/src/main/java/org/apache/manifoldcf/crawler/system/StartupThread.java
URL: http://svn.apache.org/viewvc/manifoldcf/branches/dev_1x/framework/pull-agent/src/main/java/org/apache/manifoldcf/crawler/system/StartupThread.java?rev=1646321&r1=1646320&r2=1646321&view=diff
==============================================================================
--- manifoldcf/branches/dev_1x/framework/pull-agent/src/main/java/org/apache/manifoldcf/crawler/system/StartupThread.java (original)
+++ manifoldcf/branches/dev_1x/framework/pull-agent/src/main/java/org/apache/manifoldcf/crawler/system/StartupThread.java Wed Dec 17 19:41:44 2014
@@ -139,7 +139,7 @@ public class StartupThread extends Threa
model,jobType == IJobDescription.TYPE_CONTINUOUS,lastSeedingVersion == null,
requestMinimum);
- //ManifoldCF.resetAllDocumentPriorities(threadContext,processID);
+ ManifoldCF.resetAllDocumentPriorities(threadContext,processID);
if (Logging.threads.isDebugEnabled())
Logging.threads.debug("Prepared job "+jobID.toString()+" for execution.");
Modified: manifoldcf/branches/dev_1x/site/src/documentation/content/xdocs/en_US/end-user-documentation.xml
URL: http://svn.apache.org/viewvc/manifoldcf/branches/dev_1x/site/src/documentation/content/xdocs/en_US/end-user-documentation.xml?rev=1646321&r1=1646320&r2=1646321&view=diff
==============================================================================
--- manifoldcf/branches/dev_1x/site/src/documentation/content/xdocs/en_US/end-user-documentation.xml (original)
+++ manifoldcf/branches/dev_1x/site/src/documentation/content/xdocs/en_US/end-user-documentation.xml Wed Dec 17 19:41:44 2014
@@ -1078,12 +1078,6 @@ curl -XGET http://localhost:9200/index/_
<p>Note that in this example, the Active Directory connection is not responding, which is leading to an error status message instead of "Connection working".</p>
</section>
- <section id="alfrescowebscriptauthority">
- <title>Alfresco Webscript Authority Connection</title>
- <p>The Alfresco Webscript authority connection type helps secure documents indexed using the Alfresco Webscript repository connection type.</p>
- <p>RAFA: Please fill out</p>
- </section>
-
<section id="cmisauthority">
<title>CMIS Authority Connection</title>
<p>A CMIS authority connection is required for enforcing security for documents retrieved from CMIS repositories.</p>
@@ -1518,14 +1512,6 @@ curl -XGET http://localhost:9200/index/_
<br/><br/>
</section>
- <section id="alfrescowebscriptrepository">
- <title>Alfresco Webscript Repository Connection</title>
- <p>The Alfresco Webscript Repository connection type allows you to index content from an Alfresco repository. It also supports document
- security, in conjunction with the Alfresco Webscript Authority connection Type.</p>
- <p>This connector is compatible with any Alfresco version (???). RAFA: Please fill this out.</p>
-
- </section>
-
<section id="cmisrepository">
<title>CMIS Repository Connection</title>
<p>The CMIS Repository Connection type allows you to index content from any CMIS-compliant repository.</p>