You are viewing a plain text version of this content. The canonical link for it is here.
Posted to oak-commits@jackrabbit.apache.org by mr...@apache.org on 2015/08/11 14:55:42 UTC
svn commit: r1695297 [1/3] - in /jackrabbit/oak/branches/1.0: ./
oak-commons/src/main/java/org/apache/jackrabbit/oak/commons/sort/
oak-core/src/main/java/org/apache/jackrabbit/oak/plugins/document/
oak-core/src/main/java/org/apache/jackrabbit/oak/plugi...
Author: mreutegg
Date: Tue Aug 11 12:55:41 2015
New Revision: 1695297
URL: http://svn.apache.org/r1695297
Log:
OAK-2829: Comparing node states for external changes is too slow
OAK-3002: Optimize docCache and docChildrenCache invalidation by filtering using journal
Merged revisions 1678023,1678171,1684820,1685590,1685964,1685977,1685989,1686023,1686032,1688179 from trunk
Conflicts:
oak-core/src/main/java/org/apache/jackrabbit/oak/plugins/document/Commit.java
- [conflicts: imports affected and minor manual merge required]
oak-core/src/main/java/org/apache/jackrabbit/oak/plugins/document/DocumentNodeStore.java
- [manual merge required for access modifier change of applyChanges()]
oak-core/src/main/java/org/apache/jackrabbit/oak/plugins/document/DocumentNodeStoreService.java
- [conflicts: only imports affected]
oak-core/src/main/java/org/apache/jackrabbit/oak/plugins/document/LastRevRecoveryAgent.java
- [conflict with OAK-2074 not available in 1.0 branch]
- [conflict with OAK-2131 not available in 1.0 branch]
- [conflict with OAK-2324 not available in 1.0 branch]
oak-core/src/main/java/org/apache/jackrabbit/oak/plugins/document/UnsavedModifications.java
- [conflict with OAK-2324 not available in 1.0 branch]
- [conflict with OAK-1768]
- [conflict with OAK-2888 only available in 1.0 branch]
oak-core/src/main/java/org/apache/jackrabbit/oak/plugins/document/mongo/MongoDocumentStore.java
- [conflict with OAK-1641 not available in 1.0 branch]
- [conflict with OAK-2681 not available in 1.0 branch: findUncachedWithRetry not avail in 1.0]
oak-core/src/main/java/org/apache/jackrabbit/oak/plugins/document/util/Utils.java
- [conflicts: only imports affected]
oak-jcr/src/main/java/org/apache/jackrabbit/oak/jcr/observation/ChangeProcessor.java
- [conflicts: only imports affected]
oak-run/src/main/java/org/apache/jackrabbit/oak/benchmark/ObservationTest.java
- [conflict with OAK-2717 not available in 1.0 branch]
- [ADDED JcrCreator class plus OakRepositoryFixture.setUpCluster(int n, JcrCreator customizer) from trunk]
Additional Compile Problems:
oak-core/src/test/java/org/apache/jackrabbit/oak/plugins/document/UnsavedModificationsTest.java
- [conflict with OAK-2888 only available in 1.0 branch, fixed, Snapshot.IGNORE added]
Additional Test Problems:
oak-core/src/test/java/org/apache/jackrabbit/oak/plugins/document/JournalTest.java
- [conflict with OAK-2131 not available in 1.0 branch, hence test logic had to be adapted/fixed]
Added:
jackrabbit/oak/branches/1.0/oak-core/src/main/java/org/apache/jackrabbit/oak/plugins/document/JournalEntry.java (with props)
jackrabbit/oak/branches/1.0/oak-core/src/main/java/org/apache/jackrabbit/oak/plugins/document/JournalGarbageCollector.java (with props)
jackrabbit/oak/branches/1.0/oak-core/src/test/java/org/apache/jackrabbit/oak/plugins/document/AbstractJournalTest.java (with props)
jackrabbit/oak/branches/1.0/oak-core/src/test/java/org/apache/jackrabbit/oak/plugins/document/CountingDocumentStore.java (with props)
jackrabbit/oak/branches/1.0/oak-core/src/test/java/org/apache/jackrabbit/oak/plugins/document/CountingTieredDiffCache.java (with props)
jackrabbit/oak/branches/1.0/oak-core/src/test/java/org/apache/jackrabbit/oak/plugins/document/JournalEntryTest.java (with props)
jackrabbit/oak/branches/1.0/oak-core/src/test/java/org/apache/jackrabbit/oak/plugins/document/JournalTest.java (with props)
jackrabbit/oak/branches/1.0/oak-core/src/test/java/org/apache/jackrabbit/oak/plugins/document/mongo/JournalIT.java (with props)
jackrabbit/oak/branches/1.0/oak-run/src/main/java/org/apache/jackrabbit/oak/fixture/JcrCreator.java (with props)
Modified:
jackrabbit/oak/branches/1.0/ (props changed)
jackrabbit/oak/branches/1.0/oak-commons/src/main/java/org/apache/jackrabbit/oak/commons/sort/StringSort.java
jackrabbit/oak/branches/1.0/oak-commons/src/main/java/org/apache/jackrabbit/oak/commons/sort/package-info.java
jackrabbit/oak/branches/1.0/oak-core/src/main/java/org/apache/jackrabbit/oak/plugins/document/Collection.java
jackrabbit/oak/branches/1.0/oak-core/src/main/java/org/apache/jackrabbit/oak/plugins/document/Commit.java
jackrabbit/oak/branches/1.0/oak-core/src/main/java/org/apache/jackrabbit/oak/plugins/document/DiffCache.java
jackrabbit/oak/branches/1.0/oak-core/src/main/java/org/apache/jackrabbit/oak/plugins/document/DocumentNodeState.java
jackrabbit/oak/branches/1.0/oak-core/src/main/java/org/apache/jackrabbit/oak/plugins/document/DocumentNodeStore.java
jackrabbit/oak/branches/1.0/oak-core/src/main/java/org/apache/jackrabbit/oak/plugins/document/DocumentNodeStoreService.java
jackrabbit/oak/branches/1.0/oak-core/src/main/java/org/apache/jackrabbit/oak/plugins/document/DocumentStore.java
jackrabbit/oak/branches/1.0/oak-core/src/main/java/org/apache/jackrabbit/oak/plugins/document/LastRevRecoveryAgent.java
jackrabbit/oak/branches/1.0/oak-core/src/main/java/org/apache/jackrabbit/oak/plugins/document/LocalDiffCache.java
jackrabbit/oak/branches/1.0/oak-core/src/main/java/org/apache/jackrabbit/oak/plugins/document/MemoryDiffCache.java
jackrabbit/oak/branches/1.0/oak-core/src/main/java/org/apache/jackrabbit/oak/plugins/document/MergeCommit.java
jackrabbit/oak/branches/1.0/oak-core/src/main/java/org/apache/jackrabbit/oak/plugins/document/TieredDiffCache.java
jackrabbit/oak/branches/1.0/oak-core/src/main/java/org/apache/jackrabbit/oak/plugins/document/UnsavedModifications.java
jackrabbit/oak/branches/1.0/oak-core/src/main/java/org/apache/jackrabbit/oak/plugins/document/memory/MemoryDocumentStore.java
jackrabbit/oak/branches/1.0/oak-core/src/main/java/org/apache/jackrabbit/oak/plugins/document/mongo/CacheInvalidator.java
jackrabbit/oak/branches/1.0/oak-core/src/main/java/org/apache/jackrabbit/oak/plugins/document/mongo/MongoDiffCache.java
jackrabbit/oak/branches/1.0/oak-core/src/main/java/org/apache/jackrabbit/oak/plugins/document/mongo/MongoDocumentStore.java
jackrabbit/oak/branches/1.0/oak-core/src/main/java/org/apache/jackrabbit/oak/plugins/document/rdb/RDBDocumentStore.java
jackrabbit/oak/branches/1.0/oak-core/src/main/java/org/apache/jackrabbit/oak/plugins/document/util/LoggingDocumentStoreWrapper.java
jackrabbit/oak/branches/1.0/oak-core/src/main/java/org/apache/jackrabbit/oak/plugins/document/util/SynchronizingDocumentStoreWrapper.java
jackrabbit/oak/branches/1.0/oak-core/src/main/java/org/apache/jackrabbit/oak/plugins/document/util/TimingDocumentStoreWrapper.java
jackrabbit/oak/branches/1.0/oak-core/src/main/java/org/apache/jackrabbit/oak/plugins/document/util/Utils.java
jackrabbit/oak/branches/1.0/oak-core/src/main/java/org/apache/jackrabbit/oak/plugins/observation/NodeObserver.java
jackrabbit/oak/branches/1.0/oak-core/src/test/java/org/apache/jackrabbit/oak/plugins/document/AmnesiaDiffCache.java
jackrabbit/oak/branches/1.0/oak-core/src/test/java/org/apache/jackrabbit/oak/plugins/document/ClusterTest.java
jackrabbit/oak/branches/1.0/oak-core/src/test/java/org/apache/jackrabbit/oak/plugins/document/DocumentNodeStoreTest.java
jackrabbit/oak/branches/1.0/oak-core/src/test/java/org/apache/jackrabbit/oak/plugins/document/UnsavedModificationsTest.java
jackrabbit/oak/branches/1.0/oak-core/src/test/java/org/apache/jackrabbit/oak/plugins/document/mongo/MongoDiffCacheTest.java
jackrabbit/oak/branches/1.0/oak-jcr/src/main/java/org/apache/jackrabbit/oak/jcr/observation/ChangeProcessor.java
jackrabbit/oak/branches/1.0/oak-run/src/main/java/org/apache/jackrabbit/oak/benchmark/ObservationTest.java
jackrabbit/oak/branches/1.0/oak-run/src/main/java/org/apache/jackrabbit/oak/fixture/OakRepositoryFixture.java
Propchange: jackrabbit/oak/branches/1.0/
------------------------------------------------------------------------------
--- svn:mergeinfo (original)
+++ svn:mergeinfo Tue Aug 11 12:55:41 2015
@@ -1,2 +1,2 @@
-/jackrabbit/oak/trunk:1584578,1584602,1584614,1584616,1584709,1584781,1584937,1585297,1585304-1585305,1585420,1585424,1585427,1585448,1585465,1585468,1585486,1585497,1585509,1585647,1585655-1585656,1585661,1585665-1585666,1585669-1585670,1585673,1585680,1585719,1585763,1585770,1585896,1585904,1585907,1585940,1585949,1585951,1585956,1585962-1585963,1586287,1586320,1586364,1586372,1586655,1586836,1587130,1587224,1587399,1587408,1587472,1587485,1587488,1587538,1587580,1587807,1588033,1588042,1588046,1588066,1588201,1589025,1589101,1589137,1589141,1589263,1589440,1589442,1589484,1589488,1589661,1589664,1589682,1589708,1589741,1589748,1589789,1589794,1589850,1589864,1590628,1590660,1590684,1590697,1590701,1590980,1590988,1591101,1591226,1591229,1591293,1591314,1591317,1591362,1591374,1591381,1591438,1591467,1591552,1591704,1591713,1591715,1591723,1591874,1592487,1592512,1592658,1592665,1592677,1592742,1592744,1592787,1592809,1592955,1593036,1593048,1593061,1593133,1593210-1593211,1593231
,1593245,1593250,1593294,1593304,1593317,1593342,1593554,1594158-1594164,1594166-1594167,1594169,1594237,1594800,1594808,1594835,1594888,1594987,1595147,1595457,1595856,1596241,1596474,1596534,1596844,1596889,1597569,1597795,1597854,1597860,1598292,1598302,1598352,1598369,1598595,1598631,1598696,1598732,1598797-1598798,1599160,1599299,1599332,1599416,1599434,1599671,1600088,1600935,1601309,1601388,1601578,1601649,1601676,1601757,1601768,1601814,1601833,1601838,1601853,1601878,1601888,1601922,1602156,1602170,1602174,1602179,1602183,1602201,1602207,1602227,1602256,1602261,1602342,1602796-1602797,1602800,1602809,1602853,1602872,1602914,1603155,1603307,1603401,1603441,1603748,1604166,1605030,1605036,1605038,1605292,1605447,1605526,1605670,1605725,1605831,1605852,1606077,1606079,1606087,1606638,1606641,1606644,1606708,1606711,1607031-1607032,1607077,1607127,1607141,1607152,1607185,1607196,1607331,1607362,1607366,1607392,1607526,1607557,1607664,1607737,1608560,1608731,1608783,1609064,1609
081,1609165,1609214,1609488,1610489,1610592,1610603,1610634,1610658,1610664,1611021,1611041,1611270,1611275,1611277,1611313,1611332,1611584,1612560,1612825,1612848,1612892,1612899,1612993,1613018,1613041,1614032,1614265,1614272,1614344-1614345,1614384-1614385,1614397,1614405-1614406,1614574,1614591,1614593,1614596,1614604,1614689,1614807,1614835,1614891,1615417-1615418,1616182,1616236,1616463,1616719,1617417,1617451,1617463,1617711,1618158,1618613,1618624,1618709,1619222,1619411,1619695,1619800,1619808,1619815,1619823-1619824,1620512,1620581,1620585,1620634,1620898,1620905,1621115,1621123-1621124,1621168,1621192,1621201,1621706,1621962,1622197,1622201,1622207,1622250,1622479,1623364,1623766,1623827,1623949,1623969,1623973,1624216,1624317,1624551,1624559,1624973,1624993-1624994,1625025,1625036,1625158,1625224,1625237,1625299,1625348,1625620,1625916,1625962-1625963,1626021,1626053,1626163,1626168,1626175,1626191,1626265,1626770,1627047,1627052,1627228,1627346,1627470,1627473,1627479,1
627503,1627586,1627590,1627715,1627731,1628180,1628198,1628262,1628447,1628608,1629688,1629840,1629858,1629917,1630055-1630057,1630156,1630299,1630338,1630773,1631283-1631284,1631333-1631334,1631617-1631619,1631630,1631699,1631704,1631711,1631967-1631969,1631986,1631990,1631999,1632002-1632003,1632017,1632258,1632264,1632270,1632293,1632303,1632592,1632605,1633315,1633327,1633389,1633559-1633560,1633562,1633567,1633571,1633598,1633608,1633641,1633687,1633697,1633768,1633783,1634505,1634513,1634774,1634779,1634781,1634792,1634803,1634814,1634816,1634838,1634841,1634852,1634864,1634896,1634898,1635044-1635045,1635060,1635077,1635089,1635102,1635108,1635178,1635218,1635387,1635435,1635518,1635563,1635586,1636336,1636348,1636505,1636585,1636799,1637368,1637382,1637413,1637651,1637815,1638779-1638783,1639260,1639577,1639622,1639963,1639966,1639973,1640134,1640143,1640523,1640555-1640556,1640694-1640695,1640715,1640722-1640723,1640728,1640863-1640872,1641340,1641346,1641350,1641352,164154
1,1641596-1641599,1641601,1641662,1641671,1641695,1641771,1641802,1641811,1641950,1642031,1642056,1642119,1642285,1642648,1642667,1642954,1642959,1643111,1643178,1643186,1643204,1643287,1643767,1643774,1643982,1644016,1644106,1644366,1644383,1644397-1644398,1644407,1644479,1644547,1644552,1644554,1644588,1644645,1644650,1644654,1644689,1644750,1645421,1645424,1645459,1645585,1645611,1645637,1645646,1645660-1645663,1645888,1645901,1645948,1645966,1645970-1645971,1646014,1646164,1646174,1646469,1646684,1646687,1646726-1646728,1646766,1646795,1646981,1649743,1649803,1650015,1650239,1650529,1650797,1651323,1651382,1651643,1651652,1651730,1651988-1651989,1651996,1652024,1652035,1652058-1652059,1652075,1652127,1652158,1652467,1652965,1652971,1652992,1653207,1653446,1653463,1653484,1653572,1653579,1653591,1653804,1653809,1653813,1653848-1653850,1653882,1654116,1654174,1654743,1654756,1654778,1655028,1655049,1655054-1655055,1655086,1655237,1655248,1655996,1656019,1656027,1656033,1656303,165
6394,1656400,1656425,1656427,1656432,1656453,1656628,1656678,1657128,1657132,1657163,1657188,1657265,1657511,1657766,1657804,1658470,1658977,1658983,1659285,1659483,1659527,1659550,1659578,1659765,1660100,1660154-1660155,1660383-1660384,1660409,1660426,1660676,1660870,1660872,1660897,1660903,1661069,1661122,1661146,1661158,1661226,1661630,1661643,1661645,1662313-1662315,1662323,1662381,1662450,1662456,1663241,1663275,1663288,1663448,1663526,1663528,1663565,1663578,1663614,1663666,1663705,1663730,1663753,1663854,1664038,1664184,1664228-1664229,1664231,1664381,1664569,1664947,1664987,1665184,1665257,1665271-1665272,1665274-1665275,1665436,1665604,1665634,1665758,1665835,1665892,1665897,1665910,1665918,1666100,1666102,1666177,1666218,1666220,1666351-1666352,1666381,1666384,1666426,1666491,1666787,1667062,1667184,1667293,1667462,1667498,1667502,1667573,1667590,1667696,1667782,1668160,1668275,1668624,1668641,1668645,1668649,1668665,1668671,1668683,1668688,1668845,1669072,1669096,1669135,
1669337,1669361,1669579,1669680,1669989,1670030,1670693,1670705,1671489,1671512,1671773,1671787,1671795,1672055,1672277,1672350,1672468,1672537,1672603,1672642,1672644,1672834-1672835,1673351,1673410,1673431,1673436,1673644,1673662-1673663,1673695,1673713,1673738,1673787,1673791,1674046,1674065,1674075,1674107,1674150,1674780,1675054,1675319,1675332,1675382,1675555,1675566,1676198,1676407,1676458,1676670,1676703,1677579,1677609,1677611,1677774,1677788,1677797,1677804,1677806,1677939,1677991,1678095-1678096,1678173,1678211,1678323,1678758,1678938,1678954,1679144,1679147,1679165,1679191,1679232,1679503,1679961,1680170,1680182,1680222,1680232,1680236,1680461,1680633,1680643,1680747,1680805-1680806,1680903,1681282,1681767,1681918,1682218,1682235,1682437,1682494,1682555,1682855,1682904,1683059,1683089,1683213,1683249,1683259,1683278,1683323,1683687,1683700,1684174,1684376,1684442,1684561,1684570,1684618,1684836,1684868,1685023,1685370,1685552,1685589,1685840,1685999,1686097,1686229,16862
34,1686253,1686414,1686780,1686854,1686857,1686971,1687053,1687175,1687196,1687198,1687220,1687239-1687240,1687301,1687441,1687553,1688090,1688349,1688421,1688436,1688453,1688622,1688636,1688817,1689003-1689004,1689008,1689577,1689581,1689623,1689774,1689810,1689828,1689833,1689903,1690017,1690043,1690047,1690057,1690247,1690249,1690634-1690637,1690650,1690669,1690674,1690941,1691139,1691159,1691167,1691183,1691188,1691210,1691307,1691331-1691333,1691345,1691384-1691385,1691401,1691509,1692133,1692156,1692250,1692274,1692363,1692382,1692478,1692955,1693030,1693209,1693421,1693525-1693526,1694007,1694393
+/jackrabbit/oak/trunk:1584578,1584602,1584614,1584616,1584709,1584781,1584937,1585297,1585304-1585305,1585420,1585424,1585427,1585448,1585465,1585468,1585486,1585497,1585509,1585647,1585655-1585656,1585661,1585665-1585666,1585669-1585670,1585673,1585680,1585719,1585763,1585770,1585896,1585904,1585907,1585940,1585949,1585951,1585956,1585962-1585963,1586287,1586320,1586364,1586372,1586655,1586836,1587130,1587224,1587399,1587408,1587472,1587485,1587488,1587538,1587580,1587807,1588033,1588042,1588046,1588066,1588201,1589025,1589101,1589137,1589141,1589263,1589440,1589442,1589484,1589488,1589661,1589664,1589682,1589708,1589741,1589748,1589789,1589794,1589850,1589864,1590628,1590660,1590684,1590697,1590701,1590980,1590988,1591101,1591226,1591229,1591293,1591314,1591317,1591362,1591374,1591381,1591438,1591467,1591552,1591704,1591713,1591715,1591723,1591874,1592487,1592512,1592658,1592665,1592677,1592742,1592744,1592787,1592809,1592955,1593036,1593048,1593061,1593133,1593210-1593211,1593231
,1593245,1593250,1593294,1593304,1593317,1593342,1593554,1594158-1594164,1594166-1594167,1594169,1594237,1594800,1594808,1594835,1594888,1594987,1595147,1595457,1595856,1596241,1596474,1596534,1596844,1596889,1597569,1597795,1597854,1597860,1598292,1598302,1598352,1598369,1598595,1598631,1598696,1598732,1598797-1598798,1599160,1599299,1599332,1599416,1599434,1599671,1600088,1600935,1601309,1601388,1601578,1601649,1601676,1601757,1601768,1601814,1601833,1601838,1601853,1601878,1601888,1601922,1602156,1602170,1602174,1602179,1602183,1602201,1602207,1602227,1602256,1602261,1602342,1602796-1602797,1602800,1602809,1602853,1602872,1602914,1603155,1603307,1603401,1603441,1603748,1604166,1605030,1605036,1605038,1605292,1605447,1605526,1605670,1605725,1605831,1605852,1606077,1606079,1606087,1606638,1606641,1606644,1606708,1606711,1607031-1607032,1607077,1607127,1607141,1607152,1607185,1607196,1607331,1607362,1607366,1607392,1607526,1607557,1607664,1607737,1608560,1608731,1608783,1609064,1609
081,1609165,1609214,1609488,1610489,1610592,1610603,1610634,1610658,1610664,1611021,1611041,1611270,1611275,1611277,1611313,1611332,1611584,1612560,1612825,1612848,1612892,1612899,1612993,1613018,1613041,1614032,1614265,1614272,1614344-1614345,1614384-1614385,1614397,1614405-1614406,1614574,1614591,1614593,1614596,1614604,1614689,1614807,1614835,1614891,1615417-1615418,1616182,1616236,1616463,1616719,1617417,1617451,1617463,1617711,1618158,1618613,1618624,1618709,1619222,1619411,1619695,1619800,1619808,1619815,1619823-1619824,1620512,1620581,1620585,1620634,1620898,1620905,1621115,1621123-1621124,1621168,1621192,1621201,1621706,1621962,1622197,1622201,1622207,1622250,1622479,1623364,1623766,1623827,1623949,1623969,1623973,1624216,1624317,1624551,1624559,1624973,1624993-1624994,1625025,1625036,1625158,1625224,1625237,1625299,1625348,1625620,1625916,1625962-1625963,1626021,1626053,1626163,1626168,1626175,1626191,1626265,1626770,1627047,1627052,1627228,1627346,1627470,1627473,1627479,1
627503,1627586,1627590,1627715,1627731,1628180,1628198,1628262,1628447,1628608,1629688,1629840,1629858,1629917,1630055-1630057,1630156,1630299,1630338,1630773,1631283-1631284,1631333-1631334,1631617-1631619,1631630,1631699,1631704,1631711,1631967-1631969,1631986,1631990,1631999,1632002-1632003,1632017,1632258,1632264,1632270,1632293,1632303,1632592,1632605,1633315,1633327,1633389,1633559-1633560,1633562,1633567,1633571,1633598,1633608,1633641,1633687,1633697,1633768,1633783,1634505,1634513,1634774,1634779,1634781,1634792,1634803,1634814,1634816,1634838,1634841,1634852,1634864,1634896,1634898,1635044-1635045,1635060,1635077,1635089,1635102,1635108,1635178,1635218,1635387,1635435,1635518,1635563,1635586,1636336,1636348,1636505,1636585,1636799,1637368,1637382,1637413,1637651,1637815,1638779-1638783,1639260,1639577,1639622,1639963,1639966,1639973,1640134,1640143,1640523,1640555-1640556,1640694-1640695,1640715,1640722-1640723,1640728,1640863-1640872,1641340,1641346,1641350,1641352,164154
1,1641596-1641599,1641601,1641662,1641671,1641695,1641771,1641802,1641811,1641950,1642031,1642056,1642119,1642285,1642648,1642667,1642954,1642959,1643111,1643178,1643186,1643204,1643287,1643767,1643774,1643982,1644016,1644106,1644366,1644383,1644397-1644398,1644407,1644479,1644547,1644552,1644554,1644588,1644645,1644650,1644654,1644689,1644750,1645421,1645424,1645459,1645585,1645611,1645637,1645646,1645660-1645663,1645888,1645901,1645948,1645966,1645970-1645971,1646014,1646164,1646174,1646469,1646684,1646687,1646726-1646728,1646766,1646795,1646981,1649743,1649803,1650015,1650239,1650529,1650797,1651323,1651382,1651643,1651652,1651730,1651988-1651989,1651996,1652024,1652035,1652058-1652059,1652075,1652127,1652158,1652467,1652965,1652971,1652992,1653207,1653446,1653463,1653484,1653572,1653579,1653591,1653804,1653809,1653813,1653848-1653850,1653882,1654116,1654174,1654743,1654756,1654778,1655028,1655049,1655054-1655055,1655086,1655237,1655248,1655996,1656019,1656027,1656033,1656303,165
6394,1656400,1656425,1656427,1656432,1656453,1656628,1656678,1657128,1657132,1657163,1657188,1657265,1657511,1657766,1657804,1658470,1658977,1658983,1659285,1659483,1659527,1659550,1659578,1659765,1660100,1660154-1660155,1660383-1660384,1660409,1660426,1660676,1660870,1660872,1660897,1660903,1661069,1661122,1661146,1661158,1661226,1661630,1661643,1661645,1662313-1662315,1662323,1662381,1662450,1662456,1663241,1663275,1663288,1663448,1663526,1663528,1663565,1663578,1663614,1663666,1663705,1663730,1663753,1663854,1664038,1664184,1664228-1664229,1664231,1664381,1664569,1664947,1664987,1665184,1665257,1665271-1665272,1665274-1665275,1665436,1665604,1665634,1665758,1665835,1665892,1665897,1665910,1665918,1666100,1666102,1666177,1666218,1666220,1666351-1666352,1666381,1666384,1666426,1666491,1666787,1667062,1667184,1667293,1667462,1667498,1667502,1667573,1667590,1667696,1667782,1668160,1668275,1668624,1668641,1668645,1668649,1668665,1668671,1668683,1668688,1668845,1669072,1669096,1669135,
1669337,1669361,1669579,1669680,1669989,1670030,1670693,1670705,1671489,1671512,1671773,1671787,1671795,1672055,1672277,1672350,1672468,1672537,1672603,1672642,1672644,1672834-1672835,1673351,1673410,1673431,1673436,1673644,1673662-1673663,1673695,1673713,1673738,1673787,1673791,1674046,1674065,1674075,1674107,1674150,1674780,1675054,1675319,1675332,1675382,1675555,1675566,1676198,1676407,1676458,1676670,1676703,1677579,1677609,1677611,1677774,1677788,1677797,1677804,1677806,1677939,1677991,1678023,1678095-1678096,1678171,1678173,1678211,1678323,1678758,1678938,1678954,1679144,1679147,1679165,1679191,1679232,1679503,1679961,1680170,1680182,1680222,1680232,1680236,1680461,1680633,1680643,1680747,1680805-1680806,1680903,1681282,1681767,1681918,1682218,1682235,1682437,1682494,1682555,1682855,1682904,1683059,1683089,1683213,1683249,1683259,1683278,1683323,1683687,1683700,1684174,1684376,1684442,1684561,1684570,1684618,1684820,1684836,1684868,1685023,1685370,1685552,1685589-1685590,16858
40,1685964,1685977,1685989,1685999,1686023,1686032,1686097,1686229,1686234,1686253,1686414,1686780,1686854,1686857,1686971,1687053,1687175,1687196,1687198,1687220,1687239-1687240,1687301,1687441,1687553,1688090,1688179,1688349,1688421,1688436,1688453,1688622,1688636,1688817,1689003-1689004,1689008,1689577,1689581,1689623,1689774,1689810,1689828,1689833,1689903,1690017,1690043,1690047,1690057,1690247,1690249,1690634-1690637,1690650,1690669,1690674,1690941,1691139,1691159,1691167,1691183,1691188,1691210,1691307,1691331-1691333,1691345,1691384-1691385,1691401,1691509,1692133,1692156,1692250,1692274,1692363,1692382,1692478,1692955,1693030,1693209,1693421,1693525-1693526,1694007,1694393
/jackrabbit/trunk:1345480
Modified: jackrabbit/oak/branches/1.0/oak-commons/src/main/java/org/apache/jackrabbit/oak/commons/sort/StringSort.java
URL: http://svn.apache.org/viewvc/jackrabbit/oak/branches/1.0/oak-commons/src/main/java/org/apache/jackrabbit/oak/commons/sort/StringSort.java?rev=1695297&r1=1695296&r2=1695297&view=diff
==============================================================================
--- jackrabbit/oak/branches/1.0/oak-commons/src/main/java/org/apache/jackrabbit/oak/commons/sort/StringSort.java (original)
+++ jackrabbit/oak/branches/1.0/oak-commons/src/main/java/org/apache/jackrabbit/oak/commons/sort/StringSort.java Tue Aug 11 12:55:41 2015
@@ -35,6 +35,7 @@ import com.google.common.base.Charsets;
import com.google.common.collect.Lists;
import com.google.common.io.Closer;
import com.google.common.io.Files;
+
import org.apache.commons.io.FileUtils;
import org.apache.commons.io.LineIterator;
import org.slf4j.Logger;
@@ -45,7 +46,7 @@ import org.slf4j.LoggerFactory;
* the list would be maintained in memory. If the size crosses the required threshold then
* the sorting would be performed externally
*/
-public class StringSort implements Closeable {
+public class StringSort implements Iterable<String>, Closeable {
private final Logger log = LoggerFactory.getLogger(getClass());
public static final int BATCH_SIZE = 2048;
@@ -117,6 +118,17 @@ public class StringSort implements Close
}
}
+ @Override
+ public Iterator<String> iterator() {
+ try {
+ return getIds();
+ } catch (IOException e) {
+ throw new IllegalStateException(e);
+ }
+ }
+
+ //--------------------------< internal >------------------------------------
+
private void addToBatch(String id) throws IOException {
inMemBatch.add(id);
if (inMemBatch.size() >= BATCH_SIZE) {
Modified: jackrabbit/oak/branches/1.0/oak-commons/src/main/java/org/apache/jackrabbit/oak/commons/sort/package-info.java
URL: http://svn.apache.org/viewvc/jackrabbit/oak/branches/1.0/oak-commons/src/main/java/org/apache/jackrabbit/oak/commons/sort/package-info.java?rev=1695297&r1=1695296&r2=1695297&view=diff
==============================================================================
--- jackrabbit/oak/branches/1.0/oak-commons/src/main/java/org/apache/jackrabbit/oak/commons/sort/package-info.java (original)
+++ jackrabbit/oak/branches/1.0/oak-commons/src/main/java/org/apache/jackrabbit/oak/commons/sort/package-info.java Tue Aug 11 12:55:41 2015
@@ -14,7 +14,7 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
-@Version("1.0")
+@Version("1.1")
@Export(optional = "provide:=true")
package org.apache.jackrabbit.oak.commons.sort;
Modified: jackrabbit/oak/branches/1.0/oak-core/src/main/java/org/apache/jackrabbit/oak/plugins/document/Collection.java
URL: http://svn.apache.org/viewvc/jackrabbit/oak/branches/1.0/oak-core/src/main/java/org/apache/jackrabbit/oak/plugins/document/Collection.java?rev=1695297&r1=1695296&r2=1695297&view=diff
==============================================================================
--- jackrabbit/oak/branches/1.0/oak-core/src/main/java/org/apache/jackrabbit/oak/plugins/document/Collection.java (original)
+++ jackrabbit/oak/branches/1.0/oak-core/src/main/java/org/apache/jackrabbit/oak/plugins/document/Collection.java Tue Aug 11 12:55:41 2015
@@ -70,6 +70,20 @@ public abstract class Collection<T exten
}
};
+ /**
+ * The 'journal' collection contains documents with consolidated
+ * diffs for changes performed by a cluster node between two background
+ * updates.
+ */
+ public static final Collection<JournalEntry> JOURNAL =
+ new Collection<JournalEntry>("journal") {
+ @Nonnull
+ @Override
+ public JournalEntry newDocument(DocumentStore store) {
+ return new JournalEntry(store);
+ }
+ };
+
private final String name;
public Collection(String name) {
Modified: jackrabbit/oak/branches/1.0/oak-core/src/main/java/org/apache/jackrabbit/oak/plugins/document/Commit.java
URL: http://svn.apache.org/viewvc/jackrabbit/oak/branches/1.0/oak-core/src/main/java/org/apache/jackrabbit/oak/plugins/document/Commit.java?rev=1695297&r1=1695296&r2=1695297&view=diff
==============================================================================
--- jackrabbit/oak/branches/1.0/oak-core/src/main/java/org/apache/jackrabbit/oak/plugins/document/Commit.java (original)
+++ jackrabbit/oak/branches/1.0/oak-core/src/main/java/org/apache/jackrabbit/oak/plugins/document/Commit.java Tue Aug 11 12:55:41 2015
@@ -37,6 +37,8 @@ import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import static com.google.common.base.Preconditions.checkNotNull;
+import static java.util.Collections.singletonList;
+import static org.apache.jackrabbit.oak.plugins.document.Collection.JOURNAL;
import static org.apache.jackrabbit.oak.plugins.document.Collection.NODES;
import static org.apache.jackrabbit.oak.plugins.document.NodeDocument.COLLISIONS;
import static org.apache.jackrabbit.oak.plugins.document.NodeDocument.SPLIT_CANDIDATE_THRESHOLD;
@@ -48,7 +50,7 @@ public class Commit {
private static final Logger LOG = LoggerFactory.getLogger(Commit.class);
- private final DocumentNodeStore nodeStore;
+ protected final DocumentNodeStore nodeStore;
private final DocumentNodeStoreBranch branch;
private final Revision baseRevision;
private final Revision revision;
@@ -124,6 +126,15 @@ public class Commit {
return baseRevision;
}
+ /**
+ * @return all modified paths, including ancestors without explicit
+ * modifications.
+ */
+ @Nonnull
+ Iterable<String> getModifiedPaths() {
+ return modifiedNodes;
+ }
+
void addNodeDiff(DocumentNodeState n) {
diff.tag('+').key(n.getPath());
diff.object();
@@ -271,7 +282,7 @@ public class Commit {
// so that all operations can be rolled back if there is a conflict
ArrayList<UpdateOp> opLog = new ArrayList<UpdateOp>();
- //Compute the commit root
+ // Compute the commit root
for (String p : operations.keySet()) {
markChanged(p);
if (commitRootPath == null) {
@@ -285,6 +296,16 @@ public class Commit {
}
}
}
+
+ // push branch changes to journal
+ if (baseBranchRevision != null) {
+ // store as external change
+ JournalEntry doc = JOURNAL.newDocument(store);
+ doc.modified(modifiedNodes);
+ Revision r = revision.asBranchRevision();
+ store.create(JOURNAL, singletonList(doc.asUpdateOp(r)));
+ }
+
int commitRootDepth = PathUtils.getDepth(commitRootPath);
// check if there are real changes on the commit root
boolean commitRootHasChanges = operations.containsKey(commitRootPath);
@@ -583,7 +604,7 @@ public class Commit {
}
list.add(p);
}
- DiffCache.Entry cacheEntry = nodeStore.getDiffCache().newEntry(before, revision);
+ DiffCache.Entry cacheEntry = nodeStore.getDiffCache().newEntry(before, revision, true);
List<String> added = new ArrayList<String>();
List<String> removed = new ArrayList<String>();
List<String> changed = new ArrayList<String>();
Modified: jackrabbit/oak/branches/1.0/oak-core/src/main/java/org/apache/jackrabbit/oak/plugins/document/DiffCache.java
URL: http://svn.apache.org/viewvc/jackrabbit/oak/branches/1.0/oak-core/src/main/java/org/apache/jackrabbit/oak/plugins/document/DiffCache.java?rev=1695297&r1=1695296&r2=1695297&view=diff
==============================================================================
--- jackrabbit/oak/branches/1.0/oak-core/src/main/java/org/apache/jackrabbit/oak/plugins/document/DiffCache.java (original)
+++ jackrabbit/oak/branches/1.0/oak-core/src/main/java/org/apache/jackrabbit/oak/plugins/document/DiffCache.java Tue Aug 11 12:55:41 2015
@@ -56,11 +56,14 @@ public interface DiffCache {
*
* @param from the from revision.
* @param to the to revision.
+ * @param local true indicates that the entry results from a local change,
+ * false if it results from an external change
* @return the cache entry.
*/
@Nonnull
Entry newEntry(@Nonnull Revision from,
- @Nonnull Revision to);
+ @Nonnull Revision to,
+ boolean local);
/**
* @return the statistics for this cache.
Modified: jackrabbit/oak/branches/1.0/oak-core/src/main/java/org/apache/jackrabbit/oak/plugins/document/DocumentNodeState.java
URL: http://svn.apache.org/viewvc/jackrabbit/oak/branches/1.0/oak-core/src/main/java/org/apache/jackrabbit/oak/plugins/document/DocumentNodeState.java?rev=1695297&r1=1695296&r2=1695297&view=diff
==============================================================================
--- jackrabbit/oak/branches/1.0/oak-core/src/main/java/org/apache/jackrabbit/oak/plugins/document/DocumentNodeState.java (original)
+++ jackrabbit/oak/branches/1.0/oak-core/src/main/java/org/apache/jackrabbit/oak/plugins/document/DocumentNodeState.java Tue Aug 11 12:55:41 2015
@@ -396,10 +396,9 @@ public class DocumentNodeState extends A
@Override
public String toString() {
StringBuilder buff = new StringBuilder();
- buff.append("path: ").append(path).append('\n');
- buff.append("rev: ").append(rev).append('\n');
- buff.append(properties);
- buff.append('\n');
+ buff.append("{ path: '").append(path).append("', ");
+ buff.append("rev: '").append(rev).append("', ");
+ buff.append("properties: '").append(properties.values()).append("' }");
return buff.toString();
}
Modified: jackrabbit/oak/branches/1.0/oak-core/src/main/java/org/apache/jackrabbit/oak/plugins/document/DocumentNodeStore.java
URL: http://svn.apache.org/viewvc/jackrabbit/oak/branches/1.0/oak-core/src/main/java/org/apache/jackrabbit/oak/plugins/document/DocumentNodeStore.java?rev=1695297&r1=1695296&r2=1695297&view=diff
==============================================================================
--- jackrabbit/oak/branches/1.0/oak-core/src/main/java/org/apache/jackrabbit/oak/plugins/document/DocumentNodeStore.java (original)
+++ jackrabbit/oak/branches/1.0/oak-core/src/main/java/org/apache/jackrabbit/oak/plugins/document/DocumentNodeStore.java Tue Aug 11 12:55:41 2015
@@ -21,14 +21,19 @@ import static com.google.common.base.Pre
import static com.google.common.collect.Iterables.filter;
import static com.google.common.collect.Iterables.toArray;
import static com.google.common.collect.Iterables.transform;
+import static java.util.Collections.singletonList;
import static org.apache.jackrabbit.oak.api.CommitFailedException.MERGE;
import static org.apache.jackrabbit.oak.commons.PathUtils.concat;
+import static org.apache.jackrabbit.oak.plugins.document.Collection.JOURNAL;
import static org.apache.jackrabbit.oak.plugins.document.Collection.NODES;
import static org.apache.jackrabbit.oak.plugins.document.DocumentMK.FAST_DIFF;
import static org.apache.jackrabbit.oak.plugins.document.DocumentMK.MANY_CHILDREN_THRESHOLD;
+import static org.apache.jackrabbit.oak.plugins.document.JournalEntry.fillExternalChanges;
import static org.apache.jackrabbit.oak.plugins.document.UpdateOp.Key;
import static org.apache.jackrabbit.oak.plugins.document.UpdateOp.Operation;
+import static org.apache.jackrabbit.oak.plugins.document.util.Utils.asStringValueIterable;
import static org.apache.jackrabbit.oak.plugins.document.util.Utils.getIdFromPath;
+import static org.apache.jackrabbit.oak.plugins.document.util.Utils.pathToId;
import static org.apache.jackrabbit.oak.plugins.document.util.Utils.unshareString;
import java.io.Closeable;
@@ -81,6 +86,7 @@ import org.apache.jackrabbit.oak.plugins
import org.apache.jackrabbit.oak.spi.blob.BlobStore;
import org.apache.jackrabbit.oak.commons.json.JsopStream;
import org.apache.jackrabbit.oak.commons.json.JsopWriter;
+import org.apache.jackrabbit.oak.commons.sort.StringSort;
import org.apache.jackrabbit.oak.api.Blob;
import org.apache.jackrabbit.oak.api.CommitFailedException;
import org.apache.jackrabbit.oak.cache.CacheStats;
@@ -235,6 +241,12 @@ public final class DocumentNodeStore
private final Map<String, String> splitCandidates = Maps.newConcurrentMap();
/**
+ * Summary of changes done by this cluster node to persist by the background
+ * update thread.
+ */
+ private JournalEntry changes;
+
+ /**
* The last known revision for each cluster instance.
*
* Key: the machine id, value: revision.
@@ -347,6 +359,8 @@ public final class DocumentNodeStore
private final VersionGarbageCollector versionGarbageCollector;
+ private final JournalGarbageCollector journalGarbageCollector;
+
private final Executor executor;
private final LastRevRecoveryAgent lastRevRecoveryAgent;
@@ -370,6 +384,7 @@ public final class DocumentNodeStore
s = new LoggingDocumentStoreWrapper(s);
}
this.store = s;
+ this.changes = Collection.JOURNAL.newDocument(s);
this.executor = builder.getExecutor();
this.clock = builder.getClock();
int cid = builder.getClusterId();
@@ -389,6 +404,7 @@ public final class DocumentNodeStore
this.asyncDelay = builder.getAsyncDelay();
this.versionGarbageCollector = new VersionGarbageCollector(
this, builder.createVersionGCSupport());
+ this.journalGarbageCollector = new JournalGarbageCollector(this);
this.lastRevRecoveryAgent = new LastRevRecoveryAgent(this);
this.disableBranches = builder.isDisableBranches();
this.missing = new DocumentNodeState(this, "MISSING", new Revision(0, 0, 0)) {
@@ -416,7 +432,8 @@ public final class DocumentNodeStore
checkpoints = new Checkpoints(this);
// check if root node exists
- if (store.find(Collection.NODES, Utils.getIdFromPath("/")) == null) {
+ NodeDocument rootDoc = store.find(NODES, Utils.getIdFromPath("/"));
+ if (rootDoc == null) {
// root node is missing: repository is not initialized
Revision head = newRevision();
Commit commit = new Commit(this, head, null, null);
@@ -437,6 +454,11 @@ public final class DocumentNodeStore
// no revision read from other cluster nodes
setHeadRevision(newRevision());
}
+ // check if _lastRev for our clusterId exists
+ if (!rootDoc.getLastRev().containsKey(clusterId)) {
+ unsavedLastRevisions.put("/", headRevision);
+ backgroundWrite();
+ }
}
getRevisionComparator().add(headRevision, Revision.newRevision(0));
@@ -630,6 +652,8 @@ public final class DocumentNodeStore
Revision before = getHeadRevision();
// apply changes to cache based on before revision
c.applyToCache(before, false);
+ // track modified paths
+ changes.modified(c.getModifiedPaths());
// update head revision
setHeadRevision(c.getRevision());
dispatcher.contentChanged(getRoot(), info);
@@ -1005,15 +1029,13 @@ public final class DocumentNodeStore
}
final Revision readRevision = parent.getLastRevision();
- return transform(getChildren(parent, name, limit).children,
- new Function<String, DocumentNodeState>() {
+ return transform(getChildren(parent, name, limit).children, new Function<String, DocumentNodeState>() {
@Override
public DocumentNodeState apply(String input) {
String p = concat(parent.getPath(), input);
DocumentNodeState result = getNode(p, readRevision);
if (result == null) {
- throw new DocumentStoreException("DocumentNodeState is null for revision " + readRevision + " of " + p
- + " (aborting getChildNodes())");
+ throw new DocumentStoreException("DocumentNodeState is null for revision " + readRevision + " of " + p + " (aborting getChildNodes())");
}
return result;
}
@@ -1032,10 +1054,8 @@ public final class DocumentNodeStore
path, readRevision);
return null;
}
- final DocumentNodeState result = doc.getNodeAtRevision(this,
- readRevision, lastRevision);
- PERFLOG.end(start, 1, "readNode: path={}, readRevision={}", path,
- readRevision);
+ final DocumentNodeState result = doc.getNodeAtRevision(this, readRevision, lastRevision);
+ PERFLOG.end(start, 1, "readNode: path={}, readRevision={}", path, readRevision);
return result;
}
@@ -1052,11 +1072,11 @@ public final class DocumentNodeStore
* @param changed the list of changed child nodes.
*
*/
- public void applyChanges(Revision rev, String path,
- boolean isNew, boolean pendingLastRev,
- boolean isBranchCommit, List<String> added,
- List<String> removed, List<String> changed,
- DiffCache.Entry cacheEntry) {
+ void applyChanges(Revision rev, String path,
+ boolean isNew, boolean pendingLastRev,
+ boolean isBranchCommit, List<String> added,
+ List<String> removed, List<String> changed,
+ DiffCache.Entry cacheEntry) {
LastRevTracker tracker = createTracker(rev);
if (disableBranches) {
if (pendingLastRev) {
@@ -1086,13 +1106,13 @@ public final class DocumentNodeStore
// update diff cache
JsopWriter w = new JsopStream();
for (String p : added) {
- w.tag('+').key(PathUtils.getName(p)).object().endObject().newline();
+ w.tag('+').key(PathUtils.getName(p)).object().endObject();
}
for (String p : removed) {
- w.tag('-').value(PathUtils.getName(p)).newline();
+ w.tag('-').value(PathUtils.getName(p));
}
for (String p : changed) {
- w.tag('^').key(PathUtils.getName(p)).object().endObject().newline();
+ w.tag('^').key(PathUtils.getName(p)).object().endObject();
}
cacheEntry.append(path, w.toString());
@@ -1133,6 +1153,15 @@ public final class DocumentNodeStore
}
/**
+ * Called when a branch is merged.
+ *
+ * @param revisions the revisions of the merged branch commits.
+ */
+ void revisionsMerged(@Nonnull Iterable<Revision> revisions) {
+ changes.branchCommit(revisions);
+ }
+
+ /**
* Updates a commit root document.
*
* @param commit the updates to apply on the commit root document.
@@ -1308,6 +1337,7 @@ public final class DocumentNodeStore
UpdateOp op = new UpdateOp(Utils.getIdFromPath("/"), false);
NodeDocument.setModified(op, commit.getRevision());
if (b != null) {
+ commit.addBranchCommits(b);
Iterator<Revision> mergeCommits = commit.getMergeRevisions().iterator();
for (Revision rev : b.getCommits()) {
rev = rev.asTrunkRevision();
@@ -1673,6 +1703,8 @@ public final class DocumentNodeStore
// then we saw this new revision (from another cluster node)
Revision otherSeen = Revision.newRevision(0);
+ StringSort externalSort = JournalEntry.newSorter();
+
Map<Revision, Revision> externalChanges = Maps.newHashMap();
for (Map.Entry<Integer, Revision> e : lastRevMap.entrySet()) {
int machineId = e.getKey();
@@ -1693,6 +1725,16 @@ public final class DocumentNodeStore
|| r.getTimestamp() > revisionPurgeMillis()) {
externalChanges.put(r, otherSeen);
}
+ // collect external changes
+ if (last != null && externalSort != null) {
+ // add changes for this particular clusterId to the externalSort
+ try {
+ fillExternalChanges(externalSort, last, r, store);
+ } catch (IOException e1) {
+ LOG.error("backgroundRead: Exception while reading external changes from journal: "+e1, e1);
+ externalSort = null;
+ }
+ }
}
}
@@ -1701,9 +1743,38 @@ public final class DocumentNodeStore
if (!externalChanges.isEmpty()) {
// invalidate caches
- stats.cacheStats = store.invalidateCache();
- // TODO only invalidate affected items
- docChildrenCache.invalidateAll();
+ if (externalSort == null) {
+ // if no externalSort available, then invalidate the classic way: everything
+ stats.cacheStats = store.invalidateCache();
+ docChildrenCache.invalidateAll();
+ } else {
+ try {
+ externalSort.sort();
+ stats.cacheStats = store.invalidateCache(pathToId(externalSort));
+ // OAK-3002: only invalidate affected items (using journal)
+ long origSize = docChildrenCache.size();
+ if (origSize == 0) {
+ // if docChildrenCache is empty, don't bother
+ // calling invalidateAll either way
+ // (esp calling invalidateAll(Iterable) will
+ // potentially iterate over all keys even though
+ // there's nothing to be deleted)
+ LOG.trace("backgroundRead: docChildrenCache nothing to invalidate");
+ } else {
+ // however, if the docChildrenCache is not empty,
+ // use the invalidateAll(Iterable) variant,
+ // passing it a Iterable<StringValue>, as that's
+ // what is contained in the cache
+ docChildrenCache.invalidateAll(asStringValueIterable(externalSort));
+ long newSize = docChildrenCache.size();
+ LOG.trace("backgroundRead: docChildrenCache invalidation result: orig: {}, new: {} ", origSize, newSize);
+ }
+ } catch (Exception ioe) {
+ LOG.error("backgroundRead: got IOException during external sorting/cache invalidation (as a result, invalidating entire cache): "+ioe, ioe);
+ stats.cacheStats = store.invalidateCache();
+ docChildrenCache.invalidateAll();
+ }
+ }
stats.cacheInvalidationTime = clock.getTime() - time;
time = clock.getTime();
@@ -1712,7 +1783,6 @@ public final class DocumentNodeStore
backgroundOperationLock.writeLock().lock();
try {
stats.lock = clock.getTime() - time;
- time = clock.getTime();
// the latest revisions of the current cluster node
// happened before the latest revisions of other cluster nodes
@@ -1721,9 +1791,24 @@ public final class DocumentNodeStore
for (Map.Entry<Revision, Revision> e : externalChanges.entrySet()) {
revisionComparator.add(e.getKey(), e.getValue());
}
+
+ Revision oldHead = headRevision;
// the new head revision is after other revisions
setHeadRevision(newRevision());
if (dispatchChange) {
+ time = clock.getTime();
+ if (externalSort != null) {
+ // then there were external changes and reading them
+ // was successful -> apply them to the diff cache
+ try {
+ JournalEntry.applyTo(externalSort, diffCache, oldHead, headRevision);
+ } catch (Exception e1) {
+ LOG.error("backgroundRead: Exception while processing external changes from journal: "+e1, e1);
+ }
+ }
+ stats.populateDiffCache = clock.getTime() - time;
+ time = clock.getTime();
+
dispatcher.contentChanged(getRoot().fromExternalChange(), null);
}
} finally {
@@ -1742,6 +1827,7 @@ public final class DocumentNodeStore
CacheInvalidationStats cacheStats;
long readHead;
long cacheInvalidationTime;
+ long populateDiffCache;
long lock;
long dispatchChanges;
long purge;
@@ -1756,6 +1842,7 @@ public final class DocumentNodeStore
"cacheStats:" + cacheStatsMsg +
", head:" + readHead +
", cache:" + cacheInvalidationTime +
+ ", diff: " + populateDiffCache +
", lock:" + lock +
", dispatch:" + dispatchChanges +
", purge:" + purge +
@@ -1844,7 +1931,15 @@ public final class DocumentNodeStore
}
BackgroundWriteStats backgroundWrite() {
- return unsavedLastRevisions.persist(this, backgroundOperationLock.writeLock());
+ return unsavedLastRevisions.persist(this, new UnsavedModifications.Snapshot() {
+ @Override
+ public void acquiring() {
+ if (store.create(JOURNAL,
+ singletonList(changes.asUpdateOp(getHeadRevision())))) {
+ changes = JOURNAL.newDocument(getDocumentStore());
+ }
+ }
+ }, backgroundOperationLock.writeLock());
}
//-----------------------------< internal >---------------------------------
@@ -1966,19 +2061,23 @@ public final class DocumentNodeStore
case '^': {
String name = unshareString(t.readString());
t.read(':');
- if (t.matches('{')) {
- t.read('}');
- continueComparison = diff.childNodeChanged(name,
- base.getChildNode(name),
- node.getChildNode(name));
- } else if (t.matches('[')) {
- // ignore multi valued property
- while (t.read() != ']') {
- // skip values
+ t.read('{');
+ t.read('}');
+ NodeState baseChild = base.getChildNode(name);
+ NodeState nodeChild = node.getChildNode(name);
+ if (baseChild.exists()) {
+ if (nodeChild.exists()) {
+ continueComparison = diff.childNodeChanged(name,
+ baseChild, nodeChild);
+ } else {
+ continueComparison = diff.childNodeDeleted(name,
+ baseChild);
}
} else {
- // ignore single valued property
- t.read();
+ if (nodeChild.exists()) {
+ continueComparison = diff.childNodeAdded(name,
+ nodeChild);
+ }
}
break;
}
@@ -2069,13 +2168,14 @@ public final class DocumentNodeStore
}
}
+ String diff = w.toString();
if (debug) {
long end = now();
- LOG.debug("Diff performed via '{}' at [{}] between revisions [{}] => [{}] took {} ms ({} ms)",
+ LOG.debug("Diff performed via '{}' at [{}] between revisions [{}] => [{}] took {} ms ({} ms), diff '{}'",
diffAlgo, from.getPath(), fromRev, toRev,
- end - start, getChildrenDoneIn - start);
+ end - start, getChildrenDoneIn - start, diff);
}
- return w.toString();
+ return diff;
}
private void diffManyChildren(JsopWriter w, String path, Revision fromRev, Revision toRev) {
@@ -2123,17 +2223,17 @@ public final class DocumentNodeStore
if (a == null && b == null) {
// ok
} else if (a == null || b == null || !a.equals(b)) {
- w.tag('^').key(name).object().endObject().newline();
+ w.tag('^').key(name).object().endObject();
}
} else {
// does not exist in toRev -> was removed
- w.tag('-').value(name).newline();
+ w.tag('-').value(name);
}
} else {
// does not exist in fromRev
if (toNode != null) {
// exists in toRev
- w.tag('+').key(name).object().endObject().newline();
+ w.tag('+').key(name).object().endObject();
} else {
// does not exist in either revisions
// -> do nothing
@@ -2160,7 +2260,7 @@ public final class DocumentNodeStore
Set<String> childrenSet = Sets.newHashSet(toChildren.children);
for (String n : fromChildren.children) {
if (!childrenSet.contains(n)) {
- w.tag('-').value(n).newline();
+ w.tag('-').value(n);
} else {
String path = concat(parentPath, n);
DocumentNodeState n1 = getNode(path, fromRev);
@@ -2172,14 +2272,14 @@ public final class DocumentNodeStore
checkNotNull(n1, "Node at [%s] not found for fromRev [%s]", path, fromRev);
checkNotNull(n2, "Node at [%s] not found for toRev [%s]", path, toRev);
if (!n1.getLastRevision().equals(n2.getLastRevision())) {
- w.tag('^').key(n).object().endObject().newline();
+ w.tag('^').key(n).object().endObject();
}
}
}
childrenSet = Sets.newHashSet(fromChildren.children);
for (String n : toChildren.children) {
if (!childrenSet.contains(n)) {
- w.tag('+').key(n).object().endObject().newline();
+ w.tag('+').key(n).object().endObject();
}
}
}
@@ -2465,6 +2565,12 @@ public final class DocumentNodeStore
public VersionGarbageCollector getVersionGarbageCollector() {
return versionGarbageCollector;
}
+
+ @Nonnull
+ public JournalGarbageCollector getJournalGarbageCollector() {
+ return journalGarbageCollector;
+ }
+
@Nonnull
public LastRevRecoveryAgent getLastRevRecoveryAgent() {
return lastRevRecoveryAgent;
Modified: jackrabbit/oak/branches/1.0/oak-core/src/main/java/org/apache/jackrabbit/oak/plugins/document/DocumentNodeStoreService.java
URL: http://svn.apache.org/viewvc/jackrabbit/oak/branches/1.0/oak-core/src/main/java/org/apache/jackrabbit/oak/plugins/document/DocumentNodeStoreService.java?rev=1695297&r1=1695296&r2=1695297&view=diff
==============================================================================
--- jackrabbit/oak/branches/1.0/oak-core/src/main/java/org/apache/jackrabbit/oak/plugins/document/DocumentNodeStoreService.java (original)
+++ jackrabbit/oak/branches/1.0/oak-core/src/main/java/org/apache/jackrabbit/oak/plugins/document/DocumentNodeStoreService.java Tue Aug 11 12:55:41 2015
@@ -33,6 +33,7 @@ import com.mongodb.DB;
import com.mongodb.MongoClient;
import com.mongodb.MongoClientOptions;
import com.mongodb.MongoClientURI;
+
import org.apache.felix.scr.annotations.Activate;
import org.apache.felix.scr.annotations.Component;
import org.apache.felix.scr.annotations.ConfigurationPolicy;
@@ -164,6 +165,23 @@ public class DocumentNodeStoreService {
*/
public static final String CUSTOM_BLOB_STORE = "customBlobStore";
+ private static final long DEFAULT_JOURNAL_GC_INTERVAL_MILLIS = 5*60*1000; // default is 5min
+ @Property(longValue = DEFAULT_JOURNAL_GC_INTERVAL_MILLIS,
+ label = "Journal Garbage Collection Interval (millis)",
+ description = "Long value indicating interval (in milliseconds) with which the "
+ + "journal (for external changes) is cleaned up. Default is " + DEFAULT_JOURNAL_GC_INTERVAL_MILLIS
+ )
+ private static final String PROP_JOURNAL_GC_INTERVAL_MILLIS = "journalGCInterval";
+
+ private static final long DEFAULT_JOURNAL_GC_MAX_AGE_MILLIS = 6*60*60*1000; // default is 6hours
+ @Property(longValue = DEFAULT_JOURNAL_GC_MAX_AGE_MILLIS,
+ label = "Maximum Age of Journal Entries (millis)",
+ description = "Long value indicating max age (in milliseconds) that "
+ + "journal (for external changes) entries are kept (older ones are candidates for gc). "
+ + "Default is " + DEFAULT_JOURNAL_GC_MAX_AGE_MILLIS
+ )
+ private static final String PROP_JOURNAL_GC_MAX_AGE_MILLIS = "journalGCMaxAge";
+
private static final long MB = 1024 * 1024;
private static enum DocumentStoreType {
@@ -339,6 +357,7 @@ public class DocumentNodeStoreService {
registerJMXBeans(mk.getNodeStore());
registerLastRevRecoveryJob(mk.getNodeStore());
+ registerJournalGC(mk.getNodeStore());
NodeStore store;
if (useMK) {
@@ -518,6 +537,23 @@ public class DocumentNodeStoreService {
recoverJob, TimeUnit.MILLISECONDS.toSeconds(leaseTime)));
}
+ private void registerJournalGC(final DocumentNodeStore nodeStore) {
+ long journalGCInterval = toLong(context.getProperties().get(PROP_JOURNAL_GC_INTERVAL_MILLIS),
+ DEFAULT_JOURNAL_GC_INTERVAL_MILLIS);
+ final long journalGCMaxAge = toLong(context.getProperties().get(PROP_JOURNAL_GC_MAX_AGE_MILLIS),
+ DEFAULT_JOURNAL_GC_MAX_AGE_MILLIS);
+ Runnable journalGCJob = new Runnable() {
+
+ @Override
+ public void run() {
+ nodeStore.getJournalGarbageCollector().gc(journalGCMaxAge, TimeUnit.MILLISECONDS);
+ }
+
+ };
+ registrations.add(WhiteboardUtils.scheduleWithFixedDelay(whiteboard,
+ journalGCJob, TimeUnit.MILLISECONDS.toSeconds(journalGCInterval), true/*runOnSingleClusterNode*/));
+ }
+
private Object prop(String propName) {
return prop(propName, PREFIX + propName);
}
Modified: jackrabbit/oak/branches/1.0/oak-core/src/main/java/org/apache/jackrabbit/oak/plugins/document/DocumentStore.java
URL: http://svn.apache.org/viewvc/jackrabbit/oak/branches/1.0/oak-core/src/main/java/org/apache/jackrabbit/oak/plugins/document/DocumentStore.java?rev=1695297&r1=1695296&r2=1695297&view=diff
==============================================================================
--- jackrabbit/oak/branches/1.0/oak-core/src/main/java/org/apache/jackrabbit/oak/plugins/document/DocumentStore.java (original)
+++ jackrabbit/oak/branches/1.0/oak-core/src/main/java/org/apache/jackrabbit/oak/plugins/document/DocumentStore.java Tue Aug 11 12:55:41 2015
@@ -223,6 +223,13 @@ public interface DocumentStore {
CacheInvalidationStats invalidateCache();
/**
+ * Invalidate the document cache but only with entries that match one
+ * of the keys provided.
+ */
+ @CheckForNull
+ CacheInvalidationStats invalidateCache(Iterable<String> keys);
+
+ /**
* Invalidate the document cache for the given key.
*
* @param <T> the document type
Added: jackrabbit/oak/branches/1.0/oak-core/src/main/java/org/apache/jackrabbit/oak/plugins/document/JournalEntry.java
URL: http://svn.apache.org/viewvc/jackrabbit/oak/branches/1.0/oak-core/src/main/java/org/apache/jackrabbit/oak/plugins/document/JournalEntry.java?rev=1695297&view=auto
==============================================================================
--- jackrabbit/oak/branches/1.0/oak-core/src/main/java/org/apache/jackrabbit/oak/plugins/document/JournalEntry.java (added)
+++ jackrabbit/oak/branches/1.0/oak-core/src/main/java/org/apache/jackrabbit/oak/plugins/document/JournalEntry.java Tue Aug 11 12:55:41 2015
@@ -0,0 +1,505 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.jackrabbit.oak.plugins.document;
+
+import java.io.IOException;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import javax.annotation.CheckForNull;
+import javax.annotation.Nonnull;
+
+import com.google.common.collect.AbstractIterator;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+
+import org.apache.jackrabbit.oak.commons.PathUtils;
+import org.apache.jackrabbit.oak.commons.json.JsopBuilder;
+import org.apache.jackrabbit.oak.commons.json.JsopReader;
+import org.apache.jackrabbit.oak.commons.json.JsopTokenizer;
+import org.apache.jackrabbit.oak.commons.sort.StringSort;
+import org.apache.jackrabbit.oak.plugins.document.util.Utils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static com.google.common.base.Preconditions.checkArgument;
+import static com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.jackrabbit.oak.commons.PathUtils.concat;
+import static org.apache.jackrabbit.oak.plugins.document.Collection.JOURNAL;
+
+/**
+ * Keeps track of changes performed between two consecutive background updates.
+ */
+public final class JournalEntry extends Document {
+
+ private static final Logger LOG = LoggerFactory.getLogger(JournalEntry.class);
+
+ /**
+ * The revision format for external changes:
+ * <clusterId>-<timestamp>-<counter>. The string is prefixed with
+ * "b" if it denotes a branch revision.
+ */
+ private static final String REVISION_FORMAT = "%d-%0" +
+ Long.toHexString(Long.MAX_VALUE).length() + "x-%0" +
+ Integer.toHexString(Integer.MAX_VALUE).length() + "x";
+
+ private static final String CHANGES = "_c";
+
+ private static final String BRANCH_COMMITS = "_bc";
+
+ private static final int READ_CHUNK_SIZE = 100;
+
+ /**
+ * switch to disk after 1MB
+ */
+ private static final int STRING_SORT_OVERFLOW_TO_DISK_THRESHOLD = 1024 * 1024;
+
+ private final DocumentStore store;
+
+ private volatile TreeNode changes = null;
+
+ JournalEntry(DocumentStore store) {
+ this.store = store;
+ }
+
+ static StringSort newSorter() {
+ return new StringSort(STRING_SORT_OVERFLOW_TO_DISK_THRESHOLD, new Comparator<String>() {
+ @Override
+ public int compare(String arg0, String arg1) {
+ return arg0.compareTo(arg1);
+ }
+ });
+ }
+
+ static void applyTo(@Nonnull StringSort externalSort,
+ @Nonnull DiffCache diffCache,
+ @Nonnull Revision from,
+ @Nonnull Revision to) throws IOException {
+ LOG.debug("applyTo: starting for {} to {}", from, to);
+ // note that it is not de-duplicated yet
+ LOG.debug("applyTo: sorting done.");
+
+ final DiffCache.Entry entry = checkNotNull(diffCache).newEntry(from, to, false);
+
+ final Iterator<String> it = externalSort.getIds();
+ if (!it.hasNext()) {
+ // nothing at all? that's quite unusual..
+
+ // we apply this diff as one '/' to the entry then
+ entry.append("/", "");
+ entry.done();
+ return;
+ }
+ String previousPath = it.next();
+ TreeNode node = new TreeNode();
+ node = node.getOrCreatePath(previousPath);
+ int totalCnt = 0;
+ int deDuplicatedCnt = 0;
+ while (it.hasNext()) {
+ totalCnt++;
+ final String currentPath = it.next();
+ if (previousPath.equals(currentPath)) {
+ // de-duplication
+ continue;
+ }
+ final TreeNode currentNode = node.getOrCreatePath(currentPath);
+
+ // 'node' contains one hierarchy line, eg /a, /a/b, /a/b/c, /a/b/c/d
+ // including the children on each level.
+ // these children have not yet been appended to the diffCache entry
+ // and have to be added as soon as the 'currentPath' is not
+ // part of that hierarchy anymore and we 'move elsewhere'.
+ // eg if 'currentPath' is /a/b/e, then we must flush /a/b/c/d and /a/b/c
+ while (node != null && !node.isAncestorOf(currentNode)) {
+ // add parent to the diff entry
+ entry.append(node.getPath(), getChanges(node));
+ deDuplicatedCnt++;
+ node = node.parent;
+ }
+
+ if (node == null) {
+ // we should never go 'passed' the root, hence node should
+ // never be null - if it becomes null anyway, start with
+ // a fresh root:
+ node = new TreeNode();
+ node = node.getOrCreatePath(currentPath);
+ } else {
+ // this is the normal route: we add a direct or grand-child
+ // node to the current node:
+ node = currentNode;
+ }
+ previousPath = currentPath;
+ }
+
+ // once we're done we still have the last hierarchy line contained in 'node',
+ // eg /x, /x/y, /x/y/z
+ // and that one we must now append to the diff cache entry:
+ while (node != null) {
+ entry.append(node.getPath(), getChanges(node));
+ deDuplicatedCnt++;
+ node = node.parent;
+ }
+
+ // and finally: mark the diff cache entry as 'done':
+ entry.done();
+ LOG.debug("applyTo: done. totalCnt: {}, deDuplicatedCnt: {}", totalCnt, deDuplicatedCnt);
+ }
+
+ /**
+ * Reads all external changes between the two given revisions (with the same
+ * clusterId) from the journal and appends the paths therein to the provided
+ * sorter.
+ *
+ * @param sorter the StringSort to which all externally changed paths
+ * between the provided revisions will be added
+ * @param from the lower bound of the revision range (exclusive).
+ * @param to the upper bound of the revision range (inclusive).
+ * @param store the document store to query.
+ * @throws IOException
+ */
+ static void fillExternalChanges(@Nonnull StringSort sorter,
+ @Nonnull Revision from,
+ @Nonnull Revision to,
+ @Nonnull DocumentStore store)
+ throws IOException {
+ checkArgument(checkNotNull(from).getClusterId() == checkNotNull(to).getClusterId());
+
+ // to is inclusive, but DocumentStore.query() toKey is exclusive
+ final String inclusiveToId = asId(to);
+ to = new Revision(to.getTimestamp(), to.getCounter() + 1,
+ to.getClusterId(), to.isBranch());
+
+ // read in chunks to support very large sets of changes between
+ // subsequent background reads to do this, provide a (TODO eventually configurable)
+ // limit for the number of entries to be returned per query if the
+ // number of elements returned by the query is exactly the provided
+ // limit, then loop and do subsequent queries
+ final String toId = asId(to);
+ String fromId = asId(from);
+ while (true) {
+ if (fromId.equals(inclusiveToId)) {
+ // avoid query if from and to are off by just 1 counter (which
+ // we do due to exclusiveness of query borders) as in this case
+ // the query will always be empty anyway - so avoid doing the
+ // query in the first place
+ break;
+ }
+ List<JournalEntry> partialResult = store.query(JOURNAL, fromId, toId, READ_CHUNK_SIZE);
+
+ for (JournalEntry d : partialResult) {
+ d.addTo(sorter);
+ }
+ if (partialResult.size() < READ_CHUNK_SIZE) {
+ break;
+ }
+ // otherwise set 'fromId' to the last entry just processed
+ // that works fine as the query is non-inclusive (ie does not
+ // include the from which we'd otherwise double-process)
+ fromId = partialResult.get(partialResult.size() - 1).getId();
+ }
+ }
+
+ long getRevisionTimestamp() {
+ final String[] parts = getId().split("-");
+ return Long.parseLong(parts[1], 16);
+ }
+
+ void modified(String path) {
+ TreeNode node = getChanges();
+ for (String name : PathUtils.elements(path)) {
+ node = node.getOrCreate(name);
+ }
+ }
+
+ void modified(Iterable<String> paths) {
+ for (String p : paths) {
+ modified(p);
+ }
+ }
+
+ void branchCommit(@Nonnull Iterable<Revision> revisions) {
+ String branchCommits = (String) get(BRANCH_COMMITS);
+ if (branchCommits == null) {
+ branchCommits = "";
+ }
+ for (Revision r : revisions) {
+ if (branchCommits.length() > 0) {
+ branchCommits += ",";
+ }
+ branchCommits += asId(r.asBranchRevision());
+ }
+ put(BRANCH_COMMITS, branchCommits);
+ }
+
+ String getChanges(String path) {
+ TreeNode node = getNode(path);
+ if (node == null) {
+ return "";
+ }
+ return getChanges(node);
+ }
+
+ UpdateOp asUpdateOp(@Nonnull Revision revision) {
+ String id = asId(revision);
+ UpdateOp op = new UpdateOp(id, true);
+ op.set(ID, id);
+ op.set(CHANGES, getChanges().serialize());
+ String bc = (String) get(BRANCH_COMMITS);
+ if (bc != null) {
+ op.set(BRANCH_COMMITS, bc);
+ }
+ return op;
+ }
+
+ void addTo(final StringSort sort) throws IOException {
+ TreeNode n = getChanges();
+ TraversingVisitor v = new TraversingVisitor() {
+
+ @Override
+ public void node(TreeNode node, String path) throws IOException {
+ sort.add(path);
+ }
+ };
+ n.accept(v, "/");
+ for (JournalEntry e : getBranchCommits()) {
+ e.getChanges().accept(v, "/");
+ }
+ }
+
+ /**
+ * Returns the branch commits that are related to this journal entry.
+ *
+ * @return the branch commits.
+ */
+ @Nonnull
+ Iterable<JournalEntry> getBranchCommits() {
+ final List<String> ids = Lists.newArrayList();
+ String bc = (String) get(BRANCH_COMMITS);
+ if (bc != null) {
+ for (String id : bc.split(",")) {
+ ids.add(id);
+ }
+ }
+ return new Iterable<JournalEntry>() {
+ @Override
+ public Iterator<JournalEntry> iterator() {
+ return new AbstractIterator<JournalEntry>() {
+
+ private final Iterator<String> it = ids.iterator();
+
+ @Override
+ protected JournalEntry computeNext() {
+ if (!it.hasNext()) {
+ return endOfData();
+ }
+ String id = it.next();
+ JournalEntry d = store.find(JOURNAL, id);
+ if (d == null) {
+ throw new IllegalStateException(
+ "Missing external change for branch revision: " + id);
+ }
+ return d;
+ }
+ };
+ }
+ };
+ }
+
+ //-----------------------------< internal >---------------------------------
+
+ private static String getChanges(TreeNode node) {
+ JsopBuilder builder = new JsopBuilder();
+ for (String name : node.keySet()) {
+ builder.tag('^');
+ builder.key(name);
+ builder.object().endObject();
+ }
+ return builder.toString();
+ }
+
+ static String asId(@Nonnull Revision revision) {
+ checkNotNull(revision);
+ String s = String.format(REVISION_FORMAT, revision.getClusterId(), revision.getTimestamp(), revision.getCounter());
+ if (revision.isBranch()) {
+ s = "b" + s;
+ }
+ return s;
+ }
+
+ @CheckForNull
+ private TreeNode getNode(String path) {
+ TreeNode node = getChanges();
+ for (String name : PathUtils.elements(path)) {
+ node = node.get(name);
+ if (node == null) {
+ return null;
+ }
+ }
+ return node;
+ }
+
+ @Nonnull
+ private TreeNode getChanges() {
+ if (changes == null) {
+ TreeNode node = new TreeNode();
+ String c = (String) get(CHANGES);
+ if (c != null) {
+ node.parse(new JsopTokenizer(c));
+ }
+ changes = node;
+ }
+ return changes;
+ }
+
+ private static final class TreeNode {
+
+ private static final Map<String, TreeNode> NO_CHILDREN = Collections.emptyMap();
+
+ private Map<String, TreeNode> children = NO_CHILDREN;
+
+ private final TreeNode parent;
+ private final String name;
+
+ TreeNode() {
+ this(null, "");
+ }
+
+ TreeNode(TreeNode parent, String name) {
+ checkArgument(!name.contains("/"),
+ "name must not contain '/': {}", name);
+
+ this.parent = parent;
+ this.name = name;
+ }
+
+ TreeNode getOrCreatePath(String path) {
+ TreeNode n = getRoot();
+ for (String name : PathUtils.elements(path)) {
+ n = n.getOrCreate(name);
+ }
+ return n;
+ }
+
+ boolean isAncestorOf(TreeNode other) {
+ TreeNode n = other;
+ while (n.parent != null) {
+ if (this == n.parent) {
+ return true;
+ }
+ n = n.parent;
+ }
+ return false;
+ }
+
+ @Nonnull
+ private TreeNode getRoot() {
+ TreeNode n = this;
+ while (n.parent != null) {
+ n = n.parent;
+ }
+ return n;
+ }
+
+ private String getPath() {
+ return buildPath(new StringBuilder()).toString();
+ }
+
+ private StringBuilder buildPath(StringBuilder sb) {
+ if (parent != null) {
+ parent.buildPath(sb);
+ if (parent.parent != null) {
+ // only add slash if parent is not the root
+ sb.append("/");
+ }
+ } else {
+ // this is the root
+ sb.append("/");
+ }
+ sb.append(name);
+ return sb;
+ }
+
+ void parse(JsopReader reader) {
+ reader.read('{');
+ if (!reader.matches('}')) {
+ do {
+ String name = Utils.unescapePropertyName(reader.readString());
+ reader.read(':');
+ getOrCreate(name).parse(reader);
+ } while (reader.matches(','));
+ reader.read('}');
+ }
+ }
+
+ String serialize() {
+ JsopBuilder builder = new JsopBuilder();
+ builder.object();
+ toJson(builder);
+ builder.endObject();
+ return builder.toString();
+ }
+
+ @Nonnull
+ Set<String> keySet() {
+ return children.keySet();
+ }
+
+ @CheckForNull
+ TreeNode get(String name) {
+ return children.get(name);
+ }
+
+ void accept(TraversingVisitor visitor, String path) throws IOException {
+ visitor.node(this, path);
+ for (Map.Entry<String, TreeNode> entry : children.entrySet()) {
+ entry.getValue().accept(visitor, concat(path, entry.getKey()));
+ }
+ }
+
+ private void toJson(JsopBuilder builder) {
+ for (Map.Entry<String, TreeNode> entry : children.entrySet()) {
+ builder.key(Utils.escapePropertyName(entry.getKey()));
+ builder.object();
+ entry.getValue().toJson(builder);
+ builder.endObject();
+ }
+ }
+
+ @Nonnull
+ private TreeNode getOrCreate(String name) {
+ if (children == NO_CHILDREN) {
+ children = Maps.newHashMap();
+ }
+ TreeNode c = children.get(name);
+ if (c == null) {
+ c = new TreeNode(this, name);
+ children.put(name, c);
+ }
+ return c;
+ }
+ }
+
+ private interface TraversingVisitor {
+
+ void node(TreeNode node, String path) throws IOException;
+ }
+
+}
Propchange: jackrabbit/oak/branches/1.0/oak-core/src/main/java/org/apache/jackrabbit/oak/plugins/document/JournalEntry.java
------------------------------------------------------------------------------
svn:eol-style = native
Added: jackrabbit/oak/branches/1.0/oak-core/src/main/java/org/apache/jackrabbit/oak/plugins/document/JournalGarbageCollector.java
URL: http://svn.apache.org/viewvc/jackrabbit/oak/branches/1.0/oak-core/src/main/java/org/apache/jackrabbit/oak/plugins/document/JournalGarbageCollector.java?rev=1695297&view=auto
==============================================================================
--- jackrabbit/oak/branches/1.0/oak-core/src/main/java/org/apache/jackrabbit/oak/plugins/document/JournalGarbageCollector.java (added)
+++ jackrabbit/oak/branches/1.0/oak-core/src/main/java/org/apache/jackrabbit/oak/plugins/document/JournalGarbageCollector.java Tue Aug 11 12:55:41 2015
@@ -0,0 +1,141 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.jackrabbit.oak.plugins.document;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.TimeUnit;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.base.Stopwatch;
+
+/**
+ * The JournalGarbageCollector can clean up JournalEntries that are older than a
+ * particular age.
+ * <p/>
+ * It would typically be invoked in conjunction with the VersionGarbageCollector
+ * but must not be confused with that one - 'journal' refers to the separate
+ * collection that contains changed paths per background writes used for
+ * observation.
+ */
+public class JournalGarbageCollector {
+
+ //copied from VersionGarbageCollector:
+ private static final int DELETE_BATCH_SIZE = 450;
+
+ private final DocumentStore ds;
+
+ private static final Logger log = LoggerFactory.getLogger(JournalGarbageCollector.class);
+
+ public JournalGarbageCollector(DocumentNodeStore nodeStore) {
+ this.ds = nodeStore.getDocumentStore();
+ }
+
+ /**
+ * Deletes entries in the journal that are older than the given
+ * maxRevisionAge.
+ *
+ * @param maxRevisionAge entries older than this age will be removed
+ * @param unit the timeunit for maxRevisionAge
+ * @return the number of entries that have been removed
+ */
+ public int gc(long maxRevisionAge, TimeUnit unit) {
+ long maxRevisionAgeInMillis = unit.toMillis(maxRevisionAge);
+ if (log.isDebugEnabled()) {
+ log.debug("gc: Journal garbage collection starts with maxAge: {} min.", TimeUnit.MILLISECONDS.toMinutes(maxRevisionAgeInMillis));
+ }
+ Stopwatch sw = Stopwatch.createStarted();
+
+ // the journal has ids of the following format:
+ // 1-0000014db9aaf710-00000001
+ // whereas the first number is the cluster node id.
+ // now, this format prevents from doing a generic
+ // query to get all 'old' entries, as the documentstore
+ // can only query for a sequential list of entries.
+ // (and the cluster node id here partitions the set
+ // of entries that we have to delete)
+ // To account for that, we simply iterate over all
+ // cluster node ids and clean them up individually.
+ // Note that there are possible alternatives, such
+ // as: let each node clean up its own old entries
+ // but the chosen path is also quite simple: it can
+ // be started on any instance - but best on only one.
+ // if it's run on multiple concurrently, then they
+ // will compete at deletion, which is not optimal
+ // due to performance, but does not harm.
+
+ // 1. get the list of cluster node ids
+ final List<ClusterNodeInfoDocument> clusterNodeInfos = ClusterNodeInfoDocument.all(ds);
+ int numDeleted = 0;
+ for (ClusterNodeInfoDocument clusterNodeInfoDocument : clusterNodeInfos) {
+ // current algorithm is to simply look at all cluster nodes
+ // irrespective of whether they are active or inactive etc.
+ // this could be optimized for inactive ones: at some point, all
+ // journal entries of inactive ones would have been cleaned up
+ // and at that point we could stop including those long-time-inactive ones.
+ // that 'long time' aspect would have to be tracked though, to be sure
+ // we don't leave garbage.
+ // so simpler is to quickly do a query even for long-time inactive ones
+ final int clusterNodeId = clusterNodeInfoDocument.getClusterId();
+
+ // 2. iterate over that list and do a query with
+ // a limit of 'batch size'
+ boolean branch = false;
+ long startPointer = 0;
+ while (true) {
+ String fromKey = JournalEntry.asId(new Revision(startPointer, 0, clusterNodeId, branch));
+ String toKey = JournalEntry.asId(new Revision(System.currentTimeMillis() - maxRevisionAgeInMillis, Integer.MAX_VALUE, clusterNodeId, branch));
+ int limit = DELETE_BATCH_SIZE;
+ List<JournalEntry> deletionBatch = ds.query(Collection.JOURNAL, fromKey, toKey, limit);
+ if (deletionBatch.size() > 0) {
+ ds.remove(Collection.JOURNAL, asKeys(deletionBatch));
+ numDeleted += deletionBatch.size();
+ }
+ if (deletionBatch.size() < limit) {
+ if (!branch) {
+ // do the same for branches:
+ // this will start at the beginning again with branch set to true
+ // and eventually finish too
+ startPointer = 0;
+ branch = true;
+ continue;
+ }
+ break;
+ }
+ startPointer = deletionBatch.get(deletionBatch.size() - 1).getRevisionTimestamp();
+ }
+ }
+
+ sw.stop();
+
+ log.info("gc: Journal garbage collection took {}, deleted {} entries that were older than {} min.", sw, numDeleted, TimeUnit.MILLISECONDS.toMinutes(maxRevisionAgeInMillis));
+ return numDeleted;
+ }
+
+ private List<String> asKeys(List<JournalEntry> deletionBatch) {
+ final List<String> keys = new ArrayList<String>(deletionBatch.size());
+ for (JournalEntry e : deletionBatch) {
+ keys.add(e.getId());
+ }
+ return keys;
+ }
+
+}
Propchange: jackrabbit/oak/branches/1.0/oak-core/src/main/java/org/apache/jackrabbit/oak/plugins/document/JournalGarbageCollector.java
------------------------------------------------------------------------------
svn:eol-style = native
Re: svn commit: r1695297 [1/3] - in /jackrabbit/oak/branches/1.0: ./
oak-commons/src/main/java/org/apache/jackrabbit/oak/commons/sort/
oak-core/src/main/java/org/apache/jackrabbit/oak/plugins/document/
oak-core/src/main/java/org/apache/jackrabbit/oak/p...
Posted by Marcel Reutegger <mr...@adobe.com>.
On 11/08/15 18:22, "Julian Reschke" wrote:
>On 2015-08-11 14:55, mreutegg@apache.org wrote:
>> Author: mreutegg
>> Date: Tue Aug 11 12:55:41 2015
>> New Revision: 1695297
>>
>> URL: http://svn.apache.org/r1695297
>> Log:
>> OAK-2829: Comparing node states for external changes is too slow
>> OAK-3002: Optimize docCache and docChildrenCache invalidation by
>>filtering using journal
>>
>> Merged revisions
>>1678023,1678171,1684820,1685590,1685964,1685977,1685989,1686023,1686032,1
>>688179 from trunk
>> ...
>
>In general I'm +1000 on making 1.0 as similar to 1.2 as possible.
>
>In this case however I'm a bit concerned about the timing: this would go
>into Oak 1.0.19 which is also supposed to fix a critical bug that we
>introduced in 1.0.16 (https://issues.apache.org/jira/browse/OAK-3169).
>Is it wise to push this rather large change into a release that people
>running 1.0.16...18 simply *have* to switch to ASAP?
I share your concern, but OAK-2829 is critical as well for clustered
deployments. Without OAK-2829 chances are quite high that an observation
queue fills up at some point and events are delayed for a very long time.
All tests so far look fine and we still have time to perform additional
tests until we release 1.0.19 to identify any regressions.
I think this shows the downside of a rather monolithic core artifact we
currently have. See also the modularization discussion we recently had.
If the DocumentNodeStore was in a separate module, users could e.g. pull
in the fix for OAK-3169 ASAP but delay OAK-2829.
Regards
Marcel
Re: svn commit: r1695297 [1/3] - in /jackrabbit/oak/branches/1.0:
./ oak-commons/src/main/java/org/apache/jackrabbit/oak/commons/sort/ oak-core/src/main/java/org/apache/jackrabbit/oak/plugins/document/
oak-core/src/main/java/org/apache/jackrabbit/oak/plugi...
Posted by Michael Dürig <md...@apache.org>.
On 11.8.15 6:22 , Julian Reschke wrote:
> On 2015-08-11 14:55, mreutegg@apache.org wrote:
>> Author: mreutegg
>> Date: Tue Aug 11 12:55:41 2015
>> New Revision: 1695297
>>
>> URL: http://svn.apache.org/r1695297
>> Log:
>> OAK-2829: Comparing node states for external changes is too slow
>> OAK-3002: Optimize docCache and docChildrenCache invalidation by
>> filtering using journal
>>
>> Merged revisions
>> 1678023,1678171,1684820,1685590,1685964,1685977,1685989,1686023,1686032,1688179
>> from trunk
>> ...
>
> In general I'm +1000 on making 1.0 as similar to 1.2 as possible.
>
> In this case however I'm a bit concerned about the timing: this would go
> into Oak 1.0.19 which is also supposed to fix a critical bug that we
> introduced in 1.0.16 (https://issues.apache.org/jira/browse/OAK-3169).
> Is it wise to push this rather large change into a release that people
> running 1.0.16...18 simply *have* to switch to ASAP?
That's right the point I'm trying to bring across in the modularisation
discussion: if we had proper modules such a change wouldn't impact not
affected customers.
Michael
>
> Best regards, Julian
Re: svn commit: r1695297 [1/3] - in /jackrabbit/oak/branches/1.0: ./
oak-commons/src/main/java/org/apache/jackrabbit/oak/commons/sort/
oak-core/src/main/java/org/apache/jackrabbit/oak/plugins/document/
oak-core/src/main/java/org/apache/jackrabbit/oak/plugi...
Posted by Julian Reschke <ju...@gmx.de>.
On 2015-08-11 14:55, mreutegg@apache.org wrote:
> Author: mreutegg
> Date: Tue Aug 11 12:55:41 2015
> New Revision: 1695297
>
> URL: http://svn.apache.org/r1695297
> Log:
> OAK-2829: Comparing node states for external changes is too slow
> OAK-3002: Optimize docCache and docChildrenCache invalidation by filtering using journal
>
> Merged revisions 1678023,1678171,1684820,1685590,1685964,1685977,1685989,1686023,1686032,1688179 from trunk
> ...
In general I'm +1000 on making 1.0 as similar to 1.2 as possible.
In this case however I'm a bit concerned about the timing: this would go
into Oak 1.0.19 which is also supposed to fix a critical bug that we
introduced in 1.0.16 (https://issues.apache.org/jira/browse/OAK-3169).
Is it wise to push this rather large change into a release that people
running 1.0.16...18 simply *have* to switch to ASAP?
Best regards, Julian