You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@tinkerpop.apache.org by ok...@apache.org on 2016/03/01 15:25:39 UTC
[11/11] incubator-tinkerpop git commit: updated CHANGELOG and upgrade
docs
updated CHANGELOG and upgrade docs
Project: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/commit/42aafabd
Tree: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/tree/42aafabd
Diff: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/diff/42aafabd
Branch: refs/heads/TINKERPOP-1166
Commit: 42aafabdd4413eb1310fb5021c35309d8b599d41
Parents: cbef99d
Author: Marko A. Rodriguez <ok...@gmail.com>
Authored: Tue Mar 1 07:25:22 2016 -0700
Committer: Marko A. Rodriguez <ok...@gmail.com>
Committed: Tue Mar 1 07:25:22 2016 -0700
----------------------------------------------------------------------
CHANGELOG.asciidoc | 17 ++++++
.../upgrade/release-3.2.x-incubating.asciidoc | 54 ++++++++++++++++++++
2 files changed, 71 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/42aafabd/CHANGELOG.asciidoc
----------------------------------------------------------------------
diff --git a/CHANGELOG.asciidoc b/CHANGELOG.asciidoc
index ed3f9cd..49499e9 100644
--- a/CHANGELOG.asciidoc
+++ b/CHANGELOG.asciidoc
@@ -24,6 +24,23 @@ TinkerPop 3.2.0 (XXX)
TinkerPop 3.2.0 (NOT OFFICIALLY RELEASED YET)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+* Added `MemoryComputeKey` for specification of `Memory` keys in `VertexProgram`. (*breaking*)
+* Added `VertexComputeKey` for specification of vertex compute properties in `VertexProgram`. (*breaking*)
+* Added `and`, `or`, and `addAll` to `Operator`.
+* `Memory` API changed to support setting and adding values for reduction. (*breaking*)
+* `Memory` keys can be marked as broadcast and only those values are sent to workers on each iterator.
+* `Memory` keys can be marked transient and thus deleted at the end of the OLAP job.
+* Vertex compute keys can be marked transient and thus deleted at the end of the OLAP job.
+* `VertexProgram` API changed to support `MemoryComputeKey` and `VertexComputeKey`. (*breaking*)
+* `TraversalVertexProgram` able to execute OLAP and OLTP traversal sections dynamically within the same job.
+* Removed `FinalGet` interface as all post processing of reductions should be handled by the reducing step explicitly. (*breaking*)
+* Greatly simplified all `ReducingBarrierStep` implementations as they no longer require `MapReduce` in OLAP.
+* Nearly all steps in OLAP that used `MapReduce` now use `Memory` to do their reductions which expands the list of legal traversals.
+* `GroupStep` simplified with `GroupHelper.GroupMap` no longer being needed. Related to the removal of `FinalGet`.
+* OLAP side-effects that are no longer generated by `MapReduce` are simply stored in `ComputerResult.Memory` w/ no disk persistence needed. (*breaking*)
+* Added `Generate` step interface which states that there could be a final generating phase to a side-effect or reduction (e.g. `GroupStep`).
+* `Barrier` step interface is now the means by which non-parallel steps communicate with their counterparts in OLAP.
+* Added `MemoryComputing` step interface which states that the step uses `MemoryComputeKeys` for its computation in OLAP.
* Added `PeerPressureVertexProgramStep` and `GraphTraversal.peerPressure()`.
* Added `PureTraversal` for handling pure and compiled versions of a `Traversal`. Useful in OLAP.
* Added `ScriptTraversal` which allows for delayed compilation of script-based `Traversals`.
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/42aafabd/docs/src/upgrade/release-3.2.x-incubating.asciidoc
----------------------------------------------------------------------
diff --git a/docs/src/upgrade/release-3.2.x-incubating.asciidoc b/docs/src/upgrade/release-3.2.x-incubating.asciidoc
index 055bb11..1a90ba6 100644
--- a/docs/src/upgrade/release-3.2.x-incubating.asciidoc
+++ b/docs/src/upgrade/release-3.2.x-incubating.asciidoc
@@ -29,9 +29,63 @@ TinkerPop 3.2.0
Please see the link:https://github.com/apache/incubator-tinkerpop/blob/3.2.0-incubating/CHANGELOG.asciidoc#XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX[changelog] for a complete list of all the modifications that are part of this release.
+Upgrading for Users
+~~~~~~~~~~~~~~~~~~~
+
+VertexProgram and MemoryComputeKey and VertexComputeKey
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Users that have custom `VertexProgram` implementations will need to change their implementations to support the new
+`VertexComputeKey` and `MemoryComputeKey` classes. In the `VertexPrograms` provided by TinkerPop, these changes were trivial,
+taking less than 5 minutes to make all the requisite updates.
+
+* `VertexProgram.getVertexComputeKeys()` returns a `Set<VertexComputeKey>`. No longer a `Set<String>`.
+Use `VertexComputeKey.of(String key,boolean transient)` to generate a `VertexComputeKey`.
+Transient keys were not supported in the past, so to make the implementation semantically equivalent,
+the boolean transient should be false.
+
+* `VertexProgram.getMemoryComputeKeys()` returns a `Set<MemoryComputeKey>`. No longer a `Set<String>`.
+Use `MemoryComputeKey.of(String key, BinaryOperator reducer, boolean broadcast, boolean transient)` to generate a `MemoryComputeKey`.
+Broadcasting and transients were not supported in the past so to make the implementation semantically equivalent,
+to boolean broadcast should be true and the boolean transient should be false.
+
+See link:https://issues.apache.org/jira/browse/TINKERPOP-1162[TINKERPOP-1162]
+
+SparkGraphComputer and GiraphGraphComputer Persistence
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The `MapReduce`-based steps in `TraversalVertexProgram` have been removed and replaced using the new `Memory`-reduction model.
+`MapReduce` jobs always created a persistence footprint (e.g. in HDFS). Memory data was never persisted to HDFS.
+As such, there will not be data on the disk that is accessible. For instance, no more `~reducing`, `~traversers`, and specially
+named side-effects such as those from `groupCount('m')`. The data is still accessible via `ComputerResult.memory()`,
+it just simply does not have a corresponding on-disk representation.
+
Upgrading for Providers
~~~~~~~~~~~~~~~~~~~~~~~
+GraphComputer Semantics and API
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Providers that have a custom `GraphComputer` implementation will have a lot to handle.
+Note that if the graph system simply uses `SparkGraphComputer` or `GiraphGraphComputer` provided by TinkerPop, then no updates are required.
+
+`Memory` updates:
+
+* Any `BinaryOperator` can be used for reduction and is made explicit in the `MemoryComputeKey`.
+* `MemoryComputeKeys` can be marked transient and must be removed from the resultant `ComputerResult.memory()`.
+* `MemoryComputeKeys` can be specified to not broadcast and thus, must not be available to workers to read in `VertexProgram.execute()`.
+* The `Memory` API has been changed. No more `incr()`, `and()`, etc. Now its just `set()` (setup/terminate) and `add()` (execute).
+
+See link:https://issues.apache.org/jira/browse/TINKERPOP-1166[TINKERPOP-1166]
+
+Operational semantic test cases have been added to `GraphComputerTest` to ensure that all the above behaviors are implemented correctly.
+
+Providers that have a custom `ReducingBarrierStep` implementation will need to adjust their implementation slightly to
+accommodate a new API that reflects the `Memory` updates above. This should be a simple change. Note that `FinalGet`
+no longer exists and such post-reduction processing is handled by the reducing step (via the new `Generating` interface).
+
+See link:https://issues.apache.org/jira/browse/TINKERPOP-1164[TINKERPOP-1164]
+
ScriptTraversal and Gremlin Language Variants
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^