You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@tinkerpop.apache.org by Marko Rodriguez <ok...@gmail.com> on 2017/01/13 18:23:15 UTC
XXXGraphComputer should evolve to XXXGraphActors.
Hi,
CURRENT: The GraphComputer framework assumes “vertex-centric” computing. That is, a vertex receives a message and does something with it. Moreover, it can send messages to other vertices.
We got this wrong and I think we should do it right with GraphActors.
FUTURE: The GraphActors framework assumes “partition-centric” computing. That is, a partition receives a message and does something with it. Moreover, it can send messages to other partitions.
——
VertexProgram.execute(final Vertex vertex, Iterator<M> messages)
should have been:
PartitionProgram.execute(Partition partition, Iterator<M> message)
in fact, ActorProgram’s execute() method is defined as:
ActorProgram.execute(M message).
1. Every Actor owns a Partition and thus, you don’t need to pass in the Partition.
2. To support ASP (asynchrounous) and BSP (synchronous) computing, you don’t provide an Iterator<M>, just an M as they come through (event-driven).
3. All partitions are assumed to have random access capabilities. All the data in the partition is randomly accessible.
4. A partition is a generalization of GraphComputer’s Vertex, where at the micro-limit, every Vertex is in its own Partition. This is how we think about SparkGraphComputer, GiraphGraphComputer, etc. — the “star graph." However, by generalizing to larger subgraphs than just Vertex, we can have more work being done per iteration in SparkGraphComputer, etc. Moreover, by generalizing to partition, we don’t have to have all edges of a vertex co-located and thus, can support edge-cut systems (liked DSEGraph).
So, what does this mean for the future? This injection from “vertex-centric” to “partition-centric” allows us to easily create SparkGraphActors. Next, how do you verify if a traversal will be able to legally execute against the underlying GraphActors system? It depends on “the rules” of the Partitioner. A Partitioner should have Features which define the boundaries of its data sphere. By looking at those Features and looking at the semantics of the Traversal, it is possible to ensure that the Traversal will work against the Features. If not, ActorVerificationException. If so, execute it.
In conclusion — I’m starting to see GraphComputer as our OLAP 1.0 and GraphActors as our OLAP/OLTP 2.0. I put in there OLTP because with systems like Akka that don’t require big bulk data migrations, you can execute against the Graph connection object…. Even with SparkGraphActors, you could just have workers that work against Graph connection objects (the only RDD data is messages!!!). Thus, with GraphActors, we start to smear the concept of OLAP and OLTP.
Anywho — I think if we get GraphActors right, we will solve many of the shortcomings of GraphComputer while, at the same time, providing a powerful distributed graph computing framework.
Take care,
Marko.
http://markorodriguez.com
Re: [TinkerPop] XXXGraphComputer should evolve to XXXGraphActors.
Posted by Luca Garulli <l....@gmail.com>.
On 19 January 2017 at 03:24, Marko Rodriguez <ok...@gmail.com> wrote:
> Hi,
>
> OrientDB can do distributed transactions across nodes. What do you mean
> for connections?
>
> So, I’m not so skilled when it comes to transactions and the like so bear
> with me…
>
> One of the models I want to push is that each worker in GraphActors will
> have a Graph “connection.” That is:
>
> configuration.setProperty(“host”,worker.address())
> Graph graph = GraphFactory.open(configuration); // Graph = “connection"
>
>
> This way, the worker is always talking directly to the node it is
> physically executing at. Moreover, it means that its Partition is the data
> contained at the node in the cluster. Everything is processed locally with
> GraphActors.
>
> Now, lets say we do a muting traversal such as:
>
> g.V().as(‘a’).out(‘knows’).as(‘b’).
> addE(‘likes’).from(‘a’).to(‘b’)
>
>
>
> So, this will have it such that each Graph “connection” will have writes
> to it. Now lets say we want to “globally commit” such that each Graph
> “connection” commits its transaction but if any particular one fails, they
> all fail …. or something like that. That is, how can we (OrientDB and/or
> TinkerPop) enable transaction guarantees across multiple Graph
> "connections"?
>
In OrientDB the distributed transaction starts from any node (multi-master)
that acts as coordinator for the transaction. So the client has to execute
all the operation inside the same connection to the coordinator server. At
that point the coordinator divide the operation based on
sharding/replication and send them to all the involved nodes.
Once the write quorum is satisfied (by default is "majority" of the
servers), then the coordinator sends a final commit (2 phase commit) to the
servers. If any server doesn't agree with the quorum, it's forced to have
the same state.
If the coordinator crashes before to send the final ok, well each server
has a timeout per transaction and after a while they rollback
autonoumously.
So if all the mutations are sent through the same connection, then OrientDB
can execute all of them as an ACID distributed transaction.
>
> Thanks,
> Marko.
>
> http://markorodriguez.com
>
>
>
>
>>
>> Thanks Luca,
>> Marko.
>>
>> http://markorodriguez.com
>>
>
> Luca
>
> --
> You received this message because you are subscribed to the Google Groups
> "Gremlin-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to gremlin-users+unsubscribe@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/ms
> gid/gremlin-users/CAGud208pLzs9-OT%2BoiuTCKGimXFkaLoyKoAPrro
> mVLD1ny72kg%40mail.gmail.com
> <https://groups.google.com/d/msgid/gremlin-users/CAGud208pLzs9-OT%2BoiuTCKGimXFkaLoyKoAPrromVLD1ny72kg%40mail.gmail.com?utm_medium=email&utm_source=footer>
> .
> For more options, visit https://groups.google.com/d/optout.
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Gremlin-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to gremlin-users+unsubscribe@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/ms
> gid/gremlin-users/18D3D712-6AC9-471D-AAE6-8FFDB4BE4A39%40gmail.com
> <https://groups.google.com/d/msgid/gremlin-users/18D3D712-6AC9-471D-AAE6-8FFDB4BE4A39%40gmail.com?utm_medium=email&utm_source=footer>
> .
>
> For more options, visit https://groups.google.com/d/optout.
>
Re: [TinkerPop] XXXGraphComputer should evolve to XXXGraphActors.
Posted by Marko Rodriguez <ok...@gmail.com>.
Hi,
> OrientDB can do distributed transactions across nodes. What do you mean for connections?
So, I’m not so skilled when it comes to transactions and the like so bear with me…
One of the models I want to push is that each worker in GraphActors will have a Graph “connection.” That is:
configuration.setProperty(“host”,worker.address())
Graph graph = GraphFactory.open(configuration); // Graph = “connection"
This way, the worker is always talking directly to the node it is physically executing at. Moreover, it means that its Partition is the data contained at the node in the cluster. Everything is processed locally with GraphActors.
Now, lets say we do a muting traversal such as:
g.V().as(‘a’).out(‘knows’).as(‘b’).
addE(‘likes’).from(‘a’).to(‘b’)
So, this will have it such that each Graph “connection” will have writes to it. Now lets say we want to “globally commit” such that each Graph “connection” commits its transaction but if any particular one fails, they all fail …. or something like that. That is, how can we (OrientDB and/or TinkerPop) enable transaction guarantees across multiple Graph "connections"?
Thanks,
Marko.
http://markorodriguez <http://markorodriguez/>.com
>
>
> Thanks Luca,
> Marko.
>
> http://markorodriguez <http://markorodriguez/>.com
>
> Luca
>
> --
> You received this message because you are subscribed to the Google Groups "Gremlin-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to gremlin-users+unsubscribe@googlegroups.com <ma...@googlegroups.com>.
> To view this discussion on the web visit https://groups.google.com/d/msgid/gremlin-users/CAGud208pLzs9-OT%2BoiuTCKGimXFkaLoyKoAPrromVLD1ny72kg%40mail.gmail.com <https://groups.google.com/d/msgid/gremlin-users/CAGud208pLzs9-OT%2BoiuTCKGimXFkaLoyKoAPrromVLD1ny72kg%40mail.gmail.com?utm_medium=email&utm_source=footer>.
> For more options, visit https://groups.google.com/d/optout <https://groups.google.com/d/optout>.
Re: [TinkerPop] XXXGraphComputer should evolve to XXXGraphActors.
Posted by Luca Garulli <l....@gmail.com>.
On 18 January 2017 at 03:35, Marko Rodriguez <ok...@gmail.com> wrote:
>
> *Question*: Can OrientDB do a “distributed transaction”? That is, if you
> have X number of connections to OrientDB, can you say that you want all
> connections to be tied to a single TX?
>
OrientDB can do distributed transactions across nodes. What do you mean for
connections?
>
> Thanks Luca,
> Marko.
>
> http://markorodriguez.com
>
Luca
Re: [TinkerPop] XXXGraphComputer should evolve to XXXGraphActors.
Posted by Marko Rodriguez <ok...@gmail.com>.
Hello,
> I definitely like this and it would be closer to what we are doing with OrientDB distributed architecture. How quick do you think you could create a draft of this in TP 3.x?
We already have a working version with the interfaces in gremlin-core/ and an akka-gremlin/ implementation.
interfaces: https://github.com/apache/tinkerpop/tree/TINKERPOP-1564/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/actors <https://github.com/apache/tinkerpop/tree/TINKERPOP-1564/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/actors>
implementation: https://github.com/apache/tinkerpop/tree/TINKERPOP-1564/akka-gremlin/src/main/java/org/apache/tinkerpop/gremlin/akka/process/actors <https://github.com/apache/tinkerpop/tree/TINKERPOP-1564/akka-gremlin/src/main/java/org/apache/tinkerpop/gremlin/akka/process/actors>
You can see TraversalActorsProgram which is able to execute a Traversal over the Actors framework:
https://github.com/apache/tinkerpop/tree/TINKERPOP-1564/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/actors/traversal <https://github.com/apache/tinkerpop/tree/TINKERPOP-1564/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/actors/traversal>
- a few hardcoded constants in there right now as I’m still developing/playing.
And akka-gremlin/ passes the traversal test suite (save for a few issues I’m still sorting out):
https://github.com/apache/tinkerpop/blob/TINKERPOP-1564/akka-gremlin/src/test/java/org/apache/tinkerpop/gremlin/akka/process/actors/AkkaActorsProcessActorsTest.java <https://github.com/apache/tinkerpop/blob/TINKERPOP-1564/akka-gremlin/src/test/java/org/apache/tinkerpop/gremlin/akka/process/actors/AkkaActorsProcessActorsTest.java>
Hope that is clear. Please provide any feedback as you have it.
Question: Can OrientDB do a “distributed transaction”? That is, if you have X number of connections to OrientDB, can you say that you want all connections to be tied to a single TX?
Thanks Luca,
Marko.
http://markorodriguez <http://markorodriguez/>.com
>
>
> On 13 January 2017 at 12:23, Marko Rodriguez <okrammarko@gmail.com <ma...@gmail.com>> wrote:
> Hi,
>
> CURRENT: The GraphComputer framework assumes “vertex-centric” computing. That is, a vertex receives a message and does something with it. Moreover, it can send messages to other vertices.
>
> We got this wrong and I think we should do it right with GraphActors.
>
> FUTURE: The GraphActors framework assumes “partition-centric” computing. That is, a partition receives a message and does something with it. Moreover, it can send messages to other partitions.
>
> ——
>
> VertexProgram.execute(final Vertex vertex, Iterator<M> messages)
>
> should have been:
>
> PartitionProgram.execute(Partition partition, Iterator<M> message)
>
> in fact, ActorProgram’s execute() method is defined as:
>
> ActorProgram.execute(M message).
>
> 1. Every Actor owns a Partition and thus, you don’t need to pass in the Partition.
> 2. To support ASP (asynchrounous) and BSP (synchronous) computing, you don’t provide an Iterator<M>, just an M as they come through (event-driven).
> 3. All partitions are assumed to have random access capabilities. All the data in the partition is randomly accessible.
> 4. A partition is a generalization of GraphComputer’s Vertex, where at the micro-limit, every Vertex is in its own Partition. This is how we think about SparkGraphComputer, GiraphGraphComputer, etc. — the “star graph." However, by generalizing to larger subgraphs than just Vertex, we can have more work being done per iteration in SparkGraphComputer, etc. Moreover, by generalizing to partition, we don’t have to have all edges of a vertex co-located and thus, can support edge-cut systems (liked DSEGraph).
>
> So, what does this mean for the future? This injection from “vertex-centric” to “partition-centric” allows us to easily create SparkGraphActors. Next, how do you verify if a traversal will be able to legally execute against the underlying GraphActors system? It depends on “the rules” of the Partitioner. A Partitioner should have Features which define the boundaries of its data sphere. By looking at those Features and looking at the semantics of the Traversal, it is possible to ensure that the Traversal will work against the Features. If not, ActorVerificationException. If so, execute it.
>
> In conclusion — I’m starting to see GraphComputer as our OLAP 1.0 and GraphActors as our OLAP/OLTP 2.0. I put in there OLTP because with systems like Akka that don’t require big bulk data migrations, you can execute against the Graph connection object…. Even with SparkGraphActors, you could just have workers that work against Graph connection objects (the only RDD data is messages!!!). Thus, with GraphActors, we start to smear the concept of OLAP and OLTP.
>
> Anywho — I think if we get GraphActors right, we will solve many of the shortcomings of GraphComputer while, at the same time, providing a powerful distributed graph computing framework.
>
> Take care,
> Marko.
>
> http://markorodriguez.com <http://markorodriguez.com/>
>
>
>
>
> --
> You received this message because you are subscribed to the Google Groups "Gremlin-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to gremlin-users+unsubscribe@googlegroups.com <ma...@googlegroups.com>.
> To view this discussion on the web visit https://groups.google.com/d/msgid/gremlin-users/49B70143-DB77-429D-956F-869B20A554C4%40gmail.com <https://groups.google.com/d/msgid/gremlin-users/49B70143-DB77-429D-956F-869B20A554C4%40gmail.com?utm_medium=email&utm_source=footer>.
> For more options, visit https://groups.google.com/d/optout <https://groups.google.com/d/optout>.
>
>
> --
> You received this message because you are subscribed to the Google Groups "Gremlin-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to gremlin-users+unsubscribe@googlegroups.com <ma...@googlegroups.com>.
> To view this discussion on the web visit https://groups.google.com/d/msgid/gremlin-users/CAGud20-wdi48JUeUGaurcvGSPReRu3J6gcKxV6zCVn1JFh-YDg%40mail.gmail.com <https://groups.google.com/d/msgid/gremlin-users/CAGud20-wdi48JUeUGaurcvGSPReRu3J6gcKxV6zCVn1JFh-YDg%40mail.gmail.com?utm_medium=email&utm_source=footer>.
> For more options, visit https://groups.google.com/d/optout <https://groups.google.com/d/optout>.
Re: [TinkerPop] XXXGraphComputer should evolve to XXXGraphActors.
Posted by Luca Garulli <l....@gmail.com>.
Hi Marko,
I definitely like this and it would be closer to what we are doing with
OrientDB distributed architecture. How quick do you think you could create
a draft of this in TP 3.x?
On 13 January 2017 at 12:23, Marko Rodriguez <ok...@gmail.com> wrote:
> Hi,
>
> *CURRENT*: The GraphComputer framework assumes “vertex-centric”
> computing. That is, a vertex receives a message and does something with it.
> Moreover, it can send messages to other vertices.
>
> We got this wrong and I think we should do it right with GraphActors.
>
> *FUTURE*: The GraphActors framework assumes “partition-centric”
> computing. That is, a partition receives a message and does something with
> it. Moreover, it can send messages to other partitions.
>
> ——
>
> VertexProgram.execute(final Vertex vertex, Iterator<M> messages)
>
>
> should have been:
>
> PartitionProgram.execute(Partition partition, Iterator<M> message)
>
>
> in fact, ActorProgram’s execute() method is defined as:
>
> ActorProgram.execute(M message).
>
>
> 1. Every Actor owns a Partition and thus, you don’t need to pass in the
> Partition.
> 2. To support ASP (asynchrounous) and BSP (synchronous) computing, you
> don’t provide an Iterator<M>, just an M as they come through (event-driven).
> 3. All partitions are assumed to have random access capabilities. All the
> data in the partition is randomly accessible.
> 4. A partition is a generalization of GraphComputer’s Vertex, where at the
> micro-limit, every Vertex is in its own Partition. This is how we think
> about SparkGraphComputer, GiraphGraphComputer, etc. — the “star graph."
> However, by generalizing to larger subgraphs than just Vertex, we can have
> more work being done per iteration in SparkGraphComputer, etc. Moreover, by
> generalizing to partition, we don’t have to have all edges of a vertex
> co-located and thus, can support edge-cut systems (liked DSEGraph).
>
> So, what does this mean for the future? This injection from
> “vertex-centric” to “partition-centric” allows us to easily create
> SparkGraphActors. Next, how do you verify if a traversal will be able to
> legally execute against the underlying GraphActors system? It depends on
> “the rules” of the Partitioner. A Partitioner should have Features which
> define the boundaries of its data sphere. By looking at those Features and
> looking at the semantics of the Traversal, it is possible to ensure that
> the Traversal will work against the Features. If not,
> ActorVerificationException. If so, execute it.
>
> In conclusion — I’m starting to see GraphComputer as our OLAP 1.0 and
> GraphActors as our OLAP/OLTP 2.0. I put in there OLTP because with systems
> like Akka that don’t require big bulk data migrations, you can execute
> against the Graph connection object…. Even with SparkGraphActors, you could
> just have workers that work against Graph connection objects (the only RDD
> data is messages!!!). Thus, with GraphActors, we start to smear the concept
> of OLAP and OLTP.
>
> Anywho — I think if we get GraphActors right, we will solve many of the
> shortcomings of GraphComputer while, at the same time, providing a powerful
> distributed graph computing framework.
>
> Take care,
> Marko.
>
> http://markorodriguez.com
>
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Gremlin-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to gremlin-users+unsubscribe@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/
> msgid/gremlin-users/49B70143-DB77-429D-956F-869B20A554C4%40gmail.com
> <https://groups.google.com/d/msgid/gremlin-users/49B70143-DB77-429D-956F-869B20A554C4%40gmail.com?utm_medium=email&utm_source=footer>
> .
> For more options, visit https://groups.google.com/d/optout.
>