You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@tinkerpop.apache.org by jbaker-nca <gi...@git.apache.org> on 2018/03/16 09:06:49 UTC
[GitHub] tinkerpop pull request #818: Remove null values to fix TINKERPOP-1923
GitHub user jbaker-nca opened a pull request:
https://github.com/apache/tinkerpop/pull/818
Remove null values to fix TINKERPOP-1923
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/jbaker-nca/tinkerpop TINKERPOP-1923
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/tinkerpop/pull/818.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #818
----
commit 87e5104b060c53cf7365dafbe29b2024585c2c21
Author: James Baker <ja...@...>
Date: 2018-03-15T10:23:35Z
Remove null values to fix TINKERPOP-1923
----
---
[GitHub] tinkerpop issue #818: Remove null values to fix TINKERPOP-1923
Posted by jbaker-nca <gi...@git.apache.org>.
Github user jbaker-nca commented on the issue:
https://github.com/apache/tinkerpop/pull/818
The copying code is as follows (I've simplified a little to make it clearer what's happening, as it's actually part of a bigger chunk code):
List<Edge> newEdges = new ArrayList<>();
//Copy edges from orig onto onto v
for (Edge eOrig : graph.traversal().V(orig.id()).inE().toSet()) {
if (newEdges.contains(eOrig))
continue;
LOGGER.debug("Copying IN edge {}, from {}", eOrig.id(), eOrig.outVertex().id());
Vertex vOut = eOrig.outVertex();
Edge e = vOut.addEdge(eOrig.label(), v);
ElementUtils.copyProperties(eOrig, e);
LOGGER.debug("Recording new edge {}", eOrig.id());
newEdges.add(e);
}
for (Edge eOrig : graph.traversal().V(orig.id()).outE().toSet()) {
if (newEdges.contains(eOrig))
continue;
LOGGER.debug("Copying OUT edge {}, to {}", eOrig.id(), eOrig.inVertex().id());
Vertex vIn = eOrig.inVertex();
Edge e = v.addEdge(eOrig.label(), vIn);
ElementUtils.copyProperties(eOrig, e);
LOGGER.debug("Recording new edge {}", eOrig.id());
newEdges.add(e);
}
LOGGER.debug("{} edges copied", newEdges.size());
---
[GitHub] tinkerpop pull request #818: Remove null values to fix TINKERPOP-1923
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/tinkerpop/pull/818
---
[GitHub] tinkerpop issue #818: Remove null values to fix TINKERPOP-1923
Posted by spmallette <gi...@git.apache.org>.
Github user spmallette commented on the issue:
https://github.com/apache/tinkerpop/pull/818
@rcaunt how did you do this part:
> copy the edges from the original vertex to a new one
---
[GitHub] tinkerpop issue #818: Remove null values to fix TINKERPOP-1923
Posted by spmallette <gi...@git.apache.org>.
Github user spmallette commented on the issue:
https://github.com/apache/tinkerpop/pull/818
> when removing nodes and/or edges, the collection of edges on a vertex can end up containing null values
right, i think i followed that part - i was asking more about the "why" nulls were showing up in the first place, but it sounds like you hadn't gotten that far (not a problem - i understand that you're new to the code base). I think we need to sort out the "why" before we merge a "fix" or else that fix might not be the right one.
---
[GitHub] tinkerpop issue #818: Remove null values to fix TINKERPOP-1923
Posted by spmallette <gi...@git.apache.org>.
Github user spmallette commented on the issue:
https://github.com/apache/tinkerpop/pull/818
Thanks for looking into this issue. I saw your issue in JIRA and your SO question, but just haven't had a chance to really dig into it. I can see how your solution would get rid of the `NullPointerException`, but do you understand the root cause of why this happens in the first place? Could you perhaps add your reasoning as to why this is the right fix in that context in the description of this pull request?
Also, if this is the right fix, could you get rid of the use of `stream()` in favor of TinkerPop's `IteratorUtils.filter()` - we've found `stream()` to sadly be far less performant.
Finally, could you please retarget this PR to the tp32 branch as I think this fix should be applied across all currently maintained versions? Note that we have a release coming in the next two weeks so this PR arrived just in time - thanks again.
---
[GitHub] tinkerpop issue #818: Remove null values to fix TINKERPOP-1923
Posted by spmallette <gi...@git.apache.org>.
Github user spmallette commented on the issue:
https://github.com/apache/tinkerpop/pull/818
I think I'm going to have to close this because I can't recreate the problem with the current information supplied and I don't want to merge this PR as-is because it seems like it's just making the issue go away without actually addressing the root of the issue. As there is a workaround (i.e. drop the vertex which drops the edges with it) I don't think we have a bad blocker here. If the issue arises in the future and can be recreated, I'd be happy to re-open this.
I still feel like there's something multi-threaded at the root of this somehow, but I can't think of how since you've confirmed that you don't have anything like that going on. If the issue does stem from multiple threads modifying some aspect of the TinkerGraph at the same time then this problem isn't an issue as TinkerGraph is not meant to be thread-safe for mutations.
---
[GitHub] tinkerpop issue #818: Remove null values to fix TINKERPOP-1923
Posted by jbaker-nca <gi...@git.apache.org>.
Github user jbaker-nca commented on the issue:
https://github.com/apache/tinkerpop/pull/818
Hi @spmallette, I've made the changes requested.
My understanding is that in some cases, when removing nodes and/or edges, the collection of edges on a vertex can end up containing null values (I believe they should be properly removed rather than replaced with nulls, but my knowledge of the Tinkerpop codebase is very limited so I wasn't sure why this would be happening). These null values were causing issues when looping across the collection, which was resulting in the exception thrown.
---
[GitHub] tinkerpop issue #818: Remove null values to fix TINKERPOP-1923
Posted by rcaunt <gi...@git.apache.org>.
Github user rcaunt commented on the issue:
https://github.com/apache/tinkerpop/pull/818
@spmallette I've been working with @jbaker-nca to understand this issue a bit more.
Our use case for discovering this bug was essentially merging two separate graphs into a single graph.
When two vertices are to be merged we copy the edges from the original vertex to a new one and then proceed to remove the edges on that vertex using the following:
``` java
Object[] edgeIdsArr = edgeIds.toArray();
if(edgeIdsArr.length > 0)
graph.traversal().E(edgeIdsArr).drop().iterate();
```
Once the vertices are merged we then remove the original vertices from the graph:
``` java
Object[] ids = mergeGroup.stream().map(Element::id).filter(Objects::nonNull).toArray();
if(ids != null && ids.length > 0)
graph.traversal().V(ids).drop().iterate();
```
This second traversal on vertex removal is where the error would occur yielding the following exception:
```
Exception in thread "main" java.lang.NegativeArraySizeException
at java.util.AbstractCollection.toArray(Unknown Source)
at java.util.ArrayList.addAll(Unknown Source)
at java.lang.Iterable.forEach(Unknown Source)
at org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerHelper.getEdges(TinkerHelper.java:173)
at org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerVertex.edges(TinkerVertex.java:146)
at org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerVertex.remove(TinkerVertex.java:131)
at org.apache.tinkerpop.gremlin.process.traversal.step.filter.DropStep.filter(DropStep.java:67)
at org.apache.tinkerpop.gremlin.process.traversal.step.filter.FilterStep.processNextStart(FilterStep.java:38)
at org.apache.tinkerpop.gremlin.process.traversal.step.util.AbstractStep.next(AbstractStep.java:128)
at org.apache.tinkerpop.gremlin.process.traversal.step.util.AbstractStep.next(AbstractStep.java:38)
at org.apache.tinkerpop.gremlin.process.traversal.Traversal.iterate(Traversal.java:198)
at org.apache.tinkerpop.gremlin.process.traversal.dsl.graph.GraphTraversal.iterate(GraphTraversal.java:2677)
at org.apache.tinkerpop.gremlin.process.traversal.dsl.graph.GraphTraversal$Admin.iterate(GraphTraversal.java:176)
at org.apache.tinkerpop.gremlin.process.traversal.dsl.graph.DefaultGraphTraversal.iterate(DefaultGraphTraversal.java:48)
```
I noticed the following Google Groups post - https://groups.google.com/forum/#!topic/gremlin-users/CGkuoBkNMGI - which indicates that when removing a vertex the edges should be removed at the same time. Therefore I modified our code to no longer remove the edges and only remove the vertices.
This appears to have fixed our issue and we have reverted back to Tinkerpop 3.3.1 without any further issues. However, I suspect that one should be able to remove edges and then vertices without encountering the issue.
---
[GitHub] tinkerpop issue #818: Remove null values to fix TINKERPOP-1923
Posted by robertdale <gi...@git.apache.org>.
Github user robertdale commented on the issue:
https://github.com/apache/tinkerpop/pull/818
@jbaker-nca Is there a test case that would fail without this fix so it can catch regressions?
---