You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@tinkerpop.apache.org by "Daniel Kuppitz (JIRA)" <ji...@apache.org> on 2016/12/21 15:45:58 UTC
[jira] [Commented] (TINKERPOP-1585) OLAP dedup over non elements
[ https://issues.apache.org/jira/browse/TINKERPOP-1585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15767360#comment-15767360 ]
Daniel Kuppitz commented on TINKERPOP-1585:
-------------------------------------------
To have rough estimate of how long it should take, I added 1.725.403 Strings to a Set (459.966 unique values). These are numbers from the actual customer project.
{noformat}
gremlin> clockWithResult(1) { s = [] as Set; for (i = 0; i < 1725403; i++) { s << (i%459966).toString()}; s.size() }
==>1396.075091
==>459966
{noformat}
So it doesn't take much more than 1 second to deduplicate 1.7M Strings. I think we can also ignore network limitations, since we're not talking about lots of data.
> OLAP dedup over non elements
> ----------------------------
>
> Key: TINKERPOP-1585
> URL: https://issues.apache.org/jira/browse/TINKERPOP-1585
> Project: TinkerPop
> Issue Type: Bug
> Components: hadoop, process
> Affects Versions: 3.2.3
> Reporter: Daniel Kuppitz
> Assignee: Marko A. Rodriguez
>
> OLAP {{dedup()}} is highly inefficient when it's fed with non elements.
> In a customer project a query similar tho the following returned a result in slightly more than 6 seconds:
> {noformat}
> persistedRDD.
> V().hasLabel("label1","label2").
> inE("edgeLabel1","edgeLabel2").outV().
> id().count()
> {noformat}
> The same query with {{dedup()}} added:
> {noformat}
> persistedRDD.
> V().hasLabel("label1","label2").
> inE("edgeLabel1","edgeLabel2").outV().
> id().dedup().count()
> {noformat}
> ...took more than 120 seconds.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)