You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Fabian Hueske (JIRA)" <ji...@apache.org> on 2015/07/15 00:47:04 UTC
[jira] [Commented] (FLINK-2361) flatMap + distinct gives erroneous
results for big data sets
[ https://issues.apache.org/jira/browse/FLINK-2361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14627211#comment-14627211 ]
Fabian Hueske commented on FLINK-2361:
--------------------------------------
Can you give exact instructions how to reproduce the problem? Paste the program code (or link to a branch/repository) and give a link to the input data that produces the error with the given program?
Thanks, Fabian
Btw. distinct is internally implemented as a GroupReduceFunction that only returns the first element of the group iterator.
> flatMap + distinct gives erroneous results for big data sets
> ------------------------------------------------------------
>
> Key: FLINK-2361
> URL: https://issues.apache.org/jira/browse/FLINK-2361
> Project: Flink
> Issue Type: Bug
> Components: Gelly
> Affects Versions: 0.10
> Reporter: Andra Lungu
>
> When running the simple Connected Components algorithm (currently in Gelly) on the twitter follower graph, with 1, 100 or 10000 iterations, I get the following error:
> Caused by: java.lang.Exception: Target vertex '657282846' does not exist!.
> at org.apache.flink.graph.spargel.VertexCentricIteration$VertexUpdateUdfSimpleVV.coGroup(VertexCentricIteration.java:300)
> at org.apache.flink.runtime.operators.CoGroupWithSolutionSetSecondDriver.run(CoGroupWithSolutionSetSecondDriver.java:220)
> at org.apache.flink.runtime.operators.RegularPactTask.run(RegularPactTask.java:496)
> at org.apache.flink.runtime.iterative.task.AbstractIterativePactTask.run(AbstractIterativePactTask.java:139)
> at org.apache.flink.runtime.iterative.task.IterationTailPactTask.run(IterationTailPactTask.java:107)
> at org.apache.flink.runtime.operators.RegularPactTask.invoke(RegularPactTask.java:362)
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:559)
> at java.lang.Thread.run(Thread.java:722)
> Now this is very bizzare as the DataSet of vertices is produced from the DataSet of edges... Which means there cannot be a an edge with an invalid target id... The method calls flatMap to isolate the src and trg ids and distinct to ensure their uniqueness.
> The algorithm works fine for smaller data sets...
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)