[jira] [Commented] (FLINK-1951) NullPointerException in DeltaIteration when no ForwardedFileds

["Fabian Hueske (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-1951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14521507#comment-14521507 ] 

Fabian Hueske commented on FLINK-1951:
--------------------------------------

It was a bit tricky and took some time but I think I finally found the bug :-)

The problem is the following. The failing join is the join with the solution set. The result of this join is the solution set data that is reinserted into the hashtable that was previously probed. 

If the annotation is set, the optimizer knows that the result of the join is still partitioned and can do the insert locally by chaining the calls. This means that probing and reinserting of the same HT happens in the same thread, i.e., there is no concurrent access.

If the annotation is NOT set, the optimizer does not know that the result is still partitioned. Hence, it inserts a network shuffling strategy to partition the data. The receiving task then reinserts the solution set delta records into the hash table. Since probing and inserting are happening in two separate tasks which are separated by a network connection, they happen concurrently in two different threads and might interfere.

Guarding reinsertion and probing with a simple lock solves the problem. Not sure if that's the best fix for the problem though...



> NullPointerException in DeltaIteration when no ForwardedFileds
> --------------------------------------------------------------
>
>                 Key: FLINK-1951
>                 URL: https://issues.apache.org/jira/browse/FLINK-1951
>             Project: Flink
>          Issue Type: Bug
>          Components: Iterations
>    Affects Versions: 0.9
>            Reporter: Vasia Kalavri
>            Assignee: Fabian Hueske
>            Priority: Critical
>
> The following exception is thrown by the Connected Components example, if the @ForwardedFieldsFirst("*") annotation from the ComponentIdFilter join is removed:
> Caused by: java.lang.NullPointerException
> 	at org.apache.flink.examples.java.graph.ConnectedComponents$ComponentIdFilter.join(ConnectedComponents.java:186)
> 	at org.apache.flink.examples.java.graph.ConnectedComponents$ComponentIdFilter.join(ConnectedComponents.java:1)
> 	at org.apache.flink.runtime.operators.JoinWithSolutionSetSecondDriver.run(JoinWithSolutionSetSecondDriver.java:198)
> 	at org.apache.flink.runtime.operators.RegularPactTask.run(RegularPactTask.java:496)
> 	at org.apache.flink.runtime.iterative.task.AbstractIterativePactTask.run(AbstractIterativePactTask.java:139)
> 	at org.apache.flink.runtime.iterative.task.IterationIntermediatePactTask.run(IterationIntermediatePactTask.java:92)
> 	at org.apache.flink.runtime.operators.RegularPactTask.invoke(RegularPactTask.java:362)
> 	at org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:217)
> 	at java.lang.Thread.run(Thread.java:745)
> [Code | https://github.com/vasia/flink/tree/cc-test] and [dataset | http://snap.stanford.edu/data/com-DBLP.html] to reproduce.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)