You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Fabian Hueske (JIRA)" <ji...@apache.org> on 2015/02/10 16:27:11 UTC

[jira] [Commented] (FLINK-1343) Branching Join Program Deadlocks

    [ https://issues.apache.org/jira/browse/FLINK-1343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14314326#comment-14314326 ] 

Fabian Hueske commented on FLINK-1343:
--------------------------------------

After looking into the execution plan of deadlocking program and getting stacktraces of the deadlocked execution I think I found the reason for this deadlock.

The first join is executed as HybridHashJoin. Both sides are fed from the same data source. In order to prevent a deadlock the optimizer places a tempBarrier on the probe side input. However, the implementation of the join ({{MatchDriver}}) blocks until both input are available (lines 103/104: {{this.taskContext.getInput()}}) before it starts building the hashtable. This behavior causes the deadlock because the build side is not fully consumed (as assumed by the optimizer). Instead, the HashJoin waits for the probe side to be available from the tempBarrier and gets stuck because the build input is not consumed.

Forcing the first join to be executed as MergeJoin fixes the deadlock.

> Branching Join Program Deadlocks
> --------------------------------
>
>                 Key: FLINK-1343
>                 URL: https://issues.apache.org/jira/browse/FLINK-1343
>             Project: Flink
>          Issue Type: Bug
>          Components: Optimizer
>    Affects Versions: 0.8, 0.9
>            Reporter: Fabian Hueske
>            Assignee: Fabian Hueske
>
> The following program which gets its data from a single non-parallel data source, branches two times, and joins the branches with two joins, deadlocks.
> {code:java}
> public class DeadlockProgram {
>     public static void main(String[] args) throws Exception {
>         final ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();
>         DataSet<Long> longs = env.generateSequence(0,1000000l).setParallelism(1);
>         DataSet<Long> longs2 = env.generateSequence(0, 1000000l).setParallelism(1);
>         DataSet<Tuple1<Long>> longT1 = longs.map(new TupleWrapper());
>         DataSet<Tuple1<Long>> longT2 = longT1.project(0);
>         DataSet<Tuple1<Long>> longT3 = longs.map(new TupleWrapper()); // deadlocks
> //        DataSet<Tuple1<Long>> longT3 = longs2.map(new TupleWrapper()); // works
>         longT2.join(longT3).where(0).equalTo(0).projectFirst(0)
>             .join(longT1).where(0).equalTo(0).projectFirst(0)
>             .print();
>         env.execute();
>     }
>     public static class TupleWrapper implements MapFunction<Long, Tuple1<Long>> {
>         @Override
>         public Tuple1<Long> map(Long l) throws Exception {
>             return new Tuple1<Long>(l);
>         }
>     };
> }
> {code}
> If one of the branches reads its data from a second data source (see inline comment) or if the single data source uses the default parallelism, the program executes correctly.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)