[jira] [Commented] (FLINK-2318) BroadcastVariable of unioned data set fails

["Sachin Goel (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-2318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14699101#comment-14699101 ] 

Sachin Goel commented on FLINK-2318:
------------------------------------

I was trying to debug this and found out that {{NUM_BROADCAST_INPUTS}} is incremented twice in {{TaskConfig.java}}. This shouldn't be the case.
However, making sure this field is incremented only once for one broadcast input leads to loss of data. Not all elements get through.
Changing
{code}
this.config.setInteger(NUM_BROADCAST_INPUTS, this.config.getInteger(NUM_BROADCAST_INPUTS, 0) + 1);
{code}
to
{code}
if(this.config.getInteger(grp, 0) == 1) {
			this.config.setInteger(NUM_BROADCAST_INPUTS, this.config.getInteger(NUM_BROADCAST_INPUTS, 0) + 1);
		}
{code}
and testing on:
{code}
DataSet<Long> d1 = env.generateSequence(1, 10);
		DataSet<Long> d2 = env.generateSequence(10, 20);
		DataSet<Long> d3 = env.generateSequence(20, 30);
		d1.map(new RichMapFunction<Long, Long>() {
			@Override
			public void open(Configuration parameters) throws Exception{
				List unioned = getRuntimeContext().getBroadcastVariable("UNION_BC");
				List normal = getRuntimeContext().getBroadcastVariable("NORMAL_BC");
				String output = "";
				for(int i = 0; i < unioned.size(); i++){
					output += (unioned.get(i) + "\n");
				}
				output += "-----\n";
				for(int i = 0; i < normal.size(); i++){
					output += (normal.get(i) + "\n");
				}
				System.out.println(output);
			}
			@Override
			public Long map(Long value) throws Exception {
				return value;
			}}
		)
		.withBroadcastSet(d1.union(d2), "UNION_BC")
		.withBroadcastSet(d3, "NORMAL_BC")
		.output(new DiscardingOutputFormat<Long>());
{code}
this is the output from every task:
{code}
18
5
-----
20
21
29
26
27
24
25
30
28
22
23
{code}
Both the input gates are definitely set, however, all elements are not being read.

[The non-unioned broadcast variable is to ensure that everything else works fine.]

> BroadcastVariable of unioned data set fails
> -------------------------------------------
>
>                 Key: FLINK-2318
>                 URL: https://issues.apache.org/jira/browse/FLINK-2318
>             Project: Flink
>          Issue Type: Bug
>          Components: Distributed Runtime, Optimizer
>    Affects Versions: 0.9
>            Reporter: Fabian Hueske
>
> Using a unioned data set as broadcast variable such as this:
> {code}
> DataSet d1 = [...]
> DataSet d2 = [...]
> DataSet d3 = [...]
> d1
>   .map(new MyMapper())
>   .withBroadcastSet(d2.union(d3), "myBroadcast");
> {code}
> throws an exception at runtime:
> {code}
> java.lang.Exception: Call to registerInputOutput() of invokable failed
> 	at org.apache.flink.runtime.taskmanager.Task.run(Task.java:504)
> 	at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.RuntimeException: Initializing the input streams failed in Task MapPartition (MapPartition at translatHashJoinAsMap(FlinkFlowStep.java:755)): Illegal input group size in task configuration: -1
> 	at org.apache.flink.runtime.operators.RegularPactTask.registerInputOutput(RegularPactTask.java:246)
> 	at org.apache.flink.runtime.taskmanager.Task.run(Task.java:501)
> 	... 1 more
> Caused by: java.lang.Exception: Illegal input group size in task configuration: -1
> 	at org.apache.flink.runtime.operators.RegularPactTask.initBroadcastInputReaders(RegularPactTask.java:783)
> 	at org.apache.flink.runtime.operators.RegularPactTask.registerInputOutput(RegularPactTask.java:243)
> 	... 2 more
> {code}
> A simple workaround is to apply an identity mapper on the unioned data set.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)