You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Soheil Pourbafrani <so...@gmail.com> on 2018/07/17 07:36:11 UTC
Can not get OutPutTag datastream from Windowing function
Hi, according to the documents I tried to get late data using side output.
final OutputTag<Tuple3<String, Long, JSONObject>> lateOutputTag = new
OutputTag<Tuple3<String, Long, JSONObject>>("late-data"){};
DataStream<Tuple3<String,Long, JSONObject>> res = aggregatedTuple
.assignTimestampsAndWatermarks(new Bound())
}).keyBy(1).timeWindow(Time.milliseconds(160))/*.countWindow(3)*/
.allowedLateness(Time.milliseconds(2))
.sideOutputLateData(lateOutputTag)
.reduce(Do some process);
When trying to store late data in a Datastream (As shown in document):
DataStream<Tuple3<String, Long, JSONObject>> lateData = res.
there is no predefined getSideOutput method on DataStream res!
But if I call getSideOutput just after reduce function, it is known! But I
don't want to save late data on res variable and I want to save them on
another variable!
DataStream<Tuple3<String,Long, JSONObject>> res = aggregatedTuple
.assignTimestampsAndWatermarks(new Bound())
}).keyBy(1).timeWindow(Time.milliseconds(160))/*.countWindow(3)*/
.allowedLateness(Time.milliseconds(2))
.sideOutputLateData(lateOutputTag)
.reduce(Do some process)
.getSideoutput(lateOutputTag);
What is the problem here?
Re: Can not get OutPutTag datastream from Windowing function
Posted by Dawid Wysakowicz <dw...@apache.org>.
Hi Soheil,
The /getSideOutput/ method is part of /SingleOutputStreamOperator/ which
extends /DataStream///. Try using /SingleOutputStreamOperator/ as the
type for your res variable.
Best,
Dawid
On 17/07/18 09:36, Soheil Pourbafrani wrote:
> Hi, according to theĀ documents I tried to get late data using side
> output.
>
> final OutputTag<Tuple3<String, Long, JSONObject>> lateOutputTag = new OutputTag<Tuple3<String, Long, JSONObject>>("late-data"){};
> DataStream<Tuple3<String,Long, JSONObject>> res = aggregatedTuple
> .assignTimestampsAndWatermarks(new Bound())
> }).keyBy(1).timeWindow(Time.milliseconds(160))/*.countWindow(3)*/ .allowedLateness(Time.milliseconds(2))
> .sideOutputLateData(lateOutputTag)
> .reduce(Do some process);
>
> When trying to store late data in a Datastream (As shown in document):
> DataStream<Tuple3<String, Long, JSONObject>> lateData = res.
> there is no predefined getSideOutput method on DataStream res!
> But if I callĀ getSideOutput just after reduce function, it is known!
> But I don't want to save late data on res variable and I want to save
> them on another variable!
> DataStream<Tuple3<String,Long, JSONObject>> res = aggregatedTuple
> .assignTimestampsAndWatermarks(new Bound())
> }).keyBy(1).timeWindow(Time.milliseconds(160))/*.countWindow(3)*/ .allowedLateness(Time.milliseconds(2))
> .sideOutputLateData(lateOutputTag)
> .reduce(Do some process)
> .getSideoutput(lateOutputTag);
> What is the problem here?
>
>
Re: Can not get OutPutTag datastream from Windowing function
Posted by Xingcan Cui <xi...@gmail.com>.
Hi Soheil,
The `getSideOutput()` method is defined on the operator instead of the datastream.
You can invoke it after any action (e.g., map, window) performed on a datastream.
Best,
Xingcan
> On Jul 17, 2018, at 3:36 PM, Soheil Pourbafrani <so...@gmail.com> wrote:
>
> Hi, according to the documents I tried to get late data using side output.
>
> final OutputTag<Tuple3<String, Long, JSONObject>> lateOutputTag = new OutputTag<Tuple3<String, Long, JSONObject>>("late-data"){};
> DataStream<Tuple3<String,Long, JSONObject>> res = aggregatedTuple
> .assignTimestampsAndWatermarks(new Bound())
> }).keyBy(1).timeWindow(Time.milliseconds(160))/*.countWindow(3)*/
> .allowedLateness(Time.milliseconds(2))
> .sideOutputLateData(lateOutputTag)
> .reduce(Do some process);
>
> When trying to store late data in a Datastream (As shown in document):
> DataStream<Tuple3<String, Long, JSONObject>> lateData = res.
> there is no predefined getSideOutput method on DataStream res!
> But if I call getSideOutput just after reduce function, it is known! But I don't want to save late data on res variable and I want to save them on another variable!
> DataStream<Tuple3<String,Long, JSONObject>> res = aggregatedTuple
> .assignTimestampsAndWatermarks(new Bound())
> }).keyBy(1).timeWindow(Time.milliseconds(160))/*.countWindow(3)*/
> .allowedLateness(Time.milliseconds(2))
> .sideOutputLateData(lateOutputTag)
> .reduce(Do some process)
> .getSideoutput(lateOutputTag);
> What is the problem here?
>
>