You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Puneet Kinra <pu...@customercentria.com> on 2019/05/24 08:32:58 UTC
Re: How can i just implement a crontab function using flink?
There is concept of periodic watermarker , you can use that
if you are working on eventtime.
On Fri, May 24, 2019 at 1:51 PM wanglei2@geekplus.com.cn <
wanglei2@geekplus.com.cn> wrote:
>
> I want to do something every one minute.
>
> Using TumblingWindow, the function will not be triigged if there's no
> message received during this minute. But i still need to execute the
> funtion.
>
> How can i implement it ?
>
> ------------------------------
> wanglei2@geekplus.com.cn
>
>
>
>
--
*Cheers *
*Puneet Kinra*
*Mobile:+918800167808 | Skype : puneet.kinra@customercentria.com
<pu...@customercentria.com>*
*e-mail :puneet.kinra@customercentria.com
<pu...@customercentria.com>*
Re: Re: How can i just implement a crontab function using flink?
Posted by "wanglei2@geekplus.com.cn" <wa...@geekplus.com.cn>.
I tried。 But the MyProcessWindowFunction still not tigged when there's no event in the window
Any insight on this?
source.assignTimestampsAndWatermarks(new AssignerWithPeriodicWatermarks<Map>() {
@Override
public Watermark getCurrentWatermark() {
return new Watermark(System.currentTimeMillis() - 10000);
}
@Override
public long extractTimestamp(Map map, long l) {
return System.currentTimeMillis();
}
}).windowAll(TumblingEventTimeWindows.of(Time.seconds(10))).process(new MyProcessWindowFunction());
wanglei2@geekplus.com.cn
From: Puneet Kinra
Date: 2019-05-24 17:02
To: wanglei2@geekplus.com.cn
CC: user
Subject: Re: How can i just implement a crontab function using flink?
There is concept of periodic watermarker , you can use that
if you are working on eventtime.
On Fri, May 24, 2019 at 1:51 PM wanglei2@geekplus.com.cn <wa...@geekplus.com.cn> wrote:
I want to do something every one minute.
Using TumblingWindow, the function will not be triigged if there's no message received during this minute. But i still need to execute the funtion.
How can i implement it ?
wanglei2@geekplus.com.cn
--
Cheers
Puneet Kinra
Mobile:+918800167808 | Skype : puneet.kinra@customercentria.com
e-mail :puneet.kinra@customercentria.com
Re: Unable to restore state value after job failed using
RocksDBStateBackend
Posted by Simon Su <ba...@163.com>.
Hi Lei Wang
Actually it will not work, job recovery from checkpoint by using jobid to detect the snapshot directory, but when restart it in intellj and didn’t set any configurations, jobid will regenerate and it will regard as a new job, so you get the null state every time, you can follow by https://ci.apache.org/projects/flink/flink-docs-stable/ops/state/checkpoints.html
Thanks,
Simon
On 06/25/2019 21:43，wanglei2@geekplus.com.cn<wa...@geekplus.com.cn> wrote：
I start and cancel it just in my intellij idea development environment.
First click the run button, then click the red stop button, and then click the run button again.
Let me google about the savepoint.
Thanks,
Lei Wang
wanglei2@geekplus.com.cn
From: Stephan Ewen
Date: 2019-06-25 20:36
To: user
Subject: Re: Unable to restore state value after job failed using RocksDBStateBackend
If you manually cancel and restart the job, state is only carried forward if you use a savepoint.
Can you check if that is what you are doing?
On Tue, Jun 25, 2019 at 2:21 PM Simon Su <ba...@163.com> wrote:
Hi wanglei
Can you post how you restart the job ?
Thanks,
Simon
On 06/25/2019 20:11，wanglei2@geekplus.com.cn<wa...@geekplus.com.cn> wrote：
public class StateProcessTest extends KeyedProcessFunction<String, Tuple2<String, Long>, String> {
private transient ValueState<Tuple2<Long,Long>> state;
public void processElement(Tuple2<String, Long> value, Context ctx, Collector<String> out) throws Exception {
Tuple2<Long, Long> stateValue = state.value();
if(stateValue == null){
log.info("########## initialize");
stateValue = new Tuple2(34l,56l);
}
state.update(stateValue);
}
@Override
public void open(Configuration parameters) throws Exception {
ValueStateDescriptor<Tuple2<Long,Long>> descriptor = new ValueStateDescriptor<Tuple2<Long, Long>>("avg", TypeInformation.of(
new TypeHint<Tuple2<Long, Long>>() {}));
state = getRuntimeContext().getState(descriptor);
}
}
Every time I restarted the job, The stateValue is still null.
wanglei2@geekplus.com.cn
Re: Re: Unable to restore state value after job failed using RocksDBStateBackend
Posted by "wanglei2@geekplus.com.cn" <wa...@geekplus.com.cn>.
I start and cancel it just in my intellij idea development environment.
First click the run button, then click the red stop button, and then click the run button again.
Let me google about the savepoint.
Thanks,
Lei Wang
wanglei2@geekplus.com.cn
From: Stephan Ewen
Date: 2019-06-25 20:36
To: user
Subject: Re: Unable to restore state value after job failed using RocksDBStateBackend
If you manually cancel and restart the job, state is only carried forward if you use a savepoint.
Can you check if that is what you are doing?
On Tue, Jun 25, 2019 at 2:21 PM Simon Su <ba...@163.com> wrote:
Hi wanglei
Can you post how you restart the job ?
Thanks,
Simon
On 06/25/2019 20:11，wanglei2@geekplus.com.cn<wa...@geekplus.com.cn> wrote：
public class StateProcessTest extends KeyedProcessFunction<String, Tuple2<String, Long>, String> {
private transient ValueState<Tuple2<Long,Long>> state;
public void processElement(Tuple2<String, Long> value, Context ctx, Collector<String> out) throws Exception {
Tuple2<Long, Long> stateValue = state.value();
if(stateValue == null){
log.info("########## initialize");
stateValue = new Tuple2(34l,56l);
}
state.update(stateValue);
}
@Override
public void open(Configuration parameters) throws Exception {
ValueStateDescriptor<Tuple2<Long,Long>> descriptor = new ValueStateDescriptor<Tuple2<Long, Long>>("avg", TypeInformation.of(
new TypeHint<Tuple2<Long, Long>>() {}));
state = getRuntimeContext().getState(descriptor);
}
}
Every time I restarted the job, The stateValue is still null.
wanglei2@geekplus.com.cn
Re: Unable to restore state value after job failed using RocksDBStateBackend
Posted by Stephan Ewen <se...@apache.org>.
If you manually cancel and restart the job, state is only carried forward
if you use a savepoint.
Can you check if that is what you are doing?
On Tue, Jun 25, 2019 at 2:21 PM Simon Su <ba...@163.com> wrote:
>
> Hi wanglei
>
> Can you post how you restart the job ?
>
> Thanks,
> Simon
> On 06/25/2019 20:11，wanglei2@geekplus.com.cn<wa...@geekplus.com.cn>
> <wa...@geekplus.com.cn> wrote：
>
> public class StateProcessTest extends KeyedProcessFunction<String, Tuple2<String, Long>, String> {
>
> private transient ValueState<Tuple2<Long,Long>> state;
> public void processElement(Tuple2<String, Long> value, Context ctx, Collector<String> out) throws Exception {
>
>
> Tuple2<Long, Long> stateValue = state.value();
>
> if(stateValue == null){
> log.info("########## initialize");
> stateValue = new Tuple2(34l,56l);
> }
> state.update(stateValue);
>
> }
>
> @Override
> public void open(Configuration parameters) throws Exception {
> ValueStateDescriptor<Tuple2<Long,Long>> descriptor = new ValueStateDescriptor<Tuple2<Long, Long>>("avg", TypeInformation.of(
> new TypeHint<Tuple2<Long, Long>>() {}));
> state = getRuntimeContext().getState(descriptor);
> }
> }
>
>
>
>
> Every time I restarted the job, The stateValue is still null.
>
>
> ------------------------------
> wanglei2@geekplus.com.cn
>
>
>
>
Re:Unable to restore state value after job failed using
RocksDBStateBackend
Posted by Simon Su <ba...@163.com>.
Hi wanglei
Can you post how you restart the job ?
Thanks,
Simon
On 06/25/2019 20:11，wanglei2@geekplus.com.cn<wa...@geekplus.com.cn> wrote：
public class StateProcessTest extends KeyedProcessFunction<String, Tuple2<String, Long>, String> {
private transient ValueState<Tuple2<Long,Long>> state;
public void processElement(Tuple2<String, Long> value, Context ctx, Collector<String> out) throws Exception {
Tuple2<Long, Long> stateValue = state.value();
if(stateValue == null){
log.info("########## initialize");
stateValue = new Tuple2(34l,56l);
}
state.update(stateValue);
}
@Override
public void open(Configuration parameters) throws Exception {
ValueStateDescriptor<Tuple2<Long,Long>> descriptor = new ValueStateDescriptor<Tuple2<Long, Long>>("avg", TypeInformation.of(
new TypeHint<Tuple2<Long, Long>>() {}));
state = getRuntimeContext().getState(descriptor);
}
}
Every time I restarted the job, The stateValue is still null.
wanglei2@geekplus.com.cn
Unable to restore state value after job failed using RocksDBStateBackend
Posted by "wanglei2@geekplus.com.cn" <wa...@geekplus.com.cn>.
public class StateProcessTest extends KeyedProcessFunction<String, Tuple2<String, Long>, String> {
private transient ValueState<Tuple2<Long,Long>> state;
public void processElement(Tuple2<String, Long> value, Context ctx, Collector<String> out) throws Exception {
Tuple2<Long, Long> stateValue = state.value();
if(stateValue == null){
log.info("########## initialize");
stateValue = new Tuple2(34l,56l);
}
state.update(stateValue);
}
@Override
public void open(Configuration parameters) throws Exception {
ValueStateDescriptor<Tuple2<Long,Long>> descriptor = new ValueStateDescriptor<Tuple2<Long, Long>>("avg", TypeInformation.of(
new TypeHint<Tuple2<Long, Long>>() {}));
state = getRuntimeContext().getState(descriptor);
}
}
Every time I restarted the job, The stateValue is still null.
wanglei2@geekplus.com.cn
Re: How to trigger the window function even there's no message input
in this window?
Posted by Piotr Nowojski <pi...@ververica.com>.
Hi,
As far as I know, this is currently impossible.
You can workaround this issue by maybe implementing your own custom post processing operator/flatMap function, that would:
- track the output of window operator
- register processing time timer with some desired timeout
- every time the processing time timer fires, your code would check if window operator has emitted something in the last X seconds period. If not, it could emit some default element
Piotrek
> On 14 Jun 2019, at 12:08, wanglei2@geekplus.com.cn wrote:
>
>
> windowAll(TumblingProcessingTimeWindows.of(Time.seconds(10))).process(new MyProcessWindowFunction());
> How can i trigger the MyProcessWindowFunction even there's no input during this window time?
>
> wanglei2@geekplus.com.cn <ma...@geekplus.com.cn>
How to trigger the window function even there's no message input in this window?
Posted by "wanglei2@geekplus.com.cn" <wa...@geekplus.com.cn>.
windowAll(TumblingProcessingTimeWindows.of(Time.seconds(10))).process(new MyProcessWindowFunction());How can i trigger the MyProcessWindowFunction even there's no input during this window time?
wanglei2@geekplus.com.cn
Re: Re: How can I add config file as classpath in taskmgr node when submitting a flink job?
Posted by "wanglei2@geekplus.com.cn" <wa...@geekplus.com.cn>.
Thanks. Let me have a try
wanglei2@geekplus.com.cn
From: Yang Wang
Date: 2019-05-28 09:47
To: wanglei2@geekplus.com.cn
CC: user
Subject: Re: How can I add config file as classpath in taskmgr node when submitting a flink job?
Hi, wangleiYou could use the flink distributed cache to register some config files and then access them in your task.1. Register a cached fileStreamExecutionEnvironment.registerCachedFile(inputFile.toString(), "test_data", false);2. Access the file in your taskfinal Path testFile = getRuntimeContext().getDistributedCache().getFile("test_data").toPath();
wanglei2@geekplus.com.cn <wa...@geekplus.com.cn> 于2019年5月26日周日 上午12:06写道：
When starting a single node java application, I can add some config file to it.
How can i implenment it when submitting a flink job? The config file need to be read from taskMgr node and used to initialize some classess.
wanglei2@geekplus.com.cn
Re: How can I add config file as classpath in taskmgr node when
submitting a flink job?
Posted by Yang Wang <da...@gmail.com>.
Hi, wanglei
You could use the flink distributed cache to register some config
files and then access them in your task.
1. Register a cached file
StreamExecutionEnvironment.registerCachedFile(inputFile.toString(),
"test_data", false);
2. Access the file in your task
final Path testFile =
getRuntimeContext().getDistributedCache().getFile("test_data").toPath();
wanglei2@geekplus.com.cn <wa...@geekplus.com.cn> 于2019年5月26日周日 上午12:06写道：
>
> When starting a single node java application, I can add some config file
> to it.
>
> How can i implenment it when submitting a flink job? The config file need
> to be read from taskMgr node and used to initialize some classess.
>
>
>
> ------------------------------
> wanglei2@geekplus.com.cn
>
How can I add config file as classpath in taskmgr node when submitting a flink job?
Posted by "wanglei2@geekplus.com.cn" <wa...@geekplus.com.cn>.
When starting a single node java application, I can add some config file to it.
How can i implenment it when submitting a flink job? The config file need to be read from taskMgr node and used to initialize some classess.
wanglei2@geekplus.com.cn
Re: Re: How can i just implement a crontab function using flink?
Posted by "wanglei2@geekplus.com.cn" <wa...@geekplus.com.cn>.
Thanks. got it
wanglei2@geekplus.com.cn
From: Puneet Kinra
Date: 2019-05-24 17:02
To: wanglei2@geekplus.com.cn
CC: user
Subject: Re: How can i just implement a crontab function using flink?
There is concept of periodic watermarker , you can use that
if you are working on eventtime.
On Fri, May 24, 2019 at 1:51 PM wanglei2@geekplus.com.cn <wa...@geekplus.com.cn> wrote:
I want to do something every one minute.
Using TumblingWindow, the function will not be triigged if there's no message received during this minute. But i still need to execute the funtion.
How can i implement it ?
wanglei2@geekplus.com.cn
--
Cheers
Puneet Kinra
Mobile:+918800167808 | Skype : puneet.kinra@customercentria.com
e-mail :puneet.kinra@customercentria.com