You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Maminspapin <un...@mail.ru> on 2021/05/27 07:20:54 UTC
Parallelism with onTimer() in connectedStream
Hello!
I have a ConnectedStream consists of two DataStreams:
- user (dataStream)
- event (dataStream)
It allows me to use storages of that DataStreams to manage information of
user (user info depends on event of this user) and events:
- users (store)
- events (store)
So:
- user info is changed by every event of this user
- event of user is need to be processed considering user info. It means if
we actually need to process an event. If yes, so we should register some
work using timers (not processing immidiately, but after some period using
rules and user info)
- onTimer() method use users and events stores to proper processing the
logic
Problem:
When I use parallelism =1, then onTimer() method works correctly.
But I want to have a possibility to scaling this work. Thus in my code I set
parallelism in 3.
And there are problems...
- If two users registered on the same time, onTimer() can process only one
of them but twice in two different threads.
How can I adjust the correct work in parallelism>1?
Thanks!
--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/
Re: Parallelism with onTimer() in connectedStream
Posted by Maminspapin <un...@mail.ru>.
Any idea, guys? Can timers work correctly with parallelism? May be it's my
fault it works such way....
--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/
Re: Parallelism with onTimer() in connectedStream
Posted by Maminspapin <un...@mail.ru>.
Some logs and comments:
--- user key: Test_1 __Co-Keyed-Process -> Sink: Sink (2/3)#0 // get user
and store in 2 thread (process1: user stream)
--- user key: Test_2 __Co-Keyed-Process -> Sink: Sink (3/3)#0 // get user
and store in 3 thread (process1: user stream)
--- user key: Test_3 __Co-Keyed-Process -> Sink: Sink (1/3)#0 // get user
and store in 1 thread (process1: user stream)
+++ event key: Test_1 __Co-Keyed-Process -> Sink: Sink (2/3)#0 // get event
and store in 2 thread (process2: event stream)
!!! Reg_ontimer: 2021-05-27T10:07:16.000+03:00 user: Test_1, event:
Test_1__Co-Keyed-Process -> Sink: Sink (2/3)#0
// reg the work in ontimer 2 thread (process2: event stream)
+++ event key: Test_2 __Co-Keyed-Process -> Sink: Sink (3/3)#0 // get event
and store in 3 thread (process2: event stream)
!!! Reg_ontimer: 2021-05-27T10:07:17.000+03:00 user: Test_2, event: Test_2
__Co-Keyed-Process -> Sink: Sink (3/3)#0
// reg the work in ontimer in 3 thread (process2: event stream)
+++ event key: Test_3__Co-Keyed-Process -> Sink: Sink (1/3)#0 // get event
and store in 1 thread (process2: event stream)
!!! Reg_ontimer: 2021-05-27T10:07:17.000+03:00 user: Test_3, event: Test_3
__Co-Keyed-Process -> Sink: Sink (1/3)#0
// reg the work in ontimer in 1 thread (process2: event stream)
Reg_onTimer() next push: 2021-05-27T10:07:26.000+03:00 action: reg first
push, user: Test_1 __Co-Keyed-Process -> Sink: Sink (2/3)#0 // ontimer 2
thread
Reg_onTimer() next push: 2021-05-27T10:07:27.000+03:00 action: reg first
push, user: Test_2 __Co-Keyed-Process -> Sink: Sink (3/3)#0 // ontimer 3
thread
Reg_onTimer() next push: 2021-05-27T10:07:27.000+03:00 action: reg first
push, user: Test_3 __Co-Keyed-Process -> Sink: Sink (1/3)#0 // ontimer 1
thread
---> THE SAME TIME for Test_2 and Test_3 !!!
2021-05-27T10:07:26.016+03:00 ###### action: first push, user: Test_1
__Co-Keyed-Process -> Sink: Sink (2/3)#0 // ontimer 2 thread
2021-05-27T10:07:27.005+03:00 ###### action: first push, user: Test_3
__Co-Keyed-Process -> Sink: Sink (1/3)#0 // ontimer 1 thread
2021-05-27T10:07:27.008+03:00 ###### action: first push, user: Test_3
__Co-Keyed-Process -> Sink: Sink (3/3)#0 // ontimer 3 thread
---> No action for Test_2, twice action for Test_3
--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/