You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by dhirajpraj <dh...@gmail.com> on 2018/07/08 13:47:25 UTC
How to load data in flink memory?
I want to load some data in flink memory more than the size of individual
task manager memory. Is this possible with flink? I want to use this loaded
data to process other incoming events.
--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/
Re: How to load data in flink memory?
Posted by dhirajpraj <dh...@gmail.com>.
Isn't the data loaded in memory distributed among the task managers?
--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/
Re: How to load data in flink memory?
Posted by miki haiat <mi...@gmail.com>.
If you want to use earlier or other data and process it.
You can use process function process_function
<https://ci.apache.org/projects/flink/flink-docs-release-1.5/dev/stream/operators/process_function.html>
Also loading data in to memory isn't a good idea , you can crash the task
manager .
Using state backend like rocks-db will be very useful , working-with-state
<https://ci.apache.org/projects/flink/flink-docs-release-1.5/dev/stream/state/state.html#working-with-state>
On Sun, Jul 8, 2018 at 4:47 PM dhirajpraj <dh...@gmail.com> wrote:
> I want to load some data in flink memory more than the size of individual
> task manager memory. Is this possible with flink? I want to use this loaded
> data to process other incoming events.
>
>
>
> --
> Sent from:
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/
>