You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by Lu Niu <qq...@gmail.com> on 2021/04/05 04:06:24 UTC
Zigzag shape in TM JVM used memory
Hi, Flink dev
We observed that the TM JVM used memory metric shows zigzag shape among
lots of our applications, although these applications are quite different
in business logic. The upper bound is close to the max heap size. Is this
expected in flink application? Or does flink internally
aggressively pre-allocate memory?
app1
[image: Screen Shot 2021-04-04 at 8.46.45 PM.png]
app2
[image: Screen Shot 2021-04-04 at 8.45.35 PM.png]
app3
[image: Screen Shot 2021-04-04 at 8.43.53 PM.png]
Best
Lu
Re: Zigzag shape in TM JVM used memory
Posted by Piotr Nowojski <pn...@apache.org>.
Hi,
I don't think there is a Flink specific answer to this question. Just do
what you would normally do with a normal Java application running inside a
JVM. If there is an OOM on heap space, you can either try to bump the heap
space, or reduce usage of it. The only Flink specific part is probably that
you need to leave enough memory for the framework itself, and that there
are a couple of different memory pools. You can read about those things in
the docs:
https://ci.apache.org/projects/flink/flink-docs-release-1.12/deployment/memory/mem_setup_tm.html
https://ci.apache.org/projects/flink/flink-docs-release-1.12/deployment/memory/mem_tuning.html
https://ci.apache.org/projects/flink/flink-docs-release-1.12/deployment/memory/mem_trouble.html
Piotrek
czw., 8 kwi 2021 o 02:19 Lu Niu <qq...@gmail.com> napisał(a):
> Hi, Piotr
>
> Thanks for replying. I asked this because such a pattern might imply memory
> oversubscription. For example, I tuned down the memory of one app from heap
> 2.63GB to 367MB and the job still runs fine:
> before:
>
> https://drive.google.com/file/d/1o8k9Vv3yb5gXITi4GvmlXMteQcRfmOhr/view?usp=sharing
>
> after:
>
> https://drive.google.com/file/d/1wNTHBT8aSJaAmL1rVY8jUkdp-G5znnMN/view?usp=sharing
>
>
> What's the best practice for tuning Flink job memory?
>
> 1. What’s a good start point users should try first?
> 2. How to make progress? e.g. flink application Foo currently encountered
> error OOM: java heap space. Where to move next? simply bump up
> taskmananger.memory? or just increase heap?
> 3. What’s the final state? Job running fine and ensuring XYZ headroom in
> each memory component?
>
> Best
> Lu
>
> On Tue, Apr 6, 2021 at 12:26 AM Piotr Nowojski <pn...@apache.org>
> wrote:
>
> > Hi,
> >
> > this should be posted on the user mailing list not the dev.
> >
> > Apart from that, this looks like normal/standard behaviour of JVM, and
> has
> > very little to do with Flink. Garbage Collector (GC) is kicking in when
> > memory usage is approaching some threshold:
> > https://www.google.com/search?q=jvm+heap+memory+usage&tbm=isch
> >
> > Piotrek
> >
> >
> > pon., 5 kwi 2021 o 22:54 Lu Niu <qq...@gmail.com> napisał(a):
> >
> > > Hi,
> > >
> > > we need to update our email system then :) . Here are the links:
> > >
> > >
> > >
> >
> https://drive.google.com/file/d/1lZ5_P8_NqsN1JeLzutGj4DxkyWJN75mR/view?usp=sharing
> > >
> > >
> > >
> >
> https://drive.google.com/file/d/1J6c6rQJwtDp1moAGlvQyLQXTqcuG4HjL/view?usp=sharing
> > >
> > >
> > >
> >
> https://drive.google.com/file/d/1-R2KzsABC471AEjkF5qTm5O3V47cpbBV/view?usp=sharing
> > >
> > > All are DataStream job.
> > >
> > > Best
> > > Lu
> > >
> > > On Sun, Apr 4, 2021 at 9:17 PM Yun Gao <yu...@aliyun.com> wrote:
> > >
> > > >
> > > > Hi Lu,
> > > >
> > > > The image seems not be able to shown due to the mail server
> limitation,
> > > > could you upload it somewhere and paste the link here ?
> > > >
> > > > Logically, I think zigzag usually due to there are some small object
> > get
> > > > created and eliminated soon in the heap. Are you running a SQL job
> or a
> > > > DataStream job ?
> > > >
> > > > Best,
> > > > Yun
> > > >
> > > > ------------------------------------------------------------------
> > > > Sender:Lu Niu<qq...@gmail.com>
> > > > Date:2021/04/05 12:06:24
> > > > Recipient:dev@flink.apache.org<de...@flink.apache.org>
> > > > Theme:Zigzag shape in TM JVM used memory
> > > >
> > > > Hi, Flink dev
> > > >
> > > > We observed that the TM JVM used memory metric shows zigzag shape
> among
> > > > lots of our applications, although these applications are quite
> > different
> > > > in business logic. The upper bound is close to the max heap size. Is
> > this
> > > > expected in flink application? Or does flink internally
> > > > aggressively pre-allocate memory?
> > > >
> > > > app1
> > > > [image: Screen Shot 2021-04-04 at 8.46.45 PM.png]
> > > > app2
> > > > [image: Screen Shot 2021-04-04 at 8.45.35 PM.png]
> > > > app3
> > > > [image: Screen Shot 2021-04-04 at 8.43.53 PM.png]
> > > >
> > > > Best
> > > > Lu
> > > >
> > > >
> > >
> >
>
Re: Zigzag shape in TM JVM used memory
Posted by Piotr Nowojski <pn...@apache.org>.
Hi,
I don't think there is a Flink specific answer to this question. Just do
what you would normally do with a normal Java application running inside a
JVM. If there is an OOM on heap space, you can either try to bump the heap
space, or reduce usage of it. The only Flink specific part is probably that
you need to leave enough memory for the framework itself, and that there
are a couple of different memory pools. You can read about those things in
the docs:
https://ci.apache.org/projects/flink/flink-docs-release-1.12/deployment/memory/mem_setup_tm.html
https://ci.apache.org/projects/flink/flink-docs-release-1.12/deployment/memory/mem_tuning.html
https://ci.apache.org/projects/flink/flink-docs-release-1.12/deployment/memory/mem_trouble.html
Piotrek
czw., 8 kwi 2021 o 02:19 Lu Niu <qq...@gmail.com> napisał(a):
> Hi, Piotr
>
> Thanks for replying. I asked this because such a pattern might imply memory
> oversubscription. For example, I tuned down the memory of one app from heap
> 2.63GB to 367MB and the job still runs fine:
> before:
>
> https://drive.google.com/file/d/1o8k9Vv3yb5gXITi4GvmlXMteQcRfmOhr/view?usp=sharing
>
> after:
>
> https://drive.google.com/file/d/1wNTHBT8aSJaAmL1rVY8jUkdp-G5znnMN/view?usp=sharing
>
>
> What's the best practice for tuning Flink job memory?
>
> 1. What’s a good start point users should try first?
> 2. How to make progress? e.g. flink application Foo currently encountered
> error OOM: java heap space. Where to move next? simply bump up
> taskmananger.memory? or just increase heap?
> 3. What’s the final state? Job running fine and ensuring XYZ headroom in
> each memory component?
>
> Best
> Lu
>
> On Tue, Apr 6, 2021 at 12:26 AM Piotr Nowojski <pn...@apache.org>
> wrote:
>
> > Hi,
> >
> > this should be posted on the user mailing list not the dev.
> >
> > Apart from that, this looks like normal/standard behaviour of JVM, and
> has
> > very little to do with Flink. Garbage Collector (GC) is kicking in when
> > memory usage is approaching some threshold:
> > https://www.google.com/search?q=jvm+heap+memory+usage&tbm=isch
> >
> > Piotrek
> >
> >
> > pon., 5 kwi 2021 o 22:54 Lu Niu <qq...@gmail.com> napisał(a):
> >
> > > Hi,
> > >
> > > we need to update our email system then :) . Here are the links:
> > >
> > >
> > >
> >
> https://drive.google.com/file/d/1lZ5_P8_NqsN1JeLzutGj4DxkyWJN75mR/view?usp=sharing
> > >
> > >
> > >
> >
> https://drive.google.com/file/d/1J6c6rQJwtDp1moAGlvQyLQXTqcuG4HjL/view?usp=sharing
> > >
> > >
> > >
> >
> https://drive.google.com/file/d/1-R2KzsABC471AEjkF5qTm5O3V47cpbBV/view?usp=sharing
> > >
> > > All are DataStream job.
> > >
> > > Best
> > > Lu
> > >
> > > On Sun, Apr 4, 2021 at 9:17 PM Yun Gao <yu...@aliyun.com> wrote:
> > >
> > > >
> > > > Hi Lu,
> > > >
> > > > The image seems not be able to shown due to the mail server
> limitation,
> > > > could you upload it somewhere and paste the link here ?
> > > >
> > > > Logically, I think zigzag usually due to there are some small object
> > get
> > > > created and eliminated soon in the heap. Are you running a SQL job
> or a
> > > > DataStream job ?
> > > >
> > > > Best,
> > > > Yun
> > > >
> > > > ------------------------------------------------------------------
> > > > Sender:Lu Niu<qq...@gmail.com>
> > > > Date:2021/04/05 12:06:24
> > > > Recipient:dev@flink.apache.org<de...@flink.apache.org>
> > > > Theme:Zigzag shape in TM JVM used memory
> > > >
> > > > Hi, Flink dev
> > > >
> > > > We observed that the TM JVM used memory metric shows zigzag shape
> among
> > > > lots of our applications, although these applications are quite
> > different
> > > > in business logic. The upper bound is close to the max heap size. Is
> > this
> > > > expected in flink application? Or does flink internally
> > > > aggressively pre-allocate memory?
> > > >
> > > > app1
> > > > [image: Screen Shot 2021-04-04 at 8.46.45 PM.png]
> > > > app2
> > > > [image: Screen Shot 2021-04-04 at 8.45.35 PM.png]
> > > > app3
> > > > [image: Screen Shot 2021-04-04 at 8.43.53 PM.png]
> > > >
> > > > Best
> > > > Lu
> > > >
> > > >
> > >
> >
>
Re: Zigzag shape in TM JVM used memory
Posted by Lu Niu <qq...@gmail.com>.
Hi, Piotr
Thanks for replying. I asked this because such a pattern might imply memory
oversubscription. For example, I tuned down the memory of one app from heap
2.63GB to 367MB and the job still runs fine:
before:
https://drive.google.com/file/d/1o8k9Vv3yb5gXITi4GvmlXMteQcRfmOhr/view?usp=sharing
after:
https://drive.google.com/file/d/1wNTHBT8aSJaAmL1rVY8jUkdp-G5znnMN/view?usp=sharing
What's the best practice for tuning Flink job memory?
1. What’s a good start point users should try first?
2. How to make progress? e.g. flink application Foo currently encountered
error OOM: java heap space. Where to move next? simply bump up
taskmananger.memory? or just increase heap?
3. What’s the final state? Job running fine and ensuring XYZ headroom in
each memory component?
Best
Lu
On Tue, Apr 6, 2021 at 12:26 AM Piotr Nowojski <pn...@apache.org> wrote:
> Hi,
>
> this should be posted on the user mailing list not the dev.
>
> Apart from that, this looks like normal/standard behaviour of JVM, and has
> very little to do with Flink. Garbage Collector (GC) is kicking in when
> memory usage is approaching some threshold:
> https://www.google.com/search?q=jvm+heap+memory+usage&tbm=isch
>
> Piotrek
>
>
> pon., 5 kwi 2021 o 22:54 Lu Niu <qq...@gmail.com> napisał(a):
>
> > Hi,
> >
> > we need to update our email system then :) . Here are the links:
> >
> >
> >
> https://drive.google.com/file/d/1lZ5_P8_NqsN1JeLzutGj4DxkyWJN75mR/view?usp=sharing
> >
> >
> >
> https://drive.google.com/file/d/1J6c6rQJwtDp1moAGlvQyLQXTqcuG4HjL/view?usp=sharing
> >
> >
> >
> https://drive.google.com/file/d/1-R2KzsABC471AEjkF5qTm5O3V47cpbBV/view?usp=sharing
> >
> > All are DataStream job.
> >
> > Best
> > Lu
> >
> > On Sun, Apr 4, 2021 at 9:17 PM Yun Gao <yu...@aliyun.com> wrote:
> >
> > >
> > > Hi Lu,
> > >
> > > The image seems not be able to shown due to the mail server limitation,
> > > could you upload it somewhere and paste the link here ?
> > >
> > > Logically, I think zigzag usually due to there are some small object
> get
> > > created and eliminated soon in the heap. Are you running a SQL job or a
> > > DataStream job ?
> > >
> > > Best,
> > > Yun
> > >
> > > ------------------------------------------------------------------
> > > Sender:Lu Niu<qq...@gmail.com>
> > > Date:2021/04/05 12:06:24
> > > Recipient:dev@flink.apache.org<de...@flink.apache.org>
> > > Theme:Zigzag shape in TM JVM used memory
> > >
> > > Hi, Flink dev
> > >
> > > We observed that the TM JVM used memory metric shows zigzag shape among
> > > lots of our applications, although these applications are quite
> different
> > > in business logic. The upper bound is close to the max heap size. Is
> this
> > > expected in flink application? Or does flink internally
> > > aggressively pre-allocate memory?
> > >
> > > app1
> > > [image: Screen Shot 2021-04-04 at 8.46.45 PM.png]
> > > app2
> > > [image: Screen Shot 2021-04-04 at 8.45.35 PM.png]
> > > app3
> > > [image: Screen Shot 2021-04-04 at 8.43.53 PM.png]
> > >
> > > Best
> > > Lu
> > >
> > >
> >
>
Re: Zigzag shape in TM JVM used memory
Posted by Lu Niu <qq...@gmail.com>.
Hi, Piotr
Thanks for replying. I asked this because such a pattern might imply memory
oversubscription. For example, I tuned down the memory of one app from heap
2.63GB to 367MB and the job still runs fine:
before:
https://drive.google.com/file/d/1o8k9Vv3yb5gXITi4GvmlXMteQcRfmOhr/view?usp=sharing
after:
https://drive.google.com/file/d/1wNTHBT8aSJaAmL1rVY8jUkdp-G5znnMN/view?usp=sharing
What's the best practice for tuning Flink job memory?
1. What’s a good start point users should try first?
2. How to make progress? e.g. flink application Foo currently encountered
error OOM: java heap space. Where to move next? simply bump up
taskmananger.memory? or just increase heap?
3. What’s the final state? Job running fine and ensuring XYZ headroom in
each memory component?
Best
Lu
On Tue, Apr 6, 2021 at 12:26 AM Piotr Nowojski <pn...@apache.org> wrote:
> Hi,
>
> this should be posted on the user mailing list not the dev.
>
> Apart from that, this looks like normal/standard behaviour of JVM, and has
> very little to do with Flink. Garbage Collector (GC) is kicking in when
> memory usage is approaching some threshold:
> https://www.google.com/search?q=jvm+heap+memory+usage&tbm=isch
>
> Piotrek
>
>
> pon., 5 kwi 2021 o 22:54 Lu Niu <qq...@gmail.com> napisał(a):
>
> > Hi,
> >
> > we need to update our email system then :) . Here are the links:
> >
> >
> >
> https://drive.google.com/file/d/1lZ5_P8_NqsN1JeLzutGj4DxkyWJN75mR/view?usp=sharing
> >
> >
> >
> https://drive.google.com/file/d/1J6c6rQJwtDp1moAGlvQyLQXTqcuG4HjL/view?usp=sharing
> >
> >
> >
> https://drive.google.com/file/d/1-R2KzsABC471AEjkF5qTm5O3V47cpbBV/view?usp=sharing
> >
> > All are DataStream job.
> >
> > Best
> > Lu
> >
> > On Sun, Apr 4, 2021 at 9:17 PM Yun Gao <yu...@aliyun.com> wrote:
> >
> > >
> > > Hi Lu,
> > >
> > > The image seems not be able to shown due to the mail server limitation,
> > > could you upload it somewhere and paste the link here ?
> > >
> > > Logically, I think zigzag usually due to there are some small object
> get
> > > created and eliminated soon in the heap. Are you running a SQL job or a
> > > DataStream job ?
> > >
> > > Best,
> > > Yun
> > >
> > > ------------------------------------------------------------------
> > > Sender:Lu Niu<qq...@gmail.com>
> > > Date:2021/04/05 12:06:24
> > > Recipient:dev@flink.apache.org<de...@flink.apache.org>
> > > Theme:Zigzag shape in TM JVM used memory
> > >
> > > Hi, Flink dev
> > >
> > > We observed that the TM JVM used memory metric shows zigzag shape among
> > > lots of our applications, although these applications are quite
> different
> > > in business logic. The upper bound is close to the max heap size. Is
> this
> > > expected in flink application? Or does flink internally
> > > aggressively pre-allocate memory?
> > >
> > > app1
> > > [image: Screen Shot 2021-04-04 at 8.46.45 PM.png]
> > > app2
> > > [image: Screen Shot 2021-04-04 at 8.45.35 PM.png]
> > > app3
> > > [image: Screen Shot 2021-04-04 at 8.43.53 PM.png]
> > >
> > > Best
> > > Lu
> > >
> > >
> >
>
Re: Zigzag shape in TM JVM used memory
Posted by Piotr Nowojski <pn...@apache.org>.
Hi,
this should be posted on the user mailing list not the dev.
Apart from that, this looks like normal/standard behaviour of JVM, and has
very little to do with Flink. Garbage Collector (GC) is kicking in when
memory usage is approaching some threshold:
https://www.google.com/search?q=jvm+heap+memory+usage&tbm=isch
Piotrek
pon., 5 kwi 2021 o 22:54 Lu Niu <qq...@gmail.com> napisał(a):
> Hi,
>
> we need to update our email system then :) . Here are the links:
>
>
> https://drive.google.com/file/d/1lZ5_P8_NqsN1JeLzutGj4DxkyWJN75mR/view?usp=sharing
>
>
> https://drive.google.com/file/d/1J6c6rQJwtDp1moAGlvQyLQXTqcuG4HjL/view?usp=sharing
>
>
> https://drive.google.com/file/d/1-R2KzsABC471AEjkF5qTm5O3V47cpbBV/view?usp=sharing
>
> All are DataStream job.
>
> Best
> Lu
>
> On Sun, Apr 4, 2021 at 9:17 PM Yun Gao <yu...@aliyun.com> wrote:
>
> >
> > Hi Lu,
> >
> > The image seems not be able to shown due to the mail server limitation,
> > could you upload it somewhere and paste the link here ?
> >
> > Logically, I think zigzag usually due to there are some small object get
> > created and eliminated soon in the heap. Are you running a SQL job or a
> > DataStream job ?
> >
> > Best,
> > Yun
> >
> > ------------------------------------------------------------------
> > Sender:Lu Niu<qq...@gmail.com>
> > Date:2021/04/05 12:06:24
> > Recipient:dev@flink.apache.org<de...@flink.apache.org>
> > Theme:Zigzag shape in TM JVM used memory
> >
> > Hi, Flink dev
> >
> > We observed that the TM JVM used memory metric shows zigzag shape among
> > lots of our applications, although these applications are quite different
> > in business logic. The upper bound is close to the max heap size. Is this
> > expected in flink application? Or does flink internally
> > aggressively pre-allocate memory?
> >
> > app1
> > [image: Screen Shot 2021-04-04 at 8.46.45 PM.png]
> > app2
> > [image: Screen Shot 2021-04-04 at 8.45.35 PM.png]
> > app3
> > [image: Screen Shot 2021-04-04 at 8.43.53 PM.png]
> >
> > Best
> > Lu
> >
> >
>
Re: Zigzag shape in TM JVM used memory
Posted by Piotr Nowojski <pn...@apache.org>.
Hi,
this should be posted on the user mailing list not the dev.
Apart from that, this looks like normal/standard behaviour of JVM, and has
very little to do with Flink. Garbage Collector (GC) is kicking in when
memory usage is approaching some threshold:
https://www.google.com/search?q=jvm+heap+memory+usage&tbm=isch
Piotrek
pon., 5 kwi 2021 o 22:54 Lu Niu <qq...@gmail.com> napisał(a):
> Hi,
>
> we need to update our email system then :) . Here are the links:
>
>
> https://drive.google.com/file/d/1lZ5_P8_NqsN1JeLzutGj4DxkyWJN75mR/view?usp=sharing
>
>
> https://drive.google.com/file/d/1J6c6rQJwtDp1moAGlvQyLQXTqcuG4HjL/view?usp=sharing
>
>
> https://drive.google.com/file/d/1-R2KzsABC471AEjkF5qTm5O3V47cpbBV/view?usp=sharing
>
> All are DataStream job.
>
> Best
> Lu
>
> On Sun, Apr 4, 2021 at 9:17 PM Yun Gao <yu...@aliyun.com> wrote:
>
> >
> > Hi Lu,
> >
> > The image seems not be able to shown due to the mail server limitation,
> > could you upload it somewhere and paste the link here ?
> >
> > Logically, I think zigzag usually due to there are some small object get
> > created and eliminated soon in the heap. Are you running a SQL job or a
> > DataStream job ?
> >
> > Best,
> > Yun
> >
> > ------------------------------------------------------------------
> > Sender:Lu Niu<qq...@gmail.com>
> > Date:2021/04/05 12:06:24
> > Recipient:dev@flink.apache.org<de...@flink.apache.org>
> > Theme:Zigzag shape in TM JVM used memory
> >
> > Hi, Flink dev
> >
> > We observed that the TM JVM used memory metric shows zigzag shape among
> > lots of our applications, although these applications are quite different
> > in business logic. The upper bound is close to the max heap size. Is this
> > expected in flink application? Or does flink internally
> > aggressively pre-allocate memory?
> >
> > app1
> > [image: Screen Shot 2021-04-04 at 8.46.45 PM.png]
> > app2
> > [image: Screen Shot 2021-04-04 at 8.45.35 PM.png]
> > app3
> > [image: Screen Shot 2021-04-04 at 8.43.53 PM.png]
> >
> > Best
> > Lu
> >
> >
>
Re: Zigzag shape in TM JVM used memory
Posted by Lu Niu <qq...@gmail.com>.
Hi,
we need to update our email system then :) . Here are the links:
https://drive.google.com/file/d/1lZ5_P8_NqsN1JeLzutGj4DxkyWJN75mR/view?usp=sharing
https://drive.google.com/file/d/1J6c6rQJwtDp1moAGlvQyLQXTqcuG4HjL/view?usp=sharing
https://drive.google.com/file/d/1-R2KzsABC471AEjkF5qTm5O3V47cpbBV/view?usp=sharing
All are DataStream job.
Best
Lu
On Sun, Apr 4, 2021 at 9:17 PM Yun Gao <yu...@aliyun.com> wrote:
>
> Hi Lu,
>
> The image seems not be able to shown due to the mail server limitation,
> could you upload it somewhere and paste the link here ?
>
> Logically, I think zigzag usually due to there are some small object get
> created and eliminated soon in the heap. Are you running a SQL job or a
> DataStream job ?
>
> Best,
> Yun
>
> ------------------------------------------------------------------
> Sender:Lu Niu<qq...@gmail.com>
> Date:2021/04/05 12:06:24
> Recipient:dev@flink.apache.org<de...@flink.apache.org>
> Theme:Zigzag shape in TM JVM used memory
>
> Hi, Flink dev
>
> We observed that the TM JVM used memory metric shows zigzag shape among
> lots of our applications, although these applications are quite different
> in business logic. The upper bound is close to the max heap size. Is this
> expected in flink application? Or does flink internally
> aggressively pre-allocate memory?
>
> app1
> [image: Screen Shot 2021-04-04 at 8.46.45 PM.png]
> app2
> [image: Screen Shot 2021-04-04 at 8.45.35 PM.png]
> app3
> [image: Screen Shot 2021-04-04 at 8.43.53 PM.png]
>
> Best
> Lu
>
>
Re: Zigzag shape in TM JVM used memory
Posted by Yun Gao <yu...@aliyun.com.INVALID>.
Hi Lu,
The image seems not be able to shown due to the mail server limitation, could you upload it somewhere and paste the link here ?
Logically, I think zigzag usually due to there are some small object get created and eliminated soon in the heap. Are you running a SQL job or a DataStream job ?
Best,
Yun
------------------------------------------------------------------
Sender:Lu Niu<qq...@gmail.com>
Date:2021/04/05 12:06:24
Recipient:dev@flink.apache.org<de...@flink.apache.org>
Theme:Zigzag shape in TM JVM used memory
Hi, Flink dev
We observed that the TM JVM used memory metric shows zigzag shape among lots of our applications, although these applications are quite different in business logic. The upper bound is close to the max heap size. Is this expected in flink application? Or does flink internally aggressively pre-allocate memory?
app1
app2
app3
Best
Lu