You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nuttx.apache.org by Florian Wehmeyer <fl...@gmx.net> on 2020/05/11 14:35:43 UTC
heap malloc when executing binary/elf file
Hi all,
again a question related to the execution of external / elf programs:
When I use malloc within the running elf program, allocating more than
1K bytes, malloc seems to work (doesn't return NULL), but when I use
that memory (memset) I get a assertion/Kernel panic..
Funnily this does not happen if I use the same program as built-in
program (the one which is already in ROM flash).. Here I can allocate
and use 8K or more without any problem..
So, it seems the binary loader itself consumes a lot of memory,
decreasing the available heap.. I was not able to solve that problem
even increasing all available and possibly related stacksizes in
.config
After loading the elf file, even running dmesg can cause that
assertion.. (using RAMLOG with 8K buffer and memory debug output) .
free shows about 130K of free RAM..
in the memory debug output I can see that the periodical allocation at
about 10K of distance of the next stack-end (means stackbase minus
stacksize, knowing stack decreases), which is my app..
so, I thought allocate 2K on the heap should not be a problem.
Any ideas about any configuration that's important here?
Many thanks & BR,
--
Florian WehmeyerTFW Tech-Solutions
Re: heap malloc when executing binary/elf file
Posted by Alan Carvalho de Assis <ac...@gmail.com>.
Hi Florian,
Does this problem only occur in version 9.0.0?
Or is it happening to the current master as well?
Are you following this tutorial: https://www.youtube.com/watch?v=oL6KAgkTb8M
Also you can increase the debug level (include warn and info).
BR,
Alan
On 5/16/20, Florian Wehmeyer <fl...@gmx.net> wrote:
> Hi Alan,
> updated it again to 9.0.0 and also good!
> However, got a new problem now which I've not figured out yet:
> The binary loader does not load the elf if the program is a built-in
> app... very strange..
> It does not even try to load the elf file from file-system, it directly
> goes to the built-in app..
> Funnily, it does try to load the nsh embedded tools:
> exec_spawn: ERROR: Failed to load program 'ls':
> -2
> exec_spawn: ERROR: Failed to load program 'dmesg': -2
> If I do not compile the app as a built-in app, the binary loader will
> load it:
> load_absmodule: Loading
> /mnt/ramdisk/Neo_Ledtest
> elf_loadbinary: Loading file:
> /mnt/ramdisk/Neo_Ledtest
> elf_init: filename: /mnt/ramdisk/Neo_Ledtest loadinfo: 0x2000a4cc
> Which shows that the CONFIG_BINFMT_EXEPATH=y and CONFIG_PATH_INITIAL ar
> e correctly set, and the elf file is also working correctly..
> I never had this situation before...
> BR,
> Florian
>
> --
> Florian Wehmeyer
> TFW Tech-Solutions
> On Sáb, 2020-05-16 at 16:08 -0300, Alan Carvalho de Assis wrote:
>> Hi Florian,
>>
>> Nice to know you got if fixed.
>>
>> It should be nice if you can test it again using the nuttx-9.0.0
>> release.
>>
>> BR,
>>
>> Alan
>>
>> On 5/16/20, Florian Wehmeyer <fl...@gmx.net> wrote:
>> >
>> > Hi all,
>> > just to let you know, the heap problem was solved after pulling the
>> > latest commit from master..
>> > Now I can use very much of it, also I was able to use a lot more
>> > features in the config, as cmd line history etc...
>> > So, the available memory of 256K was not the problem, @Greg.
>> > @Nathan: now malloc returns NULL if out of heap..
>> > Sorry for that, guys, normally someone should make sure to have the
>> > latest version, not a version from 4 weeks ago as I.
>> > What happens is, as there is a lot of changes going on constantly,
>> > sometimes I hesitate to make frequent pushes, because it can mess
>> > up my
>> > processes. (for example if someone changes the nuttx folder
>> > structure
>> > or similar)
>> > Very good that now there are release branches, I hope the release
>> > branch will suffer only bug-fixes as changes..
>> > Thanks again,
>> >
>> > --
>> > Florian Wehmeyer
>> > TFW Tech-Solutions
>> > On Seg, 2020-05-11 at 18:43 -0300, Florian Wehmeyer wrote:
>> > >
>> > > OK Greg,
>> > > could be that I was using too much.. using 64KB of ramdisk also..
>> > > I
>> > > disabled some of the comfortable features like cmd line history
>> > > etc,
>> > > and was already able to alloc quite a bit more than before..
>> > > Will further improve the sram usage..
>> > > Also, I missed the commit
>> > > mm/mm_heap: fix mm_heap not support BUILD_FLAT
>> > >
>> > >
>> > >
>> > > GUIDINGLI
>> > >
>> > >
>> > > authored and patacongo
>> > >
>> > > committed
>> > > on Apr 10
>> > >
>> > >
>> > >
>> > > which well might impact my stuff here, will pull and let you
>> > > know,
>> > > @Nathan I will also make tests if malloc continues returning non
>> > > NULL
>> > > pointers even if out of memory..
>> > >
>> > > Seems on a good way!!
>> > >
>> > > Many thanks,
>> > > Florian
>> > >
>> > >
>> > >
>> > > --
>> > > Florian Wehmeyer
>> > > TFW Tech-Solutions
>> > > On Seg, 2020-05-11 at 15:09 -0600, Gregory Nutt wrote:
>> > > >
>> > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > > > No custom board, it's the tm4c1294-launchpad.
>> > > > > > Seems no general problem with heap allocation, so I guess
>> > > > > > it's
>> > > > > > not in
>> > > > > > the linker script, it's rather directly linked to the usage
>> > > > > > of
>> > > > > > the elf-
>> > > > > > binary loader, and there are only two configs in the nuttx
>> > > > > > repo
>> > > > > > which
>> > > > > > use the CONFIG_ELF stuff.
>> > > > > That part has 256Kb of SRAM. That is more that many, but it
>> > > > > could
>> > > > > be
>> > > > > that you are using too much SRAM and just cannot run ELF
>> > > > > reliably.
>> > > > > It
>> > > > > does want a lot of SRAM.
>> > > > Most of the other boards that use CONFIG_ELF=y in a
>> > > > configuration
>> > > > have a
>> > > > much larger internal ram (exceptions in red):
>> > > >
>> > > > $ find . -name defconfig | xargs grep -l CONFIG_ELF=y | xargs
>> > > > grep
>> > > > RAM_SIZE
>> > > > ./arm/cxd56xx/spresense/configs/elf/defconfig:CONFIG_RAM_SIZE=1
>> > > > 5728
>> > > > 64
>> > > > ./arm/cxd56xx/spresense/configs/module/defconfig:CONFIG_RAM_SIZ
>> > > > E=15
>> > > > 72
>> > > > 864
>> > > > ./arm/cxd56xx/spresense/configs/posix_spawn/defconfig:CONFIG_RA
>> > > > M_SI
>> > > > ZE
>> > > > =1572864
>> > > > ./arm/cxd56xx/spresense/configs/rndis/defconfig:CONFIG_RAM_SIZE
>> > > > =157
>> > > > 28
>> > > > 64
>> > > > ./arm/cxd56xx/spresense/configs/wifi/defconfig:CONFIG_RAM_SIZE=
>> > > > 1572
>> > > > 86
>> > > > 4
>> > > > ./arm/lc823450/lc823450-
>> > > > xgevk/configs/elf/defconfig:CONFIG_RAM_SIZE=1044480
>> > > > ./arm/lc823450/lc823450-
>> > > > xgevk/configs/krndis/defconfig:CONFIG_RAM_SIZE=1044480
>> > > > ./arm/lc823450/lc823450-
>> > > > xgevk/configs/posix_spawn/defconfig:CONFIG_RAM_SIZE=1044480
>> > > > ./arm/lc823450/lc823450-
>> > > > xgevk/configs/rndis/defconfig:CONFIG_RAM_SIZE=1044480
>> > > > ./arm/lc823450/lc823450-
>> > > > xgevk/configs/usb/defconfig:CONFIG_RAM_SIZE=1044480
>> > > > ./arm/lpc17xx_40xx/lx_cpu/configs/nsh/defconfig:CONFIG_RAM_SIZE
>> > > > =655
>> > > > 36
>> > > > ./arm/sama5/sama5d4-
>> > > > ek/configs/elf/defconfig:CONFIG_RAM_SIZE=268435456
>> > > > ./arm/sama5/sama5d4-
>> > > > ek/configs/knsh/defconfig:CONFIG_RAM_SIZE=268435456
>> > > > ./arm/stm32/olimex-stm32-
>> > > > p407/configs/kelf/defconfig:CONFIG_RAM_SIZE=114688
>> > > > ./arm/stm32/stm32f4discovery/configs/elf/defconfig:CONFIG_RAM_S
>> > > > IZE=
>> > > > 11
>> > > > 4688
>> > > > ./arm/stm32/stm32f4discovery/configs/posix_spawn/defconfig:CONF
>> > > > IG_R
>> > > > AM
>> > > > _SIZE=114688
>> > > > ./arm/stm32/stm32f4discovery/configs/rndis/defconfig:CONFIG_RAM
>> > > > _SIZ
>> > > > E=
>> > > > 114688
>> > > > ./arm/tms570/tms570ls31x-usb-
>> > > > kit/configs/nsh/defconfig:CONFIG_RAM_SIZE=262143
>> > > > ./risc-v/k210/maix-
>> > > > bit/configs/elf/defconfig:CONFIG_RAM_SIZE=2097152
>> > > > ./risc-v/k210/maix-
>> > > > bit/configs/module/defconfig:CONFIG_RAM_SIZE=2097152
>> > > > ./risc-v/k210/maix-
>> > > > bit/configs/posix_spawn/defconfig:CONFIG_RAM_SIZE=2097152
>> > > >
>> > > > And one of these exceptions has a large external DRAM:
>> > > >
>> > > > $ find . -name defconfig | xargs grep -l CONFIG_ELF=y | xargs
>> > > > grep
>> > > > EXTDRAM
>> > > > ./arm/lpc17xx_40xx/lx_cpu/configs/nsh/defconfig:CONFIG_LPC17_40
>> > > > _EXT
>> > > > DR
>> > > > AM=y
>> > > > ./arm/lpc17xx_40xx/lx_cpu/configs/nsh/defconfig:CONFIG_LPC17_40
>> > > > _EXT
>> > > > DR
>> > > > AMSIZE=33554432
>> > > >
>> > > >
>
Re: 答复: [External Mail]Re: heap malloc when executing binary/elf file
Posted by Florian Wehmeyer <fl...@gmx.net>.
>
>
> That is true. The assumption is that the stack size of the built-in
> application will be big enough for the replacement ELF application.
> That might not be true. But in the case of replacing a built-in
> application with an ELF application, the rule is that we cannot
> modify
> the base ROM logic. Imagine that I have 100,000 products in the
> field
> and I want to replace a built-in. That is pretty easy to do by just
> writing the ELF file to the some local file system. You would like
> to
> avoid replacing the entire FLASH image.
Yep, that's pretty clear, that's the idea of using the elf file instead
of entire flash image.
This is why I think all the stack sizes should be set according to any
increased needs of updated elf files..
I merged Xiang's change. Please try the changes in the repository now
if you can make the time (without changing the name of hello).
At least seems you got a communication problem here, because nobody had
a notion about the impact of Chaos commit.
tested with commit 8997213c2aca8fae0f5537833014f9e016b51dd2 :
nsh> hello
Hello, World ELF!!
nsh> dmesg
load_absmodule: Loading /mnt/ramdisk/hello
elf_loadbinary: Loading file: /mnt/ramdisk/hello
elf_init: filename: /mnt/ramdisk/hello loadinfo: 0x20009f2c
elf_read: Read 52 bytes from offset 0
......
==> working again as supposed.
Thanks & BR,
Florian
Re: 答复: [External Mail]Re: heap malloc when executing binary/elf file
Posted by Gregory Nutt <sp...@gmail.com>.
>
>> So, in current git HEAD [1], what is the search order now?
>>
>> Is it back to:
>> - Elf
>> - Built-in
>> - NSH
>>
>> Or is it:
>> - Built-in
>> - Elf
>> - NSH
>
> It is not so clean and well partitioned. It is more like:
>
> o IF Built-in available
> - Elf
> - IF NOT Elf
> Built-in
> o Elf
> o NSH
>
> The logic is spread across a couple of files so is not in one place
> anymore. It is not pretty, but seems like it should work.
>
> The only advantage is in this case:
>
> o IF Built-in available
> - Elf
>
> For built-ins, we have task stack size and priority information. For
>
> o Elf
>
>
> But we do not and have to use a default stack size which will be most
> certainly wrong. We are contemplating fixing that via
> https://github.com/apache/incubator-nuttx/issues/1065
>
I am not sure how Xiaomi deals with the stacksize and priorities now.
From looking at the implementation in apps/build/exec_builtin it looks
like they might have "fake" built-in tasks that have a stack size and
priority but but a NULL main entry point function. Then the look-up for
the built-in task finds the stack size and priority but no entry point
so it uses that stack size and entry point when loading the ELF file
instead. In this use case the built-in app entry is just used to carry
the stacksize and priority.
Clearly, the binary file itself would be a much more appropriate vehicle
to do that. This could all benefit be a better, cleaner functional
partitioning.
The problem with the original implementation was that it did not try the
ELF binary UNLESS that entry point was NULL. That was an error and
prevented the replacement 'hello' from working because there as an entry
point. PR 258 restored the order before 255 and fixed the logic error
in exec_builtin().
Re: 答复: [External Mail]Re: heap malloc when executing binary/elf file
Posted by Gregory Nutt <sp...@gmail.com>.
> So, in current git HEAD [1], what is the search order now?
>
> Is it back to:
> - Elf
> - Built-in
> - NSH
>
> Or is it:
> - Built-in
> - Elf
> - NSH
It is not so clean and well partitioned. It is more like:
o IF Built-in available
- Elf
- IF NOT Elf
Built-in
o Elf
o NSH
The logic is spread across a couple of files so is not in one place anymore. It is not pretty, but seems like it should work.
The only advantage is in this case:
o IF Built-in available
- Elf
For built-ins, we have task stack size and priority information. For
o Elf
But we do not and have to use a default stack size which will be most certainly wrong. We are contemplating fixing that via https://github.com/apache/incubator-nuttx/issues/1065
Re: 答复: [External Mail]Re: heap malloc when executing binary/elf file
Posted by Nathan Hartman <ha...@gmail.com>.
On Mon, May 18, 2020 at 10:38 AM Gregory Nutt <sp...@gmail.com> wrote:
> >> Yes, the best solution is to extend ELF with stack and priority info.
> >> But before that happen, gathering this info from the built-in app
> >> table is workable alternative:
> >> 1.Keep the trying order(file system, built-in and nsh command)
> >> 2.And get the stack/priority from built-in app table for file system
> >> applicaiton
> >> How about we open a new issue track ELF extension for stack/priority?
> >
> > I will write something. You can fee free to edit it.
> I have added issue https://github.com/apache/incubator-nuttx/issues/1065
> per your request.
So, in current git HEAD [1], what is the search order now?
Is it back to:
- Elf
- Built-in
- NSH
Or is it:
- Built-in
- Elf
- NSH
[1] 8997213c2aca8fae0f5537833014f9e016b51dd2
Thanks
Nathan
Re: 答复: [External Mail]Re: heap malloc when executing binary/elf file
Posted by Gregory Nutt <sp...@gmail.com>.
>
>> Yes, the best solution is to extend ELF with stack and priority info.
>> But before that happen, gathering this info from the built-in app
>> table is workable alternative:
>> 1.Keep the trying order(file system, built-in and nsh command)
>> 2.And get the stack/priority from built-in app table for file system
>> applicaiton
>> How about we open a new issue track ELF extension for stack/priority?
>
> I will write something. You can fee free to edit it.
I have added issue https://github.com/apache/incubator-nuttx/issues/1065
per your request.
Re: 答复: [External Mail]Re: heap malloc when executing binary/elf file
Posted by Gregory Nutt <sp...@gmail.com>.
> Yes, the best solution is to extend ELF with stack and priority info.
> But before that happen, gathering this info from the built-in app
> table is workable alternative:
> 1.Keep the trying order(file system, built-in and nsh command)
> 2.And get the stack/priority from built-in app table for file system applicaiton
> How about we open a new issue track ELF extension for stack/priority?
I will write something. You can fee free to edit it.
Greg
Re: 答复: [External Mail]Re: heap malloc when executing binary/elf file
Posted by Xiang Xiao <xi...@gmail.com>.
On Mon, May 18, 2020 at 9:07 PM Gregory Nutt <sp...@gmail.com> wrote:
>
>
> > OK, by now I'll keep using the alternative to use another name for the
> > elf app/FILE_APPS, as seemingly intended by
> > chao.an in his commit.
> No, that was an error. The requirements have always supported
> replacement of built-in applications with ELF applications. That is a
> recommended way for updating an embedded system. The original PR broke
> that logic.
> > I don't really understand Chaos solution because if the ELF app needs a
> > bigger stack, which is quite probable because it is a
> > derivation/evolution of the built-in command with the same name, you'll
> > have to increase the spawn stack size anyway !?!?
>
> That is true. The assumption is that the stack size of the built-in
> application will be big enough for the replacement ELF application.
> That might not be true. But in the case of replacing a built-in
> application with an ELF application, the rule is that we cannot modify
> the base ROM logic. Imagine that I have 100,000 products in the field
> and I want to replace a built-in. That is pretty easy to do by just
> writing the ELF file to the some local file system. You would like to
> avoid replacing the entire FLASH image.
>
> And if it is not replacing an existing built-in application. Then there
> is no information on stack size. The ELF format does have logic to
> support a stack section, but that is not used by the loader. Other
> formats do include the stack size as part of the header (NXFLAT, for
> example). But otherwise, you have no option but to make the global
> stack size larger for all applications.
>
> We could potentially come up with an extended ELF head that includes
> stack and priority information. I think it could be done because the
> offsets in the ELF header permit the header to be extended. That would
> be a good project.
>
Yes, the best solution is to extend ELF with stack and priority info.
But before that happen, gathering this info from the built-in app
table is workable alternative:
1.Keep the trying order(file system, built-in and nsh command)
2.And get the stack/priority from built-in app table for file system applicaiton
How about we open a new issue track ELF extension for stack/priority?
> > But that's for you pros to decide.
> I merged Xiang's change. Please try the changes in the repository now
> if you can make the time (without changing the name of hello).
> > At least seems you got a communication problem here, because nobody had
> > a notion about the impact of Chaos commit.
>
> Yes, a communication problem, a testing problem, a requirements problem,
> many problems. You can help by contributing some time and effort to
> make things better 8)
>
> Greg
>
>
>
Re: 答复: [External Mail]Re: heap malloc when executing binary/elf file
Posted by Gregory Nutt <sp...@gmail.com>.
> OK, by now I'll keep using the alternative to use another name for the
> elf app/FILE_APPS, as seemingly intended by
> chao.an in his commit.
No, that was an error. The requirements have always supported
replacement of built-in applications with ELF applications. That is a
recommended way for updating an embedded system. The original PR broke
that logic.
> I don't really understand Chaos solution because if the ELF app needs a
> bigger stack, which is quite probable because it is a
> derivation/evolution of the built-in command with the same name, you'll
> have to increase the spawn stack size anyway !?!?
That is true. The assumption is that the stack size of the built-in
application will be big enough for the replacement ELF application.
That might not be true. But in the case of replacing a built-in
application with an ELF application, the rule is that we cannot modify
the base ROM logic. Imagine that I have 100,000 products in the field
and I want to replace a built-in. That is pretty easy to do by just
writing the ELF file to the some local file system. You would like to
avoid replacing the entire FLASH image.
And if it is not replacing an existing built-in application. Then there
is no information on stack size. The ELF format does have logic to
support a stack section, but that is not used by the loader. Other
formats do include the stack size as part of the header (NXFLAT, for
example). But otherwise, you have no option but to make the global
stack size larger for all applications.
We could potentially come up with an extended ELF head that includes
stack and priority information. I think it could be done because the
offsets in the ELF header permit the header to be extended. That would
be a good project.
> But that's for you pros to decide.
I merged Xiang's change. Please try the changes in the repository now
if you can make the time (without changing the name of hello).
> At least seems you got a communication problem here, because nobody had
> a notion about the impact of Chaos commit.
Yes, a communication problem, a testing problem, a requirements problem,
many problems. You can help by contributing some time and effort to
make things better 8)
Greg
Re: 答复: [External Mail]Re: heap malloc when executing binary/elf file
Posted by Florian Wehmeyer <fl...@gmx.net>.
Hi all,
OK, by now I'll keep using the alternative to use another name for the
elf app/FILE_APPS, as seemingly intended by
chao.an in his commit.
I don't really understand Chaos solution because if the ELF app needs a
bigger stack, which is quite probable because it is a
derivation/evolution of the built-in command with the same name, you'll
have to increase the spawn stack size anyway !?!?
But that's for you pros to decide.
At least seems you got a communication problem here, because nobody had
a notion about the impact of Chaos commit.
Thanks to all & BR,
--
Florian Wehmeyer
TFW Tech-Solutions
On Seg, 2020-05-18 at 06:53 +0000, 安超 wrote:
> Hi Greg,
>
> About the break commit:
>
> commit 9a28ccf836c1b9f0eb5e1163964042eddc207697
> Author: chao.an <an...@xiaomi.com>
> Date: Fri Feb 21 09:54:47 2020 +0800
>
> nsh/parse: try the builtin configuration first
>
> In the case of enable the BUILTIN_APPS/FILE_APPS at the same
> time, try the builtin list first to ensure that the relevant
> configuration(stacksize, priority) can be set normally.
>
>
> Yes, the change did not consider the case where the file and builtin
> application have the same name,
> but I think it is too violent to directly revert it ...
>
> The original intention of this change is to solve the problem that
> the builtin attribute
> (priority and stack) cannot take effect when both the file and
> builtin application are enabled.
>
> Think about what should we do if an file application needs a stack
> size larger than spawn?
>
> For example,
> 1. Suppose the program needs a large stack size:
>
> diff --git a/examples/hello/hello_main.c
> b/examples/hello/hello_main.c
> index 953597a..7b832b6 100644
> --- a/examples/hello/hello_main.c
> +++ b/examples/hello/hello_main.c
> @@ -50,6 +50,10 @@
>
> int main(int argc, FAR char *argv[])
> {
> + char dummy_array[CONFIG_EXAMPLES_HELLO_STACKSIZE - 2048];
> +
> + memset(dummy_array, 0, sizeof(dummy_array));
> printf("Hello, World!!\n");
> + sleep(10000);
> return 0;
> }
>
> 2. Check the stack usage, since the default
> CONFIG_EXAMPLES_HELLO_STACKSIZE
> follows CONFIG_TASK_SPAWN_DEFAULT_STACKSIZE are 8192, so there is
> no problem
>
> nsh> Hello, World!!
> ps
> PID PPID PRI POLICY TYPE NPX
> STATE EVENT SIGMASK STACK USED FILLED COMMAND
> 0 0 0 FIFO Kthread N-- Ready 00000000
> 000000 000000 0.0% Idle Task
> 2 1 100 FIFO Task --- Running 00000000
> 008160 002236 27.4% /system/bin/nsh
> 3 2 100 FIFO Task --- Waiting Signal 00000000
> 008176 007240 88.5%! hello
> nsh>
>
> 3. But if we change the stack size to 20480, see what happens:
> +CONFIG_EXAMPLES_HELLO_STACKSIZE=20480
>
> nsh> hello &
> hello [0:100]
> nsh> Hello, World!!
> Segmentation fault (core dumped)
>
> Size 8192 is still used in posix_spawn, which will cause a loadable
> application stack overflow,
> Of course you will say that in this case we can increase the stack
> size of posix_spawn,
> but in ths case, we will waste more stack space for small
> applications.
>
> -------------------------------------------------------------------
> ----------------
>
> To resolve this issue I sent a pull request and made a simple fix:
>
> https://github.com/apache/incubator-nuttx-apps/pull/258
>
> 1. Bring back the BUILTIN_APPS / FILE_APPS calling sequence,
> 2. Try FILE_APPS first in the case of builtin:
>
> After patch:
>
> nsh> hello &
> hello [3:100]
> nsh> Hello, World!!
> ps
> PID PPID PRI POLICY TYPE NPX
> STATE EVENT SIGMASK STACK USED FILLED COMMAND
> 0 0 0 FIFO Kthread N-- Ready 00000000
> 000000 000000 0.0% Idle Task
> 2 1 100 FIFO Task --- Running 00000000
> 008160 002300 28.1% /system/bin/nsh
> 3 2 100 FIFO Task --- Waiting Signal 00000000
> 020464 019528 95.4%! hello
>
> It is make sense to use the builtin attribute for applications if we
> enabled the BUILTIN_APPS / FILE_APPS at the same time.
>
> BRs,
> ________________________________________
> 发件人: Gregory Nutt <sp...@gmail.com>
> 发送时间: 2020年5月18日 6:59
> 收件人: dev@nuttx.apache.org
> 主题: [External Mail]Re: heap malloc when executing binary/elf file
>
> Hi, Brennan,
> >
> > >
> > > >
> > > > I just created incubator-nuttx-apps PR 255 that should restore
> > > > the
> > > > correct behavior:
> > > https://github.com/apache/incubator-nuttx-apps/pull/255
> > >
> > > This is a major bug and, if we release do a bugfix release
> > > sometime,
> > > this should be included.
> > >
> > I was thinking the same. I created a label backport/9.0 that we can
> > attach
> > to PRs that we think should be backported to a bugfix release. I
> > suspect
> > especially with the amount of changes that went into this release
> > it won't
> > be the only one.
> > @btashton btashton added the backport/9.0 label 10 seconds ago
> I agree that this should be backported. It is a major bug; a
> complete
> loss of important functionality.
>
> But let's let is simmer for a bit to make sure that it does the job
> and
> doesn't introduce new issues.
>
> Is this your preferred way of handling bugs? Adding the backport/9.0
> tag? Optionally, we could just add the commits to the tip of the
> releases/9.0 branch where they could be picked up at some later
> bugfix
> release.
>
> I have no preference. At some point, I think we also have to say
> that
> 9.0.x "is what it is". Or do you see us continuing to maintain the
> release forever? In the dark, pre-Apache past, I had about a two
> week
> period after the release after which it "is what it is". I
> distrubuted
> patches up until that point. What policy would you recommend?
>
> Greg
>
>
>
> #/******本邮件及其附件含有小米公司的保密信息，仅限于发送给上面地址中列出的个人或群组。禁止任何其他人以任何形式使用（包括但不限于全
> 部或部分地泄露、复制、或散发）本邮件中的信息。如果您错收了本邮件，请您立即电话或邮件通知发件人并删除本邮件！ This e-mail
> and its attachments contain confidential information from XIAOMI,
> which is intended only for the person or entity whose address is
> listed above. Any use of the information contained herein in any way
> (including, but not limited to, total or partial disclosure,
> reproduction, or dissemination) by persons other than the intended
> recipient(s) is prohibited. If you receive this e-mail in error,
> please notify the sender by phone or email immediately and delete
> it!******/#
答复: [External Mail]Re: heap malloc when executing binary/elf file
Posted by 安超 <an...@xiaomi.com.INVALID>.
Hi Greg,
About the break commit:
commit 9a28ccf836c1b9f0eb5e1163964042eddc207697
Author: chao.an <an...@xiaomi.com>
Date: Fri Feb 21 09:54:47 2020 +0800
nsh/parse: try the builtin configuration first
In the case of enable the BUILTIN_APPS/FILE_APPS at the same
time, try the builtin list first to ensure that the relevant
configuration(stacksize, priority) can be set normally.
Yes, the change did not consider the case where the file and builtin application have the same name,
but I think it is too violent to directly revert it ...
The original intention of this change is to solve the problem that the builtin attribute
(priority and stack) cannot take effect when both the file and builtin application are enabled.
Think about what should we do if an file application needs a stack size larger than spawn?
For example,
1. Suppose the program needs a large stack size:
diff --git a/examples/hello/hello_main.c b/examples/hello/hello_main.c
index 953597a..7b832b6 100644
--- a/examples/hello/hello_main.c
+++ b/examples/hello/hello_main.c
@@ -50,6 +50,10 @@
int main(int argc, FAR char *argv[])
{
+ char dummy_array[CONFIG_EXAMPLES_HELLO_STACKSIZE - 2048];
+
+ memset(dummy_array, 0, sizeof(dummy_array));
printf("Hello, World!!\n");
+ sleep(10000);
return 0;
}
2. Check the stack usage, since the default CONFIG_EXAMPLES_HELLO_STACKSIZE
follows CONFIG_TASK_SPAWN_DEFAULT_STACKSIZE are 8192, so there is no problem
nsh> Hello, World!!
ps
PID PPID PRI POLICY TYPE NPX STATE EVENT SIGMASK STACK USED FILLED COMMAND
0 0 0 FIFO Kthread N-- Ready 00000000 000000 000000 0.0% Idle Task
2 1 100 FIFO Task --- Running 00000000 008160 002236 27.4% /system/bin/nsh
3 2 100 FIFO Task --- Waiting Signal 00000000 008176 007240 88.5%! hello
nsh>
3. But if we change the stack size to 20480, see what happens:
+CONFIG_EXAMPLES_HELLO_STACKSIZE=20480
nsh> hello &
hello [0:100]
nsh> Hello, World!!
Segmentation fault (core dumped)
Size 8192 is still used in posix_spawn, which will cause a loadable application stack overflow,
Of course you will say that in this case we can increase the stack size of posix_spawn,
but in ths case, we will waste more stack space for small applications.
-----------------------------------------------------------------------------------
To resolve this issue I sent a pull request and made a simple fix:
https://github.com/apache/incubator-nuttx-apps/pull/258
1. Bring back the BUILTIN_APPS / FILE_APPS calling sequence,
2. Try FILE_APPS first in the case of builtin:
After patch:
nsh> hello &
hello [3:100]
nsh> Hello, World!!
ps
PID PPID PRI POLICY TYPE NPX STATE EVENT SIGMASK STACK USED FILLED COMMAND
0 0 0 FIFO Kthread N-- Ready 00000000 000000 000000 0.0% Idle Task
2 1 100 FIFO Task --- Running 00000000 008160 002300 28.1% /system/bin/nsh
3 2 100 FIFO Task --- Waiting Signal 00000000 020464 019528 95.4%! hello
It is make sense to use the builtin attribute for applications if we enabled the BUILTIN_APPS / FILE_APPS at the same time.
BRs,
________________________________________
发件人: Gregory Nutt <sp...@gmail.com>
发送时间: 2020年5月18日 6:59
收件人: dev@nuttx.apache.org
主题: [External Mail]Re: heap malloc when executing binary/elf file
Hi, Brennan,
>>> I just created incubator-nuttx-apps PR 255 that should restore the
>>> correct behavior:
>> https://github.com/apache/incubator-nuttx-apps/pull/255
>>
>> This is a major bug and, if we release do a bugfix release sometime,
>> this should be included.
>>
> I was thinking the same. I created a label backport/9.0 that we can attach
> to PRs that we think should be backported to a bugfix release. I suspect
> especially with the amount of changes that went into this release it won't
> be the only one.
> @btashton btashton added the backport/9.0 label 10 seconds ago
I agree that this should be backported. It is a major bug; a complete
loss of important functionality.
But let's let is simmer for a bit to make sure that it does the job and
doesn't introduce new issues.
Is this your preferred way of handling bugs? Adding the backport/9.0
tag? Optionally, we could just add the commits to the tip of the
releases/9.0 branch where they could be picked up at some later bugfix
release.
I have no preference. At some point, I think we also have to say that
9.0.x "is what it is". Or do you see us continuing to maintain the
release forever? In the dark, pre-Apache past, I had about a two week
period after the release after which it "is what it is". I distrubuted
patches up until that point. What policy would you recommend?
Greg
#/******本邮件及其附件含有小米公司的保密信息，仅限于发送给上面地址中列出的个人或群组。禁止任何其他人以任何形式使用（包括但不限于全部或部分地泄露、复制、或散发）本邮件中的信息。如果您错收了本邮件，请您立即电话或邮件通知发件人并删除本邮件！ This e-mail and its attachments contain confidential information from XIAOMI, which is intended only for the person or entity whose address is listed above. Any use of the information contained herein in any way (including, but not limited to, total or partial disclosure, reproduction, or dissemination) by persons other than the intended recipient(s) is prohibited. If you receive this e-mail in error, please notify the sender by phone or email immediately and delete it!******/#
Re: Backporting Fixes [was: heap malloc when executing binary/elf file]
Posted by Nathan Hartman <ha...@gmail.com>.
On Mon, May 18, 2020 at 12:12 PM Gregory Nutt <sp...@gmail.com> wrote:
> I think we need to assure that these two "tiers" are compatible and use
> the same test infrastructure. The should have these things in common:
I'd like to move this discussion to a new thread, titled "Automated
Testing" without several levels of [was: foo [was: bar]]"
Would you be willing to re-post this as a reply to that thread so that
the conversation can continue there?
Thanks,
Nathan
Re: Backporting Fixes [was: heap malloc when executing binary/elf
file]
Posted by Gregory Nutt <sp...@gmail.com>.
> * The next tier could be an automated test suite that runs in a
> simulator.
>
> This would be accessible to the greatest numbers of developers and
> contributors, because there's nothing to buy.
>
> * The highest tier could be testing on a reference hardware board.
I think we need to assure that these two "tiers" are compatible and use
the same test infrastructure. The should have these things in common:
1. Same top-level test infrastructure,
2. Common assessment of test results from the target console output.
Going from the first to the second is essentially replacing the
software-to-software interfaces:
3. Adding a reference board
4. Adding test harness hardware that is controllable by software to
load code, reset the board, collect serial console data.
5. Other hardware plumbing as need for the test.
For example, network testing on the Linux simulator would require use of
the Linux TAP device and some physical network, and test servers/clients
to close the loop on the network tests. The hardware-in-loop case would
connect directly to the physical network, otherwise it would be the same
test.
PX4 has a rack of supported hardware and runs flight qualification tests
on hardware periodically. The scope of testing an OS is not easily
constrained to a series of test like that, but the basic PX4 setup would
be a good general model.
To keep the scope under control, I would avoid low-level testing of all
OS interfaces. Instead I would focus on higher level usage. networking
is a good example. ELF module support is another. USB devices, USB
host, etc.
There is an OS interface test in apps/testing/ostest. Currently, it
does not do an exhaustive test of interfaces. It only verifies basic
functionality of interfaces and should be extended to cover all
interfaces, all interface features, and handling of bad values. The
simulator is perfectly suited for that kind of testing. I would reserve
hardware-based testing for things that are higher level integrations of
capabilities and that are sensitive to race conditions. Hardware-based
testing would be better for stress and duration testing.
Re: Backporting Fixes [was: heap malloc when executing binary/elf file]
Posted by Nathan Hartman <ha...@gmail.com>.
On Mon, May 18, 2020 at 12:03 AM Gregory Nutt <sp...@gmail.com> wrote:
> Build testing is important but does not address functionality. Things
> can be completely broken and still build perfectly.
Correct.
> Currently our functional testing strategy is simply to ask people to
> check releases. But no problems have ever been reported from that so I
> think it is not working. The related strategy is just to wait awhile
> and if no one complains it most be okay. My experience is that bugs
> fester in the repository for weeks or months before the are detected and
> reported.
At the end of the day, no test suite can ever exercise every single
possible codepath, particularly when dealing with concurrency issues.
This is not necessarily a "problem," it's just the reality of software
testing. That is why we depend on downstream users and developers to
participate in the project by sharing their experiences with us on the
mailing lists. Also, other users out there will put the code through
its paces by using it differently than the NuttX devs might have
considered. So feedback from users is crucial. I think that as a
project, we need to push for more participation by downstream
stakeholders to test our release candidates during the soak period in
their own applications and communicate their findings with the
project. I think this can also help increase participation in general.
Having said all of this:
> Is this something we want to invest in? Understanding that it would be
> a long term investment.
If we want to be a serious contender in the RTOS space, I think the
answer is yes.
Yes, it would be a long-term effort. It won't happen overnight. So
before anyone starts banging out testing code, I think we need to have
a discussion as a community and come up with a realistic strategy to
gradually build up a test suite, along with determining the premise of
how that test suite works.
On that point:
> Xiao Xiang was suggested some automated testing based on the simulator.
> At other times in the past we have talked about developing a test suite
> around some reference hardware board. Those discussions did not go anywhere.
We may need to have several tiers of testing.
* The lowest tier could be coding standard and build testing. (Which
we're already doing.)
* The next tier could be static analysis of the code.
There are free analyzers, as well as cloud based commercial ones
that are free for open source projects to use.
* The next tier could be an automated test suite that runs in a
simulator.
This would be accessible to the greatest numbers of developers and
contributors, because there's nothing to buy.
* The highest tier could be testing on a reference hardware board.
We could begin by perfecting the lowest tier and then gradually
climbing up the ladder by adding static analysis, etc.
About discussions not going anywhere, that's bound to happen in any
project run by volunteers, but as long as we're watchful and revive
those discussions from time to time, something will come of it.
So, let's talk about it!
Nathan
Re: Backporting Fixes [was: heap malloc when executing binary/elf
file]
Posted by Gregory Nutt <sp...@gmail.com>.
>>> I have no preference. At some point, I think we also have to say that
>>> 9.0.x "is what it is". Or do you see us continuing to maintain the
>>> release forever? In the dark, pre-Apache past, I had about a two week
>>> period after the release after which it "is what it is". I distrubuted
>>> patches up until that point. What policy would you recommend?
>>>
>>> My thought here is that we collect PRs that are candidates for backporting
>>> with the label and then we can apply them if they look safe and are
>>> important enough (let them bake in master for a bit). I don't think we
>>> have ever really had the concept of a Long Term Support release nor are we
>>> ready for that so I'm thinking we use judgement and try to get back to a
>>> 1-2mo release window with maybe room for 1 or 2 bug fix releases inside
>>> that if needed.
> That sounds about right to me.
>
> In the future we should think about LTS releases, but we're not ready for
> that yet. Too many big things to sort out first. (But we're making
> progress.)
Ideally, we should have some rather exhaustive, functional test suite to
qualify releases. But I understand that such extensive testing would be
prohibitively expensive. I have also said that testing is something you
have to buy by the pound. Buy as much as you afford.
Build testing is important but does not address functionality. Things
can be completely broken and still build perfectly.
Currently our functional testing strategy is simply to ask people to
check releases. But no problems have ever been reported from that so I
think it is not working. The related strategy is just to wait awhile
and if no one complains it most be okay. My experience is that bugs
fester in the repository for weeks or months before the are detected and
reported.
Is this something we want to invest in? Understanding that it would be
a long term investment.
Xiao Xiang was suggested some automated testing based on the simulator.
At other times in the past we have talked about developing a test suite
around some reference hardware board. Those discussions did not go anywhere.
Greg
Re: Backporting Fixes [was: heap malloc when executing binary/elf file]
Posted by Nathan Hartman <ha...@gmail.com>.
On Sun, May 17, 2020 at 7:11 PM Brennan Ashton <ba...@brennanashton.com>
wrote:
> On Sun, May 17, 2020, 3:59 PM Gregory Nutt <sp...@gmail.com> wrote:
>
> > Hi, Brennan,
> > >>> I just created incubator-nuttx-apps PR 255 that should restore the
> > >>> correct behavior:
> > >> https://github.com/apache/incubator-nuttx-apps/pull/255
> > >>
> > >> This is a major bug and, if we release do a bugfix release sometime,
> > >> this should be included.
> > >>
> > > I was thinking the same. I created a label backport/9.0 that we can
> > attach
> > > to PRs that we think should be backported to a bugfix release. I
> suspect
> > > especially with the amount of changes that went into this release it
> > won't
> > > be the only one.
> > > @btashton btashton added the backport/9.0 label 10 seconds ago
> >
> > I agree that this should be backported. It is a major bug; a complete
> > loss of important functionality.
> >
> > But let's let is simmer for a bit to make sure that it does the job and
> > doesn't introduce new issues.
> >
> > Is this your preferred way of handling bugs? Adding the backport/9.0
> > tag? Optionally, we could just add the commits to the tip of the
> > releases/9.0 branch where they could be picked up at some later bugfix
> > release.
> >
> > I have no preference. At some point, I think we also have to say that
> > 9.0.x "is what it is". Or do you see us continuing to maintain the
> > release forever? In the dark, pre-Apache past, I had about a two week
> > period after the release after which it "is what it is". I distrubuted
> > patches up until that point. What policy would you recommend?
> >
> > Greg
> >
>
> My thought here is that we collect PRs that are candidates for backporting
> with the label and then we can apply them if they look safe and are
> important enough (let them bake in master for a bit). I don't think we
> have ever really had the concept of a Long Term Support release nor are we
> ready for that so I'm thinking we use judgement and try to get back to a
> 1-2mo release window with maybe room for 1 or 2 bug fix releases inside
> that if needed.
That sounds about right to me.
In the future we should think about LTS releases, but we're not ready for
that yet. Too many big things to sort out first. (But we're making
progress.)
Nathan
Backporting Fixes [was: heap malloc when executing binary/elf file]
Posted by Brennan Ashton <ba...@brennanashton.com>.
On Sun, May 17, 2020, 3:59 PM Gregory Nutt <sp...@gmail.com> wrote:
> Hi, Brennan,
> >>> I just created incubator-nuttx-apps PR 255 that should restore the
> >>> correct behavior:
> >> https://github.com/apache/incubator-nuttx-apps/pull/255
> >>
> >> This is a major bug and, if we release do a bugfix release sometime,
> >> this should be included.
> >>
> > I was thinking the same. I created a label backport/9.0 that we can
> attach
> > to PRs that we think should be backported to a bugfix release. I suspect
> > especially with the amount of changes that went into this release it
> won't
> > be the only one.
> > @btashton btashton added the backport/9.0 label 10 seconds ago
>
> I agree that this should be backported. It is a major bug; a complete
> loss of important functionality.
>
> But let's let is simmer for a bit to make sure that it does the job and
> doesn't introduce new issues.
>
> Is this your preferred way of handling bugs? Adding the backport/9.0
> tag? Optionally, we could just add the commits to the tip of the
> releases/9.0 branch where they could be picked up at some later bugfix
> release.
>
> I have no preference. At some point, I think we also have to say that
> 9.0.x "is what it is". Or do you see us continuing to maintain the
> release forever? In the dark, pre-Apache past, I had about a two week
> period after the release after which it "is what it is". I distrubuted
> patches up until that point. What policy would you recommend?
>
> Greg
>
My thought here is that we collect PRs that are candidates for backporting
with the label and then we can apply them if they look safe and are
important enough (let them bake in master for a bit). I don't think we
have ever really had the concept of a Long Term Support release nor are we
ready for that so I'm thinking we use judgement and try to get back to a
1-2mo release window with maybe room for 1 or 2 bug fix releases inside
that if needed.
--Brennan
Re: heap malloc when executing binary/elf file
Posted by Gregory Nutt <sp...@gmail.com>.
Hi, Brennan,
>>> I just created incubator-nuttx-apps PR 255 that should restore the
>>> correct behavior:
>> https://github.com/apache/incubator-nuttx-apps/pull/255
>>
>> This is a major bug and, if we release do a bugfix release sometime,
>> this should be included.
>>
> I was thinking the same. I created a label backport/9.0 that we can attach
> to PRs that we think should be backported to a bugfix release. I suspect
> especially with the amount of changes that went into this release it won't
> be the only one.
> @btashton btashton added the backport/9.0 label 10 seconds ago
I agree that this should be backported. It is a major bug; a complete
loss of important functionality.
But let's let is simmer for a bit to make sure that it does the job and
doesn't introduce new issues.
Is this your preferred way of handling bugs? Adding the backport/9.0
tag? Optionally, we could just add the commits to the tip of the
releases/9.0 branch where they could be picked up at some later bugfix
release.
I have no preference. At some point, I think we also have to say that
9.0.x "is what it is". Or do you see us continuing to maintain the
release forever? In the dark, pre-Apache past, I had about a two week
period after the release after which it "is what it is". I distrubuted
patches up until that point. What policy would you recommend?
Greg
Re: heap malloc when executing binary/elf file
Posted by Brennan Ashton <ba...@brennanashton.com>.
On Sun, May 17, 2020, 3:50 PM Gregory Nutt <sp...@gmail.com> wrote:
>
> > I just created incubator-nuttx-apps PR 255 that should restore the
> > correct behavior:
> https://github.com/apache/incubator-nuttx-apps/pull/255
>
> This is a major bug and, if we release do a bugfix release sometime,
> this should be included.
>
I was thinking the same. I created a label backport/9.0 that we can attach
to PRs that we think should be backported to a bugfix release. I suspect
especially with the amount of changes that went into this release it won't
be the only one.
--Brennan
>
Re: heap malloc when executing binary/elf file
Posted by Gregory Nutt <sp...@gmail.com>.
> I just created incubator-nuttx-apps PR 255 that should restore the
> correct behavior: https://github.com/apache/incubator-nuttx-apps/pull/255
This is a major bug and, if we release do a bugfix release sometime,
this should be included.
Re: heap malloc when executing binary/elf file
Posted by Gregory Nutt <sp...@gmail.com>.
I just created incubator-nuttx-apps PR 255 that should restore the
correct behavior: https://github.com/apache/incubator-nuttx-apps/pull/255
On 5/17/2020 3:33 PM, Gregory Nutt wrote:
> Mystery solved.
>>>> It does not even try to load the elf file from file-system, it
>>>> directly
>>>> goes to the built-in app.
>>> That means that it failed to load the app on the file system. It
>>> should
>>> check in this order"
>>>
>>> - Load file from file system if possible
>>>
>>> - If not, run built-in application if possible
>>>
>>> - If not, run NSH command line command if possible
>>>
>>> - If not, report that the command was not found
>>>
>>> so if you are seeing the second action, it means that the first
>>> failed. You can load FS and Binary loader error debug output to see.
>> Yes Greg, got that, I already enabled all binary loader debug output
>> (warnings+info).
>>
>> I'll post my console output below, were you can see that the first call
>> of hello, there seems to be no attempt at all to call the elf file.. it
>> seems exec_builtin() is calling directly task_spawn() which calls the
>> built-in app..
>
> The correct behavior of NSH was broken by this PR in apps:
>
> commit 9a28ccf836c1b9f0eb5e1163964042eddc207697
> Author: chao.an <an...@xiaomi.com>
> Date: Fri Feb 21 09:54:47 2020 +0800
>
> nsh/parse: try the builtin configuration first
>
> In the case of enable the BUILTIN_APPS/FILE_APPS at the same
> time, try the builtin list first to ensure that the relevant
> configuration(stacksize, priority) can be set normally.
>
> Because of that bad change, you will no longer be able to replace
> built in applications with applications on a file system. So sad.
> That change ruined one of the most important features of the ELF loader.
>
> You can back that out and try again. That really should be fixed. If
> you get it working submit a PR for discussion.
>
> Greg
>
>
>
Re: heap malloc when executing binary/elf file
Posted by Gregory Nutt <sp...@gmail.com>.
Mystery solved.
>>> It does not even try to load the elf file from file-system, it
>>> directly
>>> goes to the built-in app.
>> That means that it failed to load the app on the file system. It
>> should
>> check in this order"
>>
>> - Load file from file system if possible
>>
>> - If not, run built-in application if possible
>>
>> - If not, run NSH command line command if possible
>>
>> - If not, report that the command was not found
>>
>> so if you are seeing the second action, it means that the first
>> failed. You can load FS and Binary loader error debug output to see.
> Yes Greg, got that, I already enabled all binary loader debug output
> (warnings+info).
>
> I'll post my console output below, were you can see that the first call
> of hello, there seems to be no attempt at all to call the elf file.. it
> seems exec_builtin() is calling directly task_spawn() which calls the
> built-in app..
The correct behavior of NSH was broken by this PR in apps:
commit 9a28ccf836c1b9f0eb5e1163964042eddc207697
Author: chao.an <an...@xiaomi.com>
Date: Fri Feb 21 09:54:47 2020 +0800
nsh/parse: try the builtin configuration first
In the case of enable the BUILTIN_APPS/FILE_APPS at the same
time, try the builtin list first to ensure that the relevant
configuration(stacksize, priority) can be set normally.
Because of that bad change, you will no longer be able to replace built
in applications with applications on a file system. So sad. That
change ruined one of the most important features of the ELF loader.
You can back that out and try again. That really should be fixed. If
you get it working submit a PR for discussion.
Greg
Re: heap malloc when executing binary/elf file
Posted by Florian Wehmeyer <fl...@gmx.net>.
Hi Greg and Alan,
> > It does not even try to load the elf file from file-system, it
> > directly
> > goes to the built-in app.
> That means that it failed to load the app on the file system. It
> should
> check in this order"
>
> - Load file from file system if possible
>
> - If not, run built-in application if possible
>
> - If not, run NSH command line command if possible
>
> - If not, report that the command was not found
>
> so if you are seeing the second action, it means that the first
> failed. You can load FS and Binary loader error debug output to see.
Yes Greg, got that, I already enabled all binary loader debug output
(warnings+info).
I'll post my console output below, were you can see that the first call
of hello, there seems to be no attempt at all to call the elf file.. it
seems exec_builtin() is calling directly task_spawn() which calls the
built-in app..
Then I just download and call renamed app (from hello to hello2), and the binary loader works fine:
nsh> dmesg
elf_initialize: Registering ELF
exec_spawn: ERROR: Failed to load program 'ls':
2
exec_spawn: ERROR: Failed to load program 'ls': 2
exec_spawn: ERROR: Failed to load program 'dmesg': 2
nsh>ls
/mnt/ramdisk: hello
nsh>hello
Hello, World!! nsh>dmesg
exec_spawn: ERROR: Failed to load program 'ls': 2
exec_spawn: ERROR: Failed to load program 'dmesg': 2
nsh>
nsh>wget http://192.168.1.115:8000/hello2
nsh>ls
/mnt/ramdisk: hello hello2
nsh>hello2
Hello, World ELF!!
nsh> dmesg
exec_spawn: ERROR: Failed to load program 'wget': 2
exec_spawn: ERROR: Failed to load program 'ls': 2
load_absmodule: Loading /mnt/ramdisk/hello2
elf_loadbinary: Loading file: /mnt/ramdisk/hello2
.........
It seems something internally of the nsh is not configured right,
something like symbol table (although I'm working without symbol
tables..)
Tried that with 9.0.0 release branch and also with the latest master
branch.. using hello app to have the most simple case..
Using the original defconfig, just adding the elf and the debug stuff..
Will go on testing, maybe first make a shell script to automatize all
those steps (make export, make elf, etc..), fatigue danger :-D
This is something which has worked well before..
Thanks & BR,
Florian
> > >
> > Funnily, it does try to load the nsh embedded tools:
> > exec_spawn: ERROR: Failed to load program 'ls':
> > -2
> yes, this is normal behavior. Se you can see from the steps above,
> ALL
> commands will try to load from the various locations and you will
> see
> the errors. But they are not "real" errors.
> >
> > If I do not compile the app as a built-in app, the binary loader
> > will
> > load it:
> > load_absmodule: Loading
> > /mnt/ramdisk/Neo_Ledtest
> > elf_loadbinary: Loading file:
> > /mnt/ramdisk/Neo_Ledtest
> > elf_init: filename: /mnt/ramdisk/Neo_Ledtest loadinfo: 0x2000a4cc
> > Which shows that the CONFIG_BINFMT_EXEPATH=y and
> > CONFIG_PATH_INITIAL ar
> > e correctly set, and the elf file is also working correctly..
> > I never had this situation before...
> It shouldn't work that way
Re: heap malloc when executing binary/elf file
Posted by Gregory Nutt <sp...@gmail.com>.
> However, got a new problem now which I've not figured out yet:
> The binary loader does not load the elf if the program is a built-in
> app... very strange..
> It does not even try to load the elf file from file-system, it directly
> goes to the built-in app.
That means that it failed to load the app on the file system. It should
check in this order"
- Load file from file system if possible
- If not, run built-in application if possible
- If not, run NSH command line command if possible
- If not, report that the command was not found
so if you are seeing the second action, it means that the first
failed. You can load FS and Binary loader error debug output to see.
> Funnily, it does try to load the nsh embedded tools:
> exec_spawn: ERROR: Failed to load program 'ls':
> -2
yes, this is normal behavior. Se you can see from the steps above, ALL
commands will try to load from the various locations and you will see
the errors. But they are not "real" errors.
> If I do not compile the app as a built-in app, the binary loader will
> load it:
> load_absmodule: Loading
> /mnt/ramdisk/Neo_Ledtest
> elf_loadbinary: Loading file:
> /mnt/ramdisk/Neo_Ledtest
> elf_init: filename: /mnt/ramdisk/Neo_Ledtest loadinfo: 0x2000a4cc
> Which shows that the CONFIG_BINFMT_EXEPATH=y and CONFIG_PATH_INITIAL ar
> e correctly set, and the elf file is also working correctly..
> I never had this situation before...
It shouldn't work that way
Re: heap malloc when executing binary/elf file
Posted by Florian Wehmeyer <fl...@gmx.net>.
Hi Alan,
updated it again to 9.0.0 and also good!
However, got a new problem now which I've not figured out yet:
The binary loader does not load the elf if the program is a built-in
app... very strange..
It does not even try to load the elf file from file-system, it directly
goes to the built-in app..
Funnily, it does try to load the nsh embedded tools:
exec_spawn: ERROR: Failed to load program 'ls':
-2
exec_spawn: ERROR: Failed to load program 'dmesg': -2
If I do not compile the app as a built-in app, the binary loader will
load it:
load_absmodule: Loading
/mnt/ramdisk/Neo_Ledtest
elf_loadbinary: Loading file:
/mnt/ramdisk/Neo_Ledtest
elf_init: filename: /mnt/ramdisk/Neo_Ledtest loadinfo: 0x2000a4cc
Which shows that the CONFIG_BINFMT_EXEPATH=y and CONFIG_PATH_INITIAL ar
e correctly set, and the elf file is also working correctly..
I never had this situation before...
BR,
Florian
--
Florian Wehmeyer
TFW Tech-Solutions
On Sáb, 2020-05-16 at 16:08 -0300, Alan Carvalho de Assis wrote:
> Hi Florian,
>
> Nice to know you got if fixed.
>
> It should be nice if you can test it again using the nuttx-9.0.0
> release.
>
> BR,
>
> Alan
>
> On 5/16/20, Florian Wehmeyer <fl...@gmx.net> wrote:
> >
> > Hi all,
> > just to let you know, the heap problem was solved after pulling the
> > latest commit from master..
> > Now I can use very much of it, also I was able to use a lot more
> > features in the config, as cmd line history etc...
> > So, the available memory of 256K was not the problem, @Greg.
> > @Nathan: now malloc returns NULL if out of heap..
> > Sorry for that, guys, normally someone should make sure to have the
> > latest version, not a version from 4 weeks ago as I.
> > What happens is, as there is a lot of changes going on constantly,
> > sometimes I hesitate to make frequent pushes, because it can mess
> > up my
> > processes. (for example if someone changes the nuttx folder
> > structure
> > or similar)
> > Very good that now there are release branches, I hope the release
> > branch will suffer only bug-fixes as changes..
> > Thanks again,
> >
> > --
> > Florian Wehmeyer
> > TFW Tech-Solutions
> > On Seg, 2020-05-11 at 18:43 -0300, Florian Wehmeyer wrote:
> > >
> > > OK Greg,
> > > could be that I was using too much.. using 64KB of ramdisk also..
> > > I
> > > disabled some of the comfortable features like cmd line history
> > > etc,
> > > and was already able to alloc quite a bit more than before..
> > > Will further improve the sram usage..
> > > Also, I missed the commit
> > > mm/mm_heap: fix mm_heap not support BUILD_FLAT
> > >
> > >
> > >
> > > GUIDINGLI
> > >
> > >
> > > authored and patacongo
> > >
> > > committed
> > > on Apr 10
> > >
> > >
> > >
> > > which well might impact my stuff here, will pull and let you
> > > know,
> > > @Nathan I will also make tests if malloc continues returning non
> > > NULL
> > > pointers even if out of memory..
> > >
> > > Seems on a good way!!
> > >
> > > Many thanks,
> > > Florian
> > >
> > >
> > >
> > > --
> > > Florian Wehmeyer
> > > TFW Tech-Solutions
> > > On Seg, 2020-05-11 at 15:09 -0600, Gregory Nutt wrote:
> > > >
> > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > No custom board, it's the tm4c1294-launchpad.
> > > > > > Seems no general problem with heap allocation, so I guess
> > > > > > it's
> > > > > > not in
> > > > > > the linker script, it's rather directly linked to the usage
> > > > > > of
> > > > > > the elf-
> > > > > > binary loader, and there are only two configs in the nuttx
> > > > > > repo
> > > > > > which
> > > > > > use the CONFIG_ELF stuff.
> > > > > That part has 256Kb of SRAM. That is more that many, but it
> > > > > could
> > > > > be
> > > > > that you are using too much SRAM and just cannot run ELF
> > > > > reliably.
> > > > > It
> > > > > does want a lot of SRAM.
> > > > Most of the other boards that use CONFIG_ELF=y in a
> > > > configuration
> > > > have a
> > > > much larger internal ram (exceptions in red):
> > > >
> > > > $ find . -name defconfig | xargs grep -l CONFIG_ELF=y | xargs
> > > > grep
> > > > RAM_SIZE
> > > > ./arm/cxd56xx/spresense/configs/elf/defconfig:CONFIG_RAM_SIZE=1
> > > > 5728
> > > > 64
> > > > ./arm/cxd56xx/spresense/configs/module/defconfig:CONFIG_RAM_SIZ
> > > > E=15
> > > > 72
> > > > 864
> > > > ./arm/cxd56xx/spresense/configs/posix_spawn/defconfig:CONFIG_RA
> > > > M_SI
> > > > ZE
> > > > =1572864
> > > > ./arm/cxd56xx/spresense/configs/rndis/defconfig:CONFIG_RAM_SIZE
> > > > =157
> > > > 28
> > > > 64
> > > > ./arm/cxd56xx/spresense/configs/wifi/defconfig:CONFIG_RAM_SIZE=
> > > > 1572
> > > > 86
> > > > 4
> > > > ./arm/lc823450/lc823450-
> > > > xgevk/configs/elf/defconfig:CONFIG_RAM_SIZE=1044480
> > > > ./arm/lc823450/lc823450-
> > > > xgevk/configs/krndis/defconfig:CONFIG_RAM_SIZE=1044480
> > > > ./arm/lc823450/lc823450-
> > > > xgevk/configs/posix_spawn/defconfig:CONFIG_RAM_SIZE=1044480
> > > > ./arm/lc823450/lc823450-
> > > > xgevk/configs/rndis/defconfig:CONFIG_RAM_SIZE=1044480
> > > > ./arm/lc823450/lc823450-
> > > > xgevk/configs/usb/defconfig:CONFIG_RAM_SIZE=1044480
> > > > ./arm/lpc17xx_40xx/lx_cpu/configs/nsh/defconfig:CONFIG_RAM_SIZE
> > > > =655
> > > > 36
> > > > ./arm/sama5/sama5d4-
> > > > ek/configs/elf/defconfig:CONFIG_RAM_SIZE=268435456
> > > > ./arm/sama5/sama5d4-
> > > > ek/configs/knsh/defconfig:CONFIG_RAM_SIZE=268435456
> > > > ./arm/stm32/olimex-stm32-
> > > > p407/configs/kelf/defconfig:CONFIG_RAM_SIZE=114688
> > > > ./arm/stm32/stm32f4discovery/configs/elf/defconfig:CONFIG_RAM_S
> > > > IZE=
> > > > 11
> > > > 4688
> > > > ./arm/stm32/stm32f4discovery/configs/posix_spawn/defconfig:CONF
> > > > IG_R
> > > > AM
> > > > _SIZE=114688
> > > > ./arm/stm32/stm32f4discovery/configs/rndis/defconfig:CONFIG_RAM
> > > > _SIZ
> > > > E=
> > > > 114688
> > > > ./arm/tms570/tms570ls31x-usb-
> > > > kit/configs/nsh/defconfig:CONFIG_RAM_SIZE=262143
> > > > ./risc-v/k210/maix-
> > > > bit/configs/elf/defconfig:CONFIG_RAM_SIZE=2097152
> > > > ./risc-v/k210/maix-
> > > > bit/configs/module/defconfig:CONFIG_RAM_SIZE=2097152
> > > > ./risc-v/k210/maix-
> > > > bit/configs/posix_spawn/defconfig:CONFIG_RAM_SIZE=2097152
> > > >
> > > > And one of these exceptions has a large external DRAM:
> > > >
> > > > $ find . -name defconfig | xargs grep -l CONFIG_ELF=y | xargs
> > > > grep
> > > > EXTDRAM
> > > > ./arm/lpc17xx_40xx/lx_cpu/configs/nsh/defconfig:CONFIG_LPC17_40
> > > > _EXT
> > > > DR
> > > > AM=y
> > > > ./arm/lpc17xx_40xx/lx_cpu/configs/nsh/defconfig:CONFIG_LPC17_40
> > > > _EXT
> > > > DR
> > > > AMSIZE=33554432
> > > >
> > > >
Re: heap malloc when executing binary/elf file
Posted by Alan Carvalho de Assis <ac...@gmail.com>.
Hi Florian,
Nice to know you got if fixed.
It should be nice if you can test it again using the nuttx-9.0.0 release.
BR,
Alan
On 5/16/20, Florian Wehmeyer <fl...@gmx.net> wrote:
> Hi all,
> just to let you know, the heap problem was solved after pulling the
> latest commit from master..
> Now I can use very much of it, also I was able to use a lot more
> features in the config, as cmd line history etc...
> So, the available memory of 256K was not the problem, @Greg.
> @Nathan: now malloc returns NULL if out of heap..
> Sorry for that, guys, normally someone should make sure to have the
> latest version, not a version from 4 weeks ago as I.
> What happens is, as there is a lot of changes going on constantly,
> sometimes I hesitate to make frequent pushes, because it can mess up my
> processes. (for example if someone changes the nuttx folder structure
> or similar)
> Very good that now there are release branches, I hope the release
> branch will suffer only bug-fixes as changes..
> Thanks again,
>
> --
> Florian Wehmeyer
> TFW Tech-Solutions
> On Seg, 2020-05-11 at 18:43 -0300, Florian Wehmeyer wrote:
>> OK Greg,
>> could be that I was using too much.. using 64KB of ramdisk also.. I
>> disabled some of the comfortable features like cmd line history etc,
>> and was already able to alloc quite a bit more than before..
>> Will further improve the sram usage..
>> Also, I missed the commit
>> mm/mm_heap: fix mm_heap not support BUILD_FLAT
>>
>>
>>
>> GUIDINGLI
>>
>>
>> authored and patacongo
>>
>> committed
>> on Apr 10
>>
>>
>>
>> which well might impact my stuff here, will pull and let you know,
>> @Nathan I will also make tests if malloc continues returning non NULL
>> pointers even if out of memory..
>>
>> Seems on a good way!!
>>
>> Many thanks,
>> Florian
>>
>>
>>
>> --
>> Florian Wehmeyer
>> TFW Tech-Solutions
>> On Seg, 2020-05-11 at 15:09 -0600, Gregory Nutt wrote:
>> >
>> > >
>> > >
>> > >
>> > > >
>> > > >
>> > > > No custom board, it's the tm4c1294-launchpad.
>> > > > Seems no general problem with heap allocation, so I guess it's
>> > > > not in
>> > > > the linker script, it's rather directly linked to the usage of
>> > > > the elf-
>> > > > binary loader, and there are only two configs in the nuttx repo
>> > > > which
>> > > > use the CONFIG_ELF stuff.
>> > > That part has 256Kb of SRAM. That is more that many, but it
>> > > could
>> > > be
>> > > that you are using too much SRAM and just cannot run ELF
>> > > reliably.
>> > > It
>> > > does want a lot of SRAM.
>> > Most of the other boards that use CONFIG_ELF=y in a configuration
>> > have a
>> > much larger internal ram (exceptions in red):
>> >
>> > $ find . -name defconfig | xargs grep -l CONFIG_ELF=y | xargs grep
>> > RAM_SIZE
>> > ./arm/cxd56xx/spresense/configs/elf/defconfig:CONFIG_RAM_SIZE=15728
>> > 64
>> > ./arm/cxd56xx/spresense/configs/module/defconfig:CONFIG_RAM_SIZE=15
>> > 72
>> > 864
>> > ./arm/cxd56xx/spresense/configs/posix_spawn/defconfig:CONFIG_RAM_SI
>> > ZE
>> > =1572864
>> > ./arm/cxd56xx/spresense/configs/rndis/defconfig:CONFIG_RAM_SIZE=157
>> > 28
>> > 64
>> > ./arm/cxd56xx/spresense/configs/wifi/defconfig:CONFIG_RAM_SIZE=1572
>> > 86
>> > 4
>> > ./arm/lc823450/lc823450-
>> > xgevk/configs/elf/defconfig:CONFIG_RAM_SIZE=1044480
>> > ./arm/lc823450/lc823450-
>> > xgevk/configs/krndis/defconfig:CONFIG_RAM_SIZE=1044480
>> > ./arm/lc823450/lc823450-
>> > xgevk/configs/posix_spawn/defconfig:CONFIG_RAM_SIZE=1044480
>> > ./arm/lc823450/lc823450-
>> > xgevk/configs/rndis/defconfig:CONFIG_RAM_SIZE=1044480
>> > ./arm/lc823450/lc823450-
>> > xgevk/configs/usb/defconfig:CONFIG_RAM_SIZE=1044480
>> > ./arm/lpc17xx_40xx/lx_cpu/configs/nsh/defconfig:CONFIG_RAM_SIZE=655
>> > 36
>> > ./arm/sama5/sama5d4-
>> > ek/configs/elf/defconfig:CONFIG_RAM_SIZE=268435456
>> > ./arm/sama5/sama5d4-
>> > ek/configs/knsh/defconfig:CONFIG_RAM_SIZE=268435456
>> > ./arm/stm32/olimex-stm32-
>> > p407/configs/kelf/defconfig:CONFIG_RAM_SIZE=114688
>> > ./arm/stm32/stm32f4discovery/configs/elf/defconfig:CONFIG_RAM_SIZE=
>> > 11
>> > 4688
>> > ./arm/stm32/stm32f4discovery/configs/posix_spawn/defconfig:CONFIG_R
>> > AM
>> > _SIZE=114688
>> > ./arm/stm32/stm32f4discovery/configs/rndis/defconfig:CONFIG_RAM_SIZ
>> > E=
>> > 114688
>> > ./arm/tms570/tms570ls31x-usb-
>> > kit/configs/nsh/defconfig:CONFIG_RAM_SIZE=262143
>> > ./risc-v/k210/maix-
>> > bit/configs/elf/defconfig:CONFIG_RAM_SIZE=2097152
>> > ./risc-v/k210/maix-
>> > bit/configs/module/defconfig:CONFIG_RAM_SIZE=2097152
>> > ./risc-v/k210/maix-
>> > bit/configs/posix_spawn/defconfig:CONFIG_RAM_SIZE=2097152
>> >
>> > And one of these exceptions has a large external DRAM:
>> >
>> > $ find . -name defconfig | xargs grep -l CONFIG_ELF=y | xargs grep
>> > EXTDRAM
>> > ./arm/lpc17xx_40xx/lx_cpu/configs/nsh/defconfig:CONFIG_LPC17_40_EXT
>> > DR
>> > AM=y
>> > ./arm/lpc17xx_40xx/lx_cpu/configs/nsh/defconfig:CONFIG_LPC17_40_EXT
>> > DR
>> > AMSIZE=33554432
>> >
>> >
>
Re: heap malloc when executing binary/elf file
Posted by Florian Wehmeyer <fl...@gmx.net>.
Hi all,
just to let you know, the heap problem was solved after pulling the
latest commit from master..
Now I can use very much of it, also I was able to use a lot more
features in the config, as cmd line history etc...
So, the available memory of 256K was not the problem, @Greg.
@Nathan: now malloc returns NULL if out of heap..
Sorry for that, guys, normally someone should make sure to have the
latest version, not a version from 4 weeks ago as I.
What happens is, as there is a lot of changes going on constantly,
sometimes I hesitate to make frequent pushes, because it can mess up my
processes. (for example if someone changes the nuttx folder structure
or similar)
Very good that now there are release branches, I hope the release
branch will suffer only bug-fixes as changes..
Thanks again,
--
Florian Wehmeyer
TFW Tech-Solutions
On Seg, 2020-05-11 at 18:43 -0300, Florian Wehmeyer wrote:
> OK Greg,
> could be that I was using too much.. using 64KB of ramdisk also.. I
> disabled some of the comfortable features like cmd line history etc,
> and was already able to alloc quite a bit more than before..
> Will further improve the sram usage..
> Also, I missed the commit
> mm/mm_heap: fix mm_heap not support BUILD_FLAT
>
>
>
> GUIDINGLI
>
>
> authored and patacongo
>
> committed
> on Apr 10
>
>
>
> which well might impact my stuff here, will pull and let you know,
> @Nathan I will also make tests if malloc continues returning non NULL
> pointers even if out of memory..
>
> Seems on a good way!!
>
> Many thanks,
> Florian
>
>
>
> --
> Florian Wehmeyer
> TFW Tech-Solutions
> On Seg, 2020-05-11 at 15:09 -0600, Gregory Nutt wrote:
> >
> > >
> > >
> > >
> > > >
> > > >
> > > > No custom board, it's the tm4c1294-launchpad.
> > > > Seems no general problem with heap allocation, so I guess it's
> > > > not in
> > > > the linker script, it's rather directly linked to the usage of
> > > > the elf-
> > > > binary loader, and there are only two configs in the nuttx repo
> > > > which
> > > > use the CONFIG_ELF stuff.
> > > That part has 256Kb of SRAM. That is more that many, but it
> > > could
> > > be
> > > that you are using too much SRAM and just cannot run ELF
> > > reliably.
> > > It
> > > does want a lot of SRAM.
> > Most of the other boards that use CONFIG_ELF=y in a configuration
> > have a
> > much larger internal ram (exceptions in red):
> >
> > $ find . -name defconfig | xargs grep -l CONFIG_ELF=y | xargs grep
> > RAM_SIZE
> > ./arm/cxd56xx/spresense/configs/elf/defconfig:CONFIG_RAM_SIZE=15728
> > 64
> > ./arm/cxd56xx/spresense/configs/module/defconfig:CONFIG_RAM_SIZE=15
> > 72
> > 864
> > ./arm/cxd56xx/spresense/configs/posix_spawn/defconfig:CONFIG_RAM_SI
> > ZE
> > =1572864
> > ./arm/cxd56xx/spresense/configs/rndis/defconfig:CONFIG_RAM_SIZE=157
> > 28
> > 64
> > ./arm/cxd56xx/spresense/configs/wifi/defconfig:CONFIG_RAM_SIZE=1572
> > 86
> > 4
> > ./arm/lc823450/lc823450-
> > xgevk/configs/elf/defconfig:CONFIG_RAM_SIZE=1044480
> > ./arm/lc823450/lc823450-
> > xgevk/configs/krndis/defconfig:CONFIG_RAM_SIZE=1044480
> > ./arm/lc823450/lc823450-
> > xgevk/configs/posix_spawn/defconfig:CONFIG_RAM_SIZE=1044480
> > ./arm/lc823450/lc823450-
> > xgevk/configs/rndis/defconfig:CONFIG_RAM_SIZE=1044480
> > ./arm/lc823450/lc823450-
> > xgevk/configs/usb/defconfig:CONFIG_RAM_SIZE=1044480
> > ./arm/lpc17xx_40xx/lx_cpu/configs/nsh/defconfig:CONFIG_RAM_SIZE=655
> > 36
> > ./arm/sama5/sama5d4-
> > ek/configs/elf/defconfig:CONFIG_RAM_SIZE=268435456
> > ./arm/sama5/sama5d4-
> > ek/configs/knsh/defconfig:CONFIG_RAM_SIZE=268435456
> > ./arm/stm32/olimex-stm32-
> > p407/configs/kelf/defconfig:CONFIG_RAM_SIZE=114688
> > ./arm/stm32/stm32f4discovery/configs/elf/defconfig:CONFIG_RAM_SIZE=
> > 11
> > 4688
> > ./arm/stm32/stm32f4discovery/configs/posix_spawn/defconfig:CONFIG_R
> > AM
> > _SIZE=114688
> > ./arm/stm32/stm32f4discovery/configs/rndis/defconfig:CONFIG_RAM_SIZ
> > E=
> > 114688
> > ./arm/tms570/tms570ls31x-usb-
> > kit/configs/nsh/defconfig:CONFIG_RAM_SIZE=262143
> > ./risc-v/k210/maix-
> > bit/configs/elf/defconfig:CONFIG_RAM_SIZE=2097152
> > ./risc-v/k210/maix-
> > bit/configs/module/defconfig:CONFIG_RAM_SIZE=2097152
> > ./risc-v/k210/maix-
> > bit/configs/posix_spawn/defconfig:CONFIG_RAM_SIZE=2097152
> >
> > And one of these exceptions has a large external DRAM:
> >
> > $ find . -name defconfig | xargs grep -l CONFIG_ELF=y | xargs grep
> > EXTDRAM
> > ./arm/lpc17xx_40xx/lx_cpu/configs/nsh/defconfig:CONFIG_LPC17_40_EXT
> > DR
> > AM=y
> > ./arm/lpc17xx_40xx/lx_cpu/configs/nsh/defconfig:CONFIG_LPC17_40_EXT
> > DR
> > AMSIZE=33554432
> >
> >
Re: heap malloc when executing binary/elf file
Posted by Florian Wehmeyer <fl...@gmx.net>.
OK Greg,
could be that I was using too much.. using 64KB of ramdisk also.. I
disabled some of the comfortable features like cmd line history etc,
and was already able to alloc quite a bit more than before..
Will further improve the sram usage..
Also, I missed the commit
mm/mm_heap: fix mm_heap not support BUILD_FLAT
GUIDINGLI
authored and patacongo
committed
on Apr 10
which well might impact my stuff here, will pull and let you know,
@Nathan I will also make tests if malloc continues returning non NULL
pointers even if out of memory..
Seems on a good way!!
Many thanks,
Florian
--
Florian Wehmeyer
TFW Tech-Solutions
On Seg, 2020-05-11 at 15:09 -0600, Gregory Nutt wrote:
> >
> >
> > >
> > > No custom board, it's the tm4c1294-launchpad.
> > > Seems no general problem with heap allocation, so I guess it's
> > > not in
> > > the linker script, it's rather directly linked to the usage of
> > > the elf-
> > > binary loader, and there are only two configs in the nuttx repo
> > > which
> > > use the CONFIG_ELF stuff.
> > That part has 256Kb of SRAM. That is more that many, but it could
> > be
> > that you are using too much SRAM and just cannot run ELF reliably.
> > It
> > does want a lot of SRAM.
> Most of the other boards that use CONFIG_ELF=y in a configuration
> have a
> much larger internal ram (exceptions in red):
>
> $ find . -name defconfig | xargs grep -l CONFIG_ELF=y | xargs grep
> RAM_SIZE
> ./arm/cxd56xx/spresense/configs/elf/defconfig:CONFIG_RAM_SIZE=1572864
> ./arm/cxd56xx/spresense/configs/module/defconfig:CONFIG_RAM_SIZE=1572
> 864
> ./arm/cxd56xx/spresense/configs/posix_spawn/defconfig:CONFIG_RAM_SIZE
> =1572864
> ./arm/cxd56xx/spresense/configs/rndis/defconfig:CONFIG_RAM_SIZE=15728
> 64
> ./arm/cxd56xx/spresense/configs/wifi/defconfig:CONFIG_RAM_SIZE=157286
> 4
> ./arm/lc823450/lc823450-
> xgevk/configs/elf/defconfig:CONFIG_RAM_SIZE=1044480
> ./arm/lc823450/lc823450-
> xgevk/configs/krndis/defconfig:CONFIG_RAM_SIZE=1044480
> ./arm/lc823450/lc823450-
> xgevk/configs/posix_spawn/defconfig:CONFIG_RAM_SIZE=1044480
> ./arm/lc823450/lc823450-
> xgevk/configs/rndis/defconfig:CONFIG_RAM_SIZE=1044480
> ./arm/lc823450/lc823450-
> xgevk/configs/usb/defconfig:CONFIG_RAM_SIZE=1044480
> ./arm/lpc17xx_40xx/lx_cpu/configs/nsh/defconfig:CONFIG_RAM_SIZE=65536
> ./arm/sama5/sama5d4-
> ek/configs/elf/defconfig:CONFIG_RAM_SIZE=268435456
> ./arm/sama5/sama5d4-
> ek/configs/knsh/defconfig:CONFIG_RAM_SIZE=268435456
> ./arm/stm32/olimex-stm32-
> p407/configs/kelf/defconfig:CONFIG_RAM_SIZE=114688
> ./arm/stm32/stm32f4discovery/configs/elf/defconfig:CONFIG_RAM_SIZE=11
> 4688
> ./arm/stm32/stm32f4discovery/configs/posix_spawn/defconfig:CONFIG_RAM
> _SIZE=114688
> ./arm/stm32/stm32f4discovery/configs/rndis/defconfig:CONFIG_RAM_SIZE=
> 114688
> ./arm/tms570/tms570ls31x-usb-
> kit/configs/nsh/defconfig:CONFIG_RAM_SIZE=262143
> ./risc-v/k210/maix-bit/configs/elf/defconfig:CONFIG_RAM_SIZE=2097152
> ./risc-v/k210/maix-
> bit/configs/module/defconfig:CONFIG_RAM_SIZE=2097152
> ./risc-v/k210/maix-
> bit/configs/posix_spawn/defconfig:CONFIG_RAM_SIZE=2097152
>
> And one of these exceptions has a large external DRAM:
>
> $ find . -name defconfig | xargs grep -l CONFIG_ELF=y | xargs grep
> EXTDRAM
> ./arm/lpc17xx_40xx/lx_cpu/configs/nsh/defconfig:CONFIG_LPC17_40_EXTDR
> AM=y
> ./arm/lpc17xx_40xx/lx_cpu/configs/nsh/defconfig:CONFIG_LPC17_40_EXTDR
> AMSIZE=33554432
>
>
Re: heap malloc when executing binary/elf file
Posted by Gregory Nutt <sp...@gmail.com>.
>
>> No custom board, it's the tm4c1294-launchpad.
>> Seems no general problem with heap allocation, so I guess it's not in
>> the linker script, it's rather directly linked to the usage of the elf-
>> binary loader, and there are only two configs in the nuttx repo which
>> use the CONFIG_ELF stuff.
>
> That part has 256Kb of SRAM. That is more that many, but it could be
> that you are using too much SRAM and just cannot run ELF reliably. It
> does want a lot of SRAM.
Most of the other boards that use CONFIG_ELF=y in a configuration have a
much larger internal ram (exceptions in red):
$ find . -name defconfig | xargs grep -l CONFIG_ELF=y | xargs grep RAM_SIZE
./arm/cxd56xx/spresense/configs/elf/defconfig:CONFIG_RAM_SIZE=1572864
./arm/cxd56xx/spresense/configs/module/defconfig:CONFIG_RAM_SIZE=1572864
./arm/cxd56xx/spresense/configs/posix_spawn/defconfig:CONFIG_RAM_SIZE=1572864
./arm/cxd56xx/spresense/configs/rndis/defconfig:CONFIG_RAM_SIZE=1572864
./arm/cxd56xx/spresense/configs/wifi/defconfig:CONFIG_RAM_SIZE=1572864
./arm/lc823450/lc823450-xgevk/configs/elf/defconfig:CONFIG_RAM_SIZE=1044480
./arm/lc823450/lc823450-xgevk/configs/krndis/defconfig:CONFIG_RAM_SIZE=1044480
./arm/lc823450/lc823450-xgevk/configs/posix_spawn/defconfig:CONFIG_RAM_SIZE=1044480
./arm/lc823450/lc823450-xgevk/configs/rndis/defconfig:CONFIG_RAM_SIZE=1044480
./arm/lc823450/lc823450-xgevk/configs/usb/defconfig:CONFIG_RAM_SIZE=1044480
./arm/lpc17xx_40xx/lx_cpu/configs/nsh/defconfig:CONFIG_RAM_SIZE=65536
./arm/sama5/sama5d4-ek/configs/elf/defconfig:CONFIG_RAM_SIZE=268435456
./arm/sama5/sama5d4-ek/configs/knsh/defconfig:CONFIG_RAM_SIZE=268435456
./arm/stm32/olimex-stm32-p407/configs/kelf/defconfig:CONFIG_RAM_SIZE=114688
./arm/stm32/stm32f4discovery/configs/elf/defconfig:CONFIG_RAM_SIZE=114688
./arm/stm32/stm32f4discovery/configs/posix_spawn/defconfig:CONFIG_RAM_SIZE=114688
./arm/stm32/stm32f4discovery/configs/rndis/defconfig:CONFIG_RAM_SIZE=114688
./arm/tms570/tms570ls31x-usb-kit/configs/nsh/defconfig:CONFIG_RAM_SIZE=262143
./risc-v/k210/maix-bit/configs/elf/defconfig:CONFIG_RAM_SIZE=2097152
./risc-v/k210/maix-bit/configs/module/defconfig:CONFIG_RAM_SIZE=2097152
./risc-v/k210/maix-bit/configs/posix_spawn/defconfig:CONFIG_RAM_SIZE=2097152
And one of these exceptions has a large external DRAM:
$ find . -name defconfig | xargs grep -l CONFIG_ELF=y | xargs grep EXTDRAM
./arm/lpc17xx_40xx/lx_cpu/configs/nsh/defconfig:CONFIG_LPC17_40_EXTDRAM=y
./arm/lpc17xx_40xx/lx_cpu/configs/nsh/defconfig:CONFIG_LPC17_40_EXTDRAMSIZE=33554432
Re: heap malloc when executing binary/elf file
Posted by Nathan Hartman <ha...@gmail.com>.
On Mon, May 11, 2020 at 5:01 PM Gregory Nutt <sp...@gmail.com> wrote:
> That part has 256Kb of SRAM. That is more that many, but it could be
> that you are using too much SRAM and just cannot run ELF reliably. It
> does want a lot of SRAM.
If there isn't enough memory, I wonder why malloc doesn't return NULL?
Haven't had the chance to look into the code yet. Maybe there's a bug
-- or perhaps there's something misconfigured that fools malloc into
thinking there's more memory than there really is.
Nathan
Re: heap malloc when executing binary/elf file
Posted by Gregory Nutt <sp...@gmail.com>.
> No custom board, it's the tm4c1294-launchpad.
> Seems no general problem with heap allocation, so I guess it's not in
> the linker script, it's rather directly linked to the usage of the elf-
> binary loader, and there are only two configs in the nuttx repo which
> use the CONFIG_ELF stuff.
That part has 256Kb of SRAM. That is more that many, but it could be
that you are using too much SRAM and just cannot run ELF reliably. It
does want a lot of SRAM.
Re: heap malloc when executing binary/elf file
Posted by Florian Wehmeyer <fl...@gmx.net>.
Thanks Nathan,
I'll check on that..
No custom board, it's the tm4c1294-launchpad.
Seems no general problem with heap allocation, so I guess it's not in
the linker script, it's rather directly linked to the usage of the elf-
binary loader, and there are only two configs in the nuttx repo which
use the CONFIG_ELF stuff..
But, I will double-check the elf linker script, maybe something wrong
with that!
BR,
--
Florian Wehmeyer
TFW Tech-Solutions
On Seg, 2020-05-11 at 16:24 -0400, Nathan Hartman wrote:
> On Mon, May 11, 2020 at 2:05 PM Florian Wehmeyer <fl...@gmx.net>
> wrote:
> >
> > OK I already saw that simply increasing the heap is no solution,
> > because it's already maximum size:
> > mm_initialize: Heap: start=0x20008924
> > size=227036
> > mm_addregion: Region 1: base=0x20008924 size=227024
> Tricky problem.
>
> Is this for a custom board or an existing configuration? If an
> existing configuration, which one?
>
> If it's for a custom board, then the place I would recommend to check
> is the linker script, Kconfig options, and compiler options. Perhaps
> try to find an existing configuration in the NuttX tree that is as
> close as possible to yours, and compare the various files (linker
> files, defconfig, etc) and see whether any differences that you find
> will affect heap allocations.
>
> Hopefully someone else here can chime in with some more specific
> items to check.
>
> Nathan
Re: heap malloc when executing binary/elf file
Posted by Nathan Hartman <ha...@gmail.com>.
On Mon, May 11, 2020 at 2:05 PM Florian Wehmeyer <fl...@gmx.net> wrote:
> OK I already saw that simply increasing the heap is no solution,
> because it's already maximum size:
> mm_initialize: Heap: start=0x20008924
> size=227036
> mm_addregion: Region 1: base=0x20008924 size=227024
Tricky problem.
Is this for a custom board or an existing configuration? If an
existing configuration, which one?
If it's for a custom board, then the place I would recommend to check
is the linker script, Kconfig options, and compiler options. Perhaps
try to find an existing configuration in the NuttX tree that is as
close as possible to yours, and compare the various files (linker
files, defconfig, etc) and see whether any differences that you find
will affect heap allocations.
Hopefully someone else here can chime in with some more specific items to check.
Nathan
Re: heap malloc when executing binary/elf file
Posted by Florian Wehmeyer <fl...@gmx.net>.
OK I already saw that simply increasing the heap is no solution,
because it's already maximum size:
mm_initialize: Heap: start=0x20008924
size=227036
mm_addregion: Region 1: base=0x20008924 size=227024
--
Florian Wehmeyer
TFW Tech-Solutions
On Seg, 2020-05-11 at 14:31 -0300, Florian Wehmeyer wrote:
> looking more into the memory allocation debug, there is a lot more
> dynamic allocation (obviously) when loading the elf program than
> when loading the same program as built-in..
> So, seems the problem is due to a high heap fragmentation, what
> explains that a call to dmesg can also make the system crash..
> So, maybe my question should be, is increasing the heap a solution
> for
> that? If yes, how do I increase it?
> BR,
> Florian
> prints of debug output with my comments:
> calling the Elf program (Neo_Ledtest):
> PID PRI POLICY TYPE NPX STATE EVENT SIGMASK STACK
> COMMAND
> 0 0 FIFO Kthread N-- Ready 00000000 000000
> Idle
> Task
> 1 224 FIFO Kthread --- Waiting Signal 00000000 002028
> hpwork
> 2 100 FIFO Task --- Running 00000000 002028
> init
> 3 80 RR pthread --- Waiting Semaphore 00000000 001564
> netinit 0x0
> 4 100 FIFO Task --- Waiting Semaphore 00000000 002004
> Telnet daemon0
> 6 100 FIFO Task --- Waiting Semaphore 00000000 006116
> Neo_Ledtest
> nsh> cat
> /proc/6/stack
>
> StackBase: 0x0x200214b0
>
>
> StackSize: 6116
> StackEnd = 0x200214b0 - 6116d = 0x2001FCCC
> nsh> dmesg //called right after calling the elf program,
> after flushing the
> syslog..
> mm_free: Freeing
> 0x2001d010
> mm_malloc: Allocated 0x2000cbd0, size
> 64
> mm_malloc: Allocated 0x20009710, size
> 32
> mm_malloc: Allocated 0x2000cc10, size
> 48
> mm_malloc: Allocated 0x2001d010, size
> 48
> mm_malloc: Allocated 0x2001d040, size
> 528
> mm_malloc: Allocated 0x2001d250, size
> 448
> mm_malloc: Allocated 0x2001d410, size
> 6624
> mm_malloc: Allocated 0x2001edf0, size
> 528
> mm_malloc: Allocated 0x2001f000, size
> 2064
> mm_malloc: Allocated 0x2001f810, size
> 48
> mm_malloc: Allocated 0x2001f840, size
> 48
> mm_malloc: Allocated 0x2001f870, size
> 48
> mm_malloc: Allocated 0x2001f8a0, size
> 48
> mm_malloc: Allocated 0x2001f8d0, size
> 48
> mm_malloc: Allocated 0x2001f900, size
> 48
> mm_malloc: Allocated 0x2001f930, size
> 48
> mm_malloc: Allocated 0x2001f960, size
> 48
> mm_malloc: Allocated 0x2001f990, size
> 48
> mm_malloc: Allocated 0x2001f9c0, size
> 48
> mm_malloc: Allocated 0x2001f9f0, size
> 48
> mm_malloc: Allocated 0x2001fa20, size
> 48
> mm_malloc: Allocated 0x2001fa50, size
> 48
> mm_malloc: Allocated 0x2001fa80, size
> 48
> mm_malloc: Allocated 0x2001fab0, size
> 48
> mm_malloc: Allocated 0x2001fae0, size
> 48
> mm_malloc: Allocated 0x2001fb10, size
> 48
> mm_malloc: Allocated 0x2001fb40, size
> 48
> mm_malloc: Allocated 0x2001fb70, size
> 48
> mm_malloc: Allocated 0x2001fba0, size
> 48
> mm_malloc: Allocated 0x2001fbd0, size
> 48
> mm_malloc: Allocated 0x2001fc00, size
> 48
> mm_malloc: Allocated 0x2001fc30, size
> 48
> mm_malloc: Allocated 0x2001fc60, size
> 48
> mm_malloc: Allocated 0x2001fc90, size
> 48
> mm_malloc: Allocated 0x2001fcc0, size
> 48
> mm_malloc: Allocated 0x2001fcf0, size
> 48
> mm_malloc: Allocated 0x2001fd20, size
> 48
> mm_malloc: Allocated 0x2001fd50, size
> 48
> mm_malloc: Allocated 0x2001fd80, size
> 48
> mm_malloc: Allocated 0x2001fdb0, size
> 48
> mm_malloc: Allocated 0x2001fde0, size
> 48
> mm_malloc: Allocated 0x2001fe10, size
> 48
> mm_malloc: Allocated 0x2001fe40, size
> 48
> mm_malloc: Allocated 0x2001fe70, size
> 48
> mm_malloc: Allocated 0x2001fea0, size
> 48
> mm_malloc: Allocated 0x2001fed0, size
> 48
> mm_malloc: Allocated 0x2001ff00, size
> 48
> mm_malloc: Allocated 0x2001ff30, size
> 48
> mm_malloc: Allocated 0x2001ff60, size
> 48
> mm_malloc: Allocated 0x2001ff90, size
> 48
> mm_malloc: Allocated 0x2001ffc0, size
> 48
> mm_malloc: Allocated 0x2001fff0, size
> 48
> mm_malloc: Allocated 0x20020020, size
> 48
> mm_free: Freeing
> 0x2001f000
> mm_free: Freeing
> 0x2001f930
> mm_free: Freeing
> 0x2001fd50
> mm_free: Freeing
> 0x2001fa50
> mm_free: Freeing
> 0x20020020
> mm_free: Freeing
> 0x2001fff0
> mm_free: Freeing
> 0x2001f840
> mm_free: Freeing
> 0x2001f9c0
> mm_free: Freeing
> 0x2001ffc0
> mm_free: Freeing
> 0x2001ff90
> mm_free: Freeing
> 0x2001fab0
> mm_free: Freeing
> 0x2001fa80
> mm_free: Freeing
> 0x2001fed0
> mm_free: Freeing
> 0x2001fd20
> mm_free: Freeing
> 0x2001fe70
> mm_free: Freeing
> 0x2001fe40
> mm_free: Freeing
> 0x2001ff30
> mm_free: Freeing
> 0x2001ff60
> mm_free: Freeing
> 0x2001fea0
> mm_free: Freeing
> 0x2001ff00
> mm_free: Freeing
> 0x2001fba0
> mm_free: Freeing
> 0x2001fb40
> mm_free: Freeing
> 0x2001fae0
> mm_free: Freeing
> 0x2001fe10
> mm_free: Freeing
> 0x2001fcf0
> mm_free: Freeing
> 0x2001fb10
> mm_free: Freeing
> 0x2001fde0
> mm_free: Freeing
> 0x2001fdb0
> mm_free: Freeing
> 0x2001fd80
> mm_free: Freeing
> 0x2001fa20
> mm_free: Freeing
> 0x2001fcc0
> mm_free: Freeing
> 0x2001fc90
> mm_free: Freeing
> 0x2001fc60
> mm_free: Freeing
> 0x2001fc30
> mm_free: Freeing
> 0x2001fc00
> mm_free: Freeing
> 0x2001fbd0
> mm_free: Freeing
> 0x2001fb70
> mm_free: Freeing
> 0x2001f9f0
> mm_free: Freeing
> 0x2001f990
> mm_free: Freeing
> 0x2001f960
> mm_free: Freeing
> 0x2001f900
> mm_free: Freeing
> 0x2001f8d0
> mm_free: Freeing
> 0x2001f8a0
> mm_free: Freeing
> 0x2001f870
> mm_free: Freeing
> 0x2001f810
> mm_free: Freeing
> 0x2001d250
> mm_free: Freeing
> 0x2001edf0
> mm_free: Freeing
> 0x2001d040
> mm_free: Freeing
> 0x2001d010
> mm_free: Freeing
> 0x2000cc10
> mm_free: Freeing
> 0x20009710
> mm_malloc: Allocated 0x2001d010, size
> 256
> mm_malloc: Allocated 0x2001edf0, size
> 2064
> mm_malloc: Allocated 0x2001f600, size
> 832
> mm_malloc: Allocated 0x2000cc10, size
> 64
> mm_malloc: Allocated 0x2001d110, size
> 80
> mm_malloc: Allocated 0x2001d160, size
> 80
> mm_malloc: Allocated 0x2001d1b0, size
> 80
> mm_malloc: Allocated 0x2001d200, size
> 256
> mm_malloc: Allocated 0x2001f940, size
> 832
> mm_malloc: Allocated 0x2001d300, size
> 64
> mm_malloc: Allocated 0x2001d340, size
> 80
> mm_malloc: Allocated 0x2001d390, size
> 80
> mm_malloc: Allocated 0x2001fc80, size
> 80
> mm_malloc: Allocated 0x2001fcd0, size 6160 //here allocating my
> (ELF)
> tasks stack, just befor my stack-end 0x2001FCCC
> mm_malloc: Allocated 0x20009710, size
> 32
> mm_free: Freeing
> 0x2001d110
> mm_free: Freeing
> 0x2001d160
> mm_free: Freeing
> 0x2001d1b0
> mm_free: Freeing
> 0x2000cc10
> mm_free: Freeing
> 0x2001d410
> mm_free: Freeing
> 0x2000cbd0
> mm_free: Freeing
> 0x2001edf0
> mm_free: Freeing
> 0x2001d010
> mm_free: Freeing
> 0x2001f600
> mm_free: Freeing
> 0x20009710
> mm_malloc: Allocated 0x2001d3e0, size 1008 //here allocating 1K in
> my
> task, heap not within my
> stack
>
>
> mm_free: Freeing 0x2001d3e0 //after short usage test,
> freeing the memory within my task
> mm_malloc: Allocated 0x2000cbd0, size 64 //allocation for dmesg:
> CONFIG_STDIO_BUFFER_SIZE
> mm_malloc: Allocated 0x20009710, size 32 //allocation for dmesg
> mm_malloc: Allocated 0x2000a8d0, size
> 32 //also
> mm_free: Freeing
> 0x2000a8d0
> mm_free: Freeing
> 0x20009710
> mm_free: Freeing
> 0x2000cbd0
> mm_malloc: Allocated 0x2001d3e0, size 528 //also caused by dmesg,
> maybe
> CONFIG_NSH_IOBUFFER_SIZE
> //16K space left to my
> stack-end
> //all other tasks,
> pthreads
> etc shown by ps command have stack allocated before this address!!
> ----------------------------
> calling the Neo_Ledtest as builtin:
> nsh>
> ps
>
>
> PID PRI POLICY TYPE NPX STATE EVENT SIGMASK STACK
> COMMAND
> 0 0 FIFO Kthread N-- Ready 00000000 000000
> Idle
> Task
> 1 224 FIFO Kthread --- Waiting Signal 00000000 002028
> hpwork
> 2 100 FIFO Task --- Running 00000000 002028
> init
> 3 80 RR pthread --- Waiting Semaphore 00000000 001564
> netinit 0x0
> 4 100 FIFO Task --- Waiting Semaphore 00000000 002004
> Telnet daemon0
> 6 100 FIFO Task --- Waiting Semaphore 00000000 006116
> Neo_Ledtest
> nsh>
>
>
> nsh>
>
>
> nsh> cat
> /proc/6/stack
>
> StackBase: 0x0x20020a50
>
>
> StackSize: 6116
> StackEnd = 0x200214b0 - 6116d = 0x2001F26C
> nsh>
> dmesg
>
>
> mm_free: Freeing
> 0x2001d010
> mm_malloc: Allocated 0x2000cbd0, size
> 64
> mm_malloc: Allocated 0x20009710, size
> 32
> mm_malloc: Allocated 0x2000cc10, size
> 48
> mm_free: Freeing
> 0x2000cc10
> mm_free: Freeing
> 0x20009710
> mm_free: Freeing
> 0x2000cbd0
> mm_malloc: Allocated 0x2001d010, size
> 256
> mm_malloc: Allocated 0x2001d110, size
> 832
> mm_malloc: Allocated 0x2000cbd0, size
> 64
> mm_malloc: Allocated 0x2000cc10, size
> 80
> mm_malloc: Allocated 0x2001d450, size
> 80
> mm_malloc: Allocated 0x2001d4a0, size
> 80
> mm_malloc: Allocated 0x2001d4f0, size
> 6160
> mm_malloc: Allocated 0x2001ed00, size
> 256
> mm_malloc: Allocated 0x2001ee00, size
> 832
> mm_malloc: Allocated 0x2001f140, size
> 64
> mm_malloc: Allocated 0x2001f180, size
> 80
> mm_malloc: Allocated 0x2001f1d0, size
> 80
> mm_malloc: Allocated 0x2001f220, size
> 80
> mm_malloc: Allocated 0x2001f270, size 6160 //here is my builtin task
> stack allocated, just before calculated stack-
> end
> mm_malloc: Allocated 0x20009710, size
> 32
> mm_free: Freeing
> 0x2000cc10
> mm_free: Freeing
> 0x2001d450
> mm_free: Freeing
> 0x2001d4a0
> mm_free: Freeing
> 0x2000cbd0
> mm_free: Freeing
> 0x2001d4f0
> mm_free: Freeing
> 0x2001d010
> mm_free: Freeing
> 0x2001d110
> mm_free: Freeing
> 0x20009710
> mm_malloc: Allocated 0x2001d010, size 7008 //allocating 7000Bytes
> in
> my task
> mm_free: Freeing
> 0x2001d010 //freeing
>
>
> mm_malloc: Allocated 0x2000cbd0, size 64 //dmesg
> stuff..
> mm_malloc: Allocated 0x20009710, size 32 //dmesg
> stuff..
> mm_malloc: Allocated 0x2000a8d0, size 32 //dmesg
> stuff..
> mm_free: Freeing
> 0x2000a8d0
> mm_free: Freeing
> 0x20009710
> mm_free: Freeing
> 0x2000cbd0
> mm_malloc: Allocated 0x2001d010, size 528 //dmesg stuff..
>
> //8.5K left until my stack-end
>
> --
> Florian WehmeyerTFW Tech-SolutionsOn Seg, 2020-05-11 at 11:35 -0300,
> Florian Wehmeyer wrote:
> >
> > Hi all,
> >
> > again a question related to the execution of external / elf
> > programs:
> >
> > When I use malloc within the running elf program, allocating more
> > than
> > 1K bytes, malloc seems to work (doesn't return NULL), but when I
> > use
> > that memory (memset) I get a assertion/Kernel panic..
> >
> > Funnily this does not happen if I use the same program as built-in
> > program (the one which is already in ROM flash).. Here I can
> > allocate
> > and use 8K or more without any problem..
> >
> > So, it seems the binary loader itself consumes a lot of memory,
> > decreasing the available heap.. I was not able to solve that
> > problem
> > even increasing all available and possibly related stacksizes in
> > .config
> > After loading the elf file, even running dmesg can cause that
> > assertion.. (using RAMLOG with 8K buffer and memory debug output) .
> > free shows about 130K of free RAM..
> >
> > in the memory debug output I can see that the periodical
> > allocation
> > at
> > about 10K of distance of the next stack-end (means stackbase minus
> > stacksize, knowing stack decreases), which is my app..
> > so, I thought allocate 2K on the heap should not be a problem.
> >
> >
> > Any ideas about any configuration that's important here?
> >
> >
> > Many thanks & BR,
> >
> >
> > --
> > Florian WehmeyerTFW Tech-Solutions
Re: heap malloc when executing binary/elf file
Posted by Florian Wehmeyer <fl...@gmx.net>.
looking more into the memory allocation debug, there is a lot more
dynamic allocation (obviously) when loading the elf program than
when loading the same program as built-in..
So, seems the problem is due to a high heap fragmentation, what
explains that a call to dmesg can also make the system crash..
So, maybe my question should be, is increasing the heap a solution for
that? If yes, how do I increase it?
BR,
Florian
prints of debug output with my comments:
calling the Elf program (Neo_Ledtest):
PID PRI POLICY TYPE NPX STATE EVENT SIGMASK STACK
COMMAND
0 0 FIFO Kthread N-- Ready 00000000 000000 Idle
Task
1 224 FIFO Kthread --- Waiting Signal 00000000 002028
hpwork
2 100 FIFO Task --- Running 00000000 002028
init
3 80 RR pthread --- Waiting Semaphore 00000000 001564
netinit 0x0
4 100 FIFO Task --- Waiting Semaphore 00000000 002004
Telnet daemon0
6 100 FIFO Task --- Waiting Semaphore 00000000 006116
Neo_Ledtest
nsh> cat
/proc/6/stack
StackBase: 0x0x200214b0
StackSize: 6116
StackEnd = 0x200214b0 - 6116d = 0x2001FCCC
nsh> dmesg //called right after calling the elf program,
after flushing the
syslog..
mm_free: Freeing
0x2001d010
mm_malloc: Allocated 0x2000cbd0, size
64
mm_malloc: Allocated 0x20009710, size
32
mm_malloc: Allocated 0x2000cc10, size
48
mm_malloc: Allocated 0x2001d010, size
48
mm_malloc: Allocated 0x2001d040, size
528
mm_malloc: Allocated 0x2001d250, size
448
mm_malloc: Allocated 0x2001d410, size
6624
mm_malloc: Allocated 0x2001edf0, size
528
mm_malloc: Allocated 0x2001f000, size
2064
mm_malloc: Allocated 0x2001f810, size
48
mm_malloc: Allocated 0x2001f840, size
48
mm_malloc: Allocated 0x2001f870, size
48
mm_malloc: Allocated 0x2001f8a0, size
48
mm_malloc: Allocated 0x2001f8d0, size
48
mm_malloc: Allocated 0x2001f900, size
48
mm_malloc: Allocated 0x2001f930, size
48
mm_malloc: Allocated 0x2001f960, size
48
mm_malloc: Allocated 0x2001f990, size
48
mm_malloc: Allocated 0x2001f9c0, size
48
mm_malloc: Allocated 0x2001f9f0, size
48
mm_malloc: Allocated 0x2001fa20, size
48
mm_malloc: Allocated 0x2001fa50, size
48
mm_malloc: Allocated 0x2001fa80, size
48
mm_malloc: Allocated 0x2001fab0, size
48
mm_malloc: Allocated 0x2001fae0, size
48
mm_malloc: Allocated 0x2001fb10, size
48
mm_malloc: Allocated 0x2001fb40, size
48
mm_malloc: Allocated 0x2001fb70, size
48
mm_malloc: Allocated 0x2001fba0, size
48
mm_malloc: Allocated 0x2001fbd0, size
48
mm_malloc: Allocated 0x2001fc00, size
48
mm_malloc: Allocated 0x2001fc30, size
48
mm_malloc: Allocated 0x2001fc60, size
48
mm_malloc: Allocated 0x2001fc90, size
48
mm_malloc: Allocated 0x2001fcc0, size
48
mm_malloc: Allocated 0x2001fcf0, size
48
mm_malloc: Allocated 0x2001fd20, size
48
mm_malloc: Allocated 0x2001fd50, size
48
mm_malloc: Allocated 0x2001fd80, size
48
mm_malloc: Allocated 0x2001fdb0, size
48
mm_malloc: Allocated 0x2001fde0, size
48
mm_malloc: Allocated 0x2001fe10, size
48
mm_malloc: Allocated 0x2001fe40, size
48
mm_malloc: Allocated 0x2001fe70, size
48
mm_malloc: Allocated 0x2001fea0, size
48
mm_malloc: Allocated 0x2001fed0, size
48
mm_malloc: Allocated 0x2001ff00, size
48
mm_malloc: Allocated 0x2001ff30, size
48
mm_malloc: Allocated 0x2001ff60, size
48
mm_malloc: Allocated 0x2001ff90, size
48
mm_malloc: Allocated 0x2001ffc0, size
48
mm_malloc: Allocated 0x2001fff0, size
48
mm_malloc: Allocated 0x20020020, size
48
mm_free: Freeing
0x2001f000
mm_free: Freeing
0x2001f930
mm_free: Freeing
0x2001fd50
mm_free: Freeing
0x2001fa50
mm_free: Freeing
0x20020020
mm_free: Freeing
0x2001fff0
mm_free: Freeing
0x2001f840
mm_free: Freeing
0x2001f9c0
mm_free: Freeing
0x2001ffc0
mm_free: Freeing
0x2001ff90
mm_free: Freeing
0x2001fab0
mm_free: Freeing
0x2001fa80
mm_free: Freeing
0x2001fed0
mm_free: Freeing
0x2001fd20
mm_free: Freeing
0x2001fe70
mm_free: Freeing
0x2001fe40
mm_free: Freeing
0x2001ff30
mm_free: Freeing
0x2001ff60
mm_free: Freeing
0x2001fea0
mm_free: Freeing
0x2001ff00
mm_free: Freeing
0x2001fba0
mm_free: Freeing
0x2001fb40
mm_free: Freeing
0x2001fae0
mm_free: Freeing
0x2001fe10
mm_free: Freeing
0x2001fcf0
mm_free: Freeing
0x2001fb10
mm_free: Freeing
0x2001fde0
mm_free: Freeing
0x2001fdb0
mm_free: Freeing
0x2001fd80
mm_free: Freeing
0x2001fa20
mm_free: Freeing
0x2001fcc0
mm_free: Freeing
0x2001fc90
mm_free: Freeing
0x2001fc60
mm_free: Freeing
0x2001fc30
mm_free: Freeing
0x2001fc00
mm_free: Freeing
0x2001fbd0
mm_free: Freeing
0x2001fb70
mm_free: Freeing
0x2001f9f0
mm_free: Freeing
0x2001f990
mm_free: Freeing
0x2001f960
mm_free: Freeing
0x2001f900
mm_free: Freeing
0x2001f8d0
mm_free: Freeing
0x2001f8a0
mm_free: Freeing
0x2001f870
mm_free: Freeing
0x2001f810
mm_free: Freeing
0x2001d250
mm_free: Freeing
0x2001edf0
mm_free: Freeing
0x2001d040
mm_free: Freeing
0x2001d010
mm_free: Freeing
0x2000cc10
mm_free: Freeing
0x20009710
mm_malloc: Allocated 0x2001d010, size
256
mm_malloc: Allocated 0x2001edf0, size
2064
mm_malloc: Allocated 0x2001f600, size
832
mm_malloc: Allocated 0x2000cc10, size
64
mm_malloc: Allocated 0x2001d110, size
80
mm_malloc: Allocated 0x2001d160, size
80
mm_malloc: Allocated 0x2001d1b0, size
80
mm_malloc: Allocated 0x2001d200, size
256
mm_malloc: Allocated 0x2001f940, size
832
mm_malloc: Allocated 0x2001d300, size
64
mm_malloc: Allocated 0x2001d340, size
80
mm_malloc: Allocated 0x2001d390, size
80
mm_malloc: Allocated 0x2001fc80, size
80
mm_malloc: Allocated 0x2001fcd0, size 6160 //here allocating my (ELF)
tasks stack, just befor my stack-end 0x2001FCCC
mm_malloc: Allocated 0x20009710, size
32
mm_free: Freeing
0x2001d110
mm_free: Freeing
0x2001d160
mm_free: Freeing
0x2001d1b0
mm_free: Freeing
0x2000cc10
mm_free: Freeing
0x2001d410
mm_free: Freeing
0x2000cbd0
mm_free: Freeing
0x2001edf0
mm_free: Freeing
0x2001d010
mm_free: Freeing
0x2001f600
mm_free: Freeing
0x20009710
mm_malloc: Allocated 0x2001d3e0, size 1008 //here allocating 1K in my
task, heap not within my
stack
mm_free: Freeing 0x2001d3e0 //after short usage test,
freeing the memory within my task
mm_malloc: Allocated 0x2000cbd0, size 64 //allocation for dmesg:
CONFIG_STDIO_BUFFER_SIZE
mm_malloc: Allocated 0x20009710, size 32 //allocation for dmesg
mm_malloc: Allocated 0x2000a8d0, size
32 //also
mm_free: Freeing
0x2000a8d0
mm_free: Freeing
0x20009710
mm_free: Freeing
0x2000cbd0
mm_malloc: Allocated 0x2001d3e0, size 528 //also caused by dmesg, maybe
CONFIG_NSH_IOBUFFER_SIZE
//16K space left to my
stack-end
//all other tasks, pthreads
etc shown by ps command have stack allocated before this address!!
----------------------------
calling the Neo_Ledtest as builtin:
nsh>
ps
PID PRI POLICY TYPE NPX STATE EVENT SIGMASK STACK
COMMAND
0 0 FIFO Kthread N-- Ready 00000000 000000 Idle
Task
1 224 FIFO Kthread --- Waiting Signal 00000000 002028
hpwork
2 100 FIFO Task --- Running 00000000 002028
init
3 80 RR pthread --- Waiting Semaphore 00000000 001564
netinit 0x0
4 100 FIFO Task --- Waiting Semaphore 00000000 002004
Telnet daemon0
6 100 FIFO Task --- Waiting Semaphore 00000000 006116
Neo_Ledtest
nsh>
nsh>
nsh> cat
/proc/6/stack
StackBase: 0x0x20020a50
StackSize: 6116
StackEnd = 0x200214b0 - 6116d = 0x2001F26C
nsh>
dmesg
mm_free: Freeing
0x2001d010
mm_malloc: Allocated 0x2000cbd0, size
64
mm_malloc: Allocated 0x20009710, size
32
mm_malloc: Allocated 0x2000cc10, size
48
mm_free: Freeing
0x2000cc10
mm_free: Freeing
0x20009710
mm_free: Freeing
0x2000cbd0
mm_malloc: Allocated 0x2001d010, size
256
mm_malloc: Allocated 0x2001d110, size
832
mm_malloc: Allocated 0x2000cbd0, size
64
mm_malloc: Allocated 0x2000cc10, size
80
mm_malloc: Allocated 0x2001d450, size
80
mm_malloc: Allocated 0x2001d4a0, size
80
mm_malloc: Allocated 0x2001d4f0, size
6160
mm_malloc: Allocated 0x2001ed00, size
256
mm_malloc: Allocated 0x2001ee00, size
832
mm_malloc: Allocated 0x2001f140, size
64
mm_malloc: Allocated 0x2001f180, size
80
mm_malloc: Allocated 0x2001f1d0, size
80
mm_malloc: Allocated 0x2001f220, size
80
mm_malloc: Allocated 0x2001f270, size 6160 //here is my builtin task
stack allocated, just before calculated stack-
end
mm_malloc: Allocated 0x20009710, size
32
mm_free: Freeing
0x2000cc10
mm_free: Freeing
0x2001d450
mm_free: Freeing
0x2001d4a0
mm_free: Freeing
0x2000cbd0
mm_free: Freeing
0x2001d4f0
mm_free: Freeing
0x2001d010
mm_free: Freeing
0x2001d110
mm_free: Freeing
0x20009710
mm_malloc: Allocated 0x2001d010, size 7008 //allocating 7000Bytes in
my task
mm_free: Freeing
0x2001d010 //freeing
mm_malloc: Allocated 0x2000cbd0, size 64 //dmesg
stuff..
mm_malloc: Allocated 0x20009710, size 32 //dmesg
stuff..
mm_malloc: Allocated 0x2000a8d0, size 32 //dmesg
stuff..
mm_free: Freeing
0x2000a8d0
mm_free: Freeing
0x20009710
mm_free: Freeing
0x2000cbd0
mm_malloc: Allocated 0x2001d010, size 528 //dmesg stuff..
//8.5K left until my stack-end
--
Florian WehmeyerTFW Tech-SolutionsOn Seg, 2020-05-11 at 11:35 -0300, Florian Wehmeyer wrote:
> Hi all,
>
> again a question related to the execution of external / elf
> programs:
>
> When I use malloc within the running elf program, allocating more
> than
> 1K bytes, malloc seems to work (doesn't return NULL), but when I use
> that memory (memset) I get a assertion/Kernel panic..
>
> Funnily this does not happen if I use the same program as built-in
> program (the one which is already in ROM flash).. Here I can allocate
> and use 8K or more without any problem..
>
> So, it seems the binary loader itself consumes a lot of memory,
> decreasing the available heap.. I was not able to solve that problem
> even increasing all available and possibly related stacksizes in
> .config
> After loading the elf file, even running dmesg can cause that
> assertion.. (using RAMLOG with 8K buffer and memory debug output) .
> free shows about 130K of free RAM..
>
> in the memory debug output I can see that the periodical allocation
> at
> about 10K of distance of the next stack-end (means stackbase minus
> stacksize, knowing stack decreases), which is my app..
> so, I thought allocate 2K on the heap should not be a problem.
>
>
> Any ideas about any configuration that's important here?
>
>
> Many thanks & BR,
>
>
> --
> Florian WehmeyerTFW Tech-Solutions