You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nuttx.apache.org by Matteo Golin <ma...@gmail.com> on 2024/08/01 01:15:53 UTC
aarch64 boot crash when printing non-static string
Hello all,
I've been working on this problem for a while now and I'm not quite sure
how to progress in my troubleshooting.
I am trying to port NuttX to the Raspberry Pi 4B, which has been going well
thanks to the documentation suggestions from my previous email (especially
Lup's blogs which have been invaluable).
There is an option for aarch64 to enable early printing within the NuttX
boot sequence. This enables the use of a PRINT macro
<https://github.com/apache/nuttx/blob/fe45d8aace2683b212e4ca2b9166a5df93c760fe/arch/arm64/src/common/arm64_head.S#L63>
in the startup code which is quite useful to me as debug output to see how
far along the boot is getting.
I have been able to implement the `lowputc` requirements to enable early
printing, but I am encountering what I consider to be a strange issue.
My `arm64_earlyprintinit`
function
<https://github.com/apache/nuttx/blob/fe45d8aace2683b212e4ca2b9166a5df93c760fe/arch/arm64/src/common/arm64_head.S#L215>
is being called and executed just fine, and same for the `arm64_lowputc`
function. I expected to see the output of the PRINT macros in my serial
console, but just got some garbled output followed by a crash (indicated by
the Pi LED). I thought I had configured the UART wrong, but after some
debugging involving adding direct calls to `arm64_lowputc` at the end of
the `arm64_earlyprintinit`, I saw that UART was configured just fine
because I was able to see the characters printed by my direct calls.
My next step was to print a string like "Hello NuttX!", so I did that by
using `arm64_lowputc` calls in a for loop. This didn't work, and I got some
garbled output again. After some more tinkering, I discovered that if I
marked the string literal as `static`, my printing in a for-loop succeeded.
I thought maybe this was a problem with the stack initialization, but the
`uint8_t i` variable I use as a counter in my for-loop is working just fine.
MESS:00:00:06.225070:0: Loaded 'nuttx.img' to 0x200000 size 0x46000
> MESS:00:00:06.231469:0: Kernel relocated to 0x480000
> MESS:00:00:06.235742:0: Device tree loaded to 0x2eff2000 (size 0xdf76)
> MESS:00:00:06.243988:0: uart: Set PL011 baud rate to 103448.300000 Hz
> MESS:00:00:06.251071:0: uart: Baud rate change done...
> MESS:00:00:06.253089:0:Hello from NuttX!
*The debug output of the Raspberry Pi start.elf program, followed by the
printing of a static "Hello from NuttX!" string. Immediately after should
be a second printed output, but it does not show because the string is
non-static and this seems to crash the boot process.*
Other bare-metal kernels for the RPi that I've seen online do not have this
issue, and can print non-static strings over UART just fine. I've even
tried compiling one of the tutorial kernels to try on the Pi and it does
boot and print the non-static string (see here
<https://github.com/babbleberry/rpi4-osdev/tree/master/part3-helloworld>).
I cannot wrap my head around why I would be able to print a static string,
but not a non-static one. Both strings appear in the `.rodata` section of
the image when I inspect it with `objdump` as well. Even the assembly
`PRINT` macro in the arm64_head.S startup code declares the strings as
rodata. I'm not sure what the difference is here, but it's preventing me
from continuing to debug the boot process. I'm by no means very
experienced, so I feel a bit stuck with this issue. I thought I would check
if anyone here has encountered something similar.
In case it's of any use, my development branch can be found here:
https://github.com/linguini1/nuttx/tree/bcm2711
You can compile the exact same image as me with `./tools/configure.sh
raspberrypi-4b:nsh && make` (in case you have a Pi 4B and want to try).
Any tips or pointers of what else I can check would be really appreciated!
Re: aarch64 boot crash when printing non-static string
Posted by mi...@bofc.pl.
On 2024-07-31 21:15:53, Matteo Golin wrote:
> I cannot wrap my head around why I would be able to print a static string,
> but not a non-static one. Both strings appear in the `.rodata` section of
> the image when I inspect it with `objdump` as well. Even the assembly
> `PRINT` macro in the arm64_head.S startup code declares the strings as
> rodata. I'm not sure what the difference is here, but it's preventing me
> from continuing to debug the boot process. I'm by no means very
> experienced, so I feel a bit stuck with this issue. I thought I would check
> if anyone here has encountered something similar.
>
> Any tips or pointers of what else I can check would be really appreciated!
Can you maybe post stack trace from GDB after crash? Also check what address
is dereferenced. Static will probably have different address than non static.
aarch64 has some security and atf, maybe that is causing the problem? This is
just a wild guess.
Check failing instruction, dump info from it like addresses, and then do the
same for working static data. Maybe this will shed some light onto this
problem. It's good to know which instruction is failing and what arguments it
takes.
--
.-----------------.-------------------.----------------------.-----------------.
| Michal Lyszczek | Embedded C, Linux | Company Address | .-. opensource |
| +48 727 564 419 | Software Engineer | Akacjowa 10a; 55-330 | oo| supporter |
| https://bofc.pl `----.--------------: Brzezinka Sredzka PL | /`'\ & |
| GPG FF1EBFE7E3A974B1 | Bits of Code | NIP: 813 349 58 78 |(\_;/) programer |
`----------------------^--------------^----------------------^-----------------'
Re: aarch64 boot crash when printing non-static string
Posted by Matteo Golin <ma...@gmail.com>.
Hello Petro and Michal,
It seems like it was the linker script load address that was incorrect in my case. After more debugging, I read through
the Raspberry Pi boot loader's output and saw the kernel was being loaded at 0x480000 instead of 0x80000 as I had read
in some other resources for the Raspberry Pi. I could have sworn I had tried this address already, but must have
forgotten a 0.
Thank you for your help again,
--
Matteo Golin
B. Eng Computer Systems Student
Re: aarch64 boot crash when printing non-static string
Posted by Petro Karashchenko <pe...@gmail.com>.
Hello Matteo,
It is quite hard to make any assumptions without having an example visible
in front of my eyes. The case without `static` is a bit different, so
despite string content is located in .rodata.str when you do not have
`static` and string (char array) is declared in a function there is some
additional code that populates the string content into an allocated stack
(for a local string) and then stack address should be passed to `PRINT`. If
the stack address is valid and there is a copy routine that populates data
from .rodata.str to your stack string (char array) then I can assume that
maybe some addresses in linker script are messed up and wrong address is
taken when code tries to copy from .rodata.str to stack.
Not sure if that helps as again there may be another thing that is hidden
and it is hard to check without inspecting linker script and elf.
Best regards,
Petro
On Sat, Aug 3, 2024, 2:09 AM Matteo Golin <ma...@gmail.com> wrote:
> Hello,
>
> Thank you Petro for the help! I've taken a look over the past couple days,
> tried a few different variations, and I'm still very confused by what's
> going on. You're correct that the program seems to get stuck when I print a
> string which is *not* marked as `static` in `arm64_earlyprintinit`
> (printing the static string works just fine).
>
> What I'm still confused about is the failing of the assembly `PRINT` macro.
> This macro marks any string passed to it as being in the `.rodata.str`
> section, which is the same section that my statically declared string from
> C code has been put in. This is not on the stack, so there is something
> else wrong (if not in addition to the stack issues). I have disassembled
> and inspected the generated binary using `objdump`, and everything seems
> correct: the address of the strings in .rodata are loaded into a register,
> the string is iterated over like an array and the `arm64_lowputc` function
> is called to print each character. The string data is present in the
> .rodata section. The addresses are seemingly all correct in the
> disassembly, the binary is loaded fully into RAM at the correct load
> address, etc. Yet still, garbage is printed to the screen when the first
> `PRINT` macro call is reached.
>
> I have tested calling `arm64_lowputc` from assembly with a character value
> placed in `w0` to be passed as the argument, which works. I've tested
> printing static strings from a C function call, which works as well. I've
> tested commenting out some of the earlier function calls to the
> `arm64_boot_primary_c_routine` and `arm64_data_initialize` with no
> difference in results.
>
> This same `PRINT` macro works just fine when tested with the same boot
> sequence on QEMU (and presumably for all the other arm64 platforms merged
> into NuttX), so something has to be wrong with my implementation. I am
> struggling to figure out what difference has caused the issue though, as my
> linker script is almost identical to the other arm64 implementations
> outside of requiring a different load address. It seems to not be an issue
> with the `lowputc` functions either, as printing is working with static
> strings declared in C code.
>
> Any ideas of where else I can look?
>
> Thank you,
> Matteo
>
> On Thu, Aug 1, 2024 at 12:14 AM Petro Karashchenko <
> petro.karashchenko@gmail.com> wrote:
>
> > Hi,
> >
> > Based on what you describe it seems like a stack configuration issue.
> About
> > "but the
> > `uint8_t i` variable I use as a counter in my for-loop is working just
> > fine" -- I think that the compiler just optimizes out loop variable and
> > does not use stack for it. There are a few things here to try here. You
> can
> > try to add "static" to your strings that you are trying to print, so
> those
> > will be not placed on stack (not sure if they are now). If you confirm
> that
> > the issue is with printing strings located on stack then you can focus on
> > making sure that stack pointer is pointing to valid memory.
> >
> > Best regards,
> > Petro
> >
> > чт, 1 серп. 2024 р. о 03:13 Matteo Golin <ma...@gmail.com> пише:
> >
> > > Hello all,
> > >
> > > I've been working on this problem for a while now and I'm not quite
> sure
> > > how to progress in my troubleshooting.
> > >
> > > I am trying to port NuttX to the Raspberry Pi 4B, which has been going
> > well
> > > thanks to the documentation suggestions from my previous email
> > (especially
> > > Lup's blogs which have been invaluable).
> > >
> > > There is an option for aarch64 to enable early printing within the
> NuttX
> > > boot sequence. This enables the use of a PRINT macro
> > > <
> > >
> >
> https://github.com/apache/nuttx/blob/fe45d8aace2683b212e4ca2b9166a5df93c760fe/arch/arm64/src/common/arm64_head.S#L63
> > > >
> > > in the startup code which is quite useful to me as debug output to see
> > how
> > > far along the boot is getting.
> > >
> > > I have been able to implement the `lowputc` requirements to enable
> early
> > > printing, but I am encountering what I consider to be a strange issue.
> > > My `arm64_earlyprintinit`
> > > function
> > > <
> > >
> >
> https://github.com/apache/nuttx/blob/fe45d8aace2683b212e4ca2b9166a5df93c760fe/arch/arm64/src/common/arm64_head.S#L215
> > > >
> > > is being called and executed just fine, and same for the
> `arm64_lowputc`
> > > function. I expected to see the output of the PRINT macros in my serial
> > > console, but just got some garbled output followed by a crash
> (indicated
> > by
> > > the Pi LED). I thought I had configured the UART wrong, but after some
> > > debugging involving adding direct calls to `arm64_lowputc` at the end
> of
> > > the `arm64_earlyprintinit`, I saw that UART was configured just fine
> > > because I was able to see the characters printed by my direct calls.
> > >
> > > My next step was to print a string like "Hello NuttX!", so I did that
> by
> > > using `arm64_lowputc` calls in a for loop. This didn't work, and I got
> > some
> > > garbled output again. After some more tinkering, I discovered that if I
> > > marked the string literal as `static`, my printing in a for-loop
> > succeeded.
> > > I thought maybe this was a problem with the stack initialization, but
> the
> > > `uint8_t i` variable I use as a counter in my for-loop is working just
> > > fine.
> > >
> > > MESS:00:00:06.225070:0: Loaded 'nuttx.img' to 0x200000 size 0x46000
> > > > MESS:00:00:06.231469:0: Kernel relocated to 0x480000
> > > > MESS:00:00:06.235742:0: Device tree loaded to 0x2eff2000 (size
> 0xdf76)
> > > > MESS:00:00:06.243988:0: uart: Set PL011 baud rate to 103448.300000 Hz
> > > > MESS:00:00:06.251071:0: uart: Baud rate change done...
> > > > MESS:00:00:06.253089:0:Hello from NuttX!
> > >
> > >
> > > *The debug output of the Raspberry Pi start.elf program, followed by
> the
> > > printing of a static "Hello from NuttX!" string. Immediately after
> should
> > > be a second printed output, but it does not show because the string is
> > > non-static and this seems to crash the boot process.*
> > >
> > > Other bare-metal kernels for the RPi that I've seen online do not have
> > this
> > > issue, and can print non-static strings over UART just fine. I've even
> > > tried compiling one of the tutorial kernels to try on the Pi and it
> does
> > > boot and print the non-static string (see here
> > > <
> https://github.com/babbleberry/rpi4-osdev/tree/master/part3-helloworld
> > >).
> > >
> > > I cannot wrap my head around why I would be able to print a static
> > string,
> > > but not a non-static one. Both strings appear in the `.rodata` section
> of
> > > the image when I inspect it with `objdump` as well. Even the assembly
> > > `PRINT` macro in the arm64_head.S startup code declares the strings as
> > > rodata. I'm not sure what the difference is here, but it's preventing
> me
> > > from continuing to debug the boot process. I'm by no means very
> > > experienced, so I feel a bit stuck with this issue. I thought I would
> > check
> > > if anyone here has encountered something similar.
> > >
> > > In case it's of any use, my development branch can be found here:
> > > https://github.com/linguini1/nuttx/tree/bcm2711
> > > You can compile the exact same image as me with `./tools/configure.sh
> > > raspberrypi-4b:nsh && make` (in case you have a Pi 4B and want to try).
> > >
> > > Any tips or pointers of what else I can check would be really
> > appreciated!
> > >
> >
>
Re: aarch64 boot crash when printing non-static string
Posted by Matteo Golin <ma...@gmail.com>.
Hello,
Thank you Petro for the help! I've taken a look over the past couple days,
tried a few different variations, and I'm still very confused by what's
going on. You're correct that the program seems to get stuck when I print a
string which is *not* marked as `static` in `arm64_earlyprintinit`
(printing the static string works just fine).
What I'm still confused about is the failing of the assembly `PRINT` macro.
This macro marks any string passed to it as being in the `.rodata.str`
section, which is the same section that my statically declared string from
C code has been put in. This is not on the stack, so there is something
else wrong (if not in addition to the stack issues). I have disassembled
and inspected the generated binary using `objdump`, and everything seems
correct: the address of the strings in .rodata are loaded into a register,
the string is iterated over like an array and the `arm64_lowputc` function
is called to print each character. The string data is present in the
.rodata section. The addresses are seemingly all correct in the
disassembly, the binary is loaded fully into RAM at the correct load
address, etc. Yet still, garbage is printed to the screen when the first
`PRINT` macro call is reached.
I have tested calling `arm64_lowputc` from assembly with a character value
placed in `w0` to be passed as the argument, which works. I've tested
printing static strings from a C function call, which works as well. I've
tested commenting out some of the earlier function calls to the
`arm64_boot_primary_c_routine` and `arm64_data_initialize` with no
difference in results.
This same `PRINT` macro works just fine when tested with the same boot
sequence on QEMU (and presumably for all the other arm64 platforms merged
into NuttX), so something has to be wrong with my implementation. I am
struggling to figure out what difference has caused the issue though, as my
linker script is almost identical to the other arm64 implementations
outside of requiring a different load address. It seems to not be an issue
with the `lowputc` functions either, as printing is working with static
strings declared in C code.
Any ideas of where else I can look?
Thank you,
Matteo
On Thu, Aug 1, 2024 at 12:14 AM Petro Karashchenko <
petro.karashchenko@gmail.com> wrote:
> Hi,
>
> Based on what you describe it seems like a stack configuration issue. About
> "but the
> `uint8_t i` variable I use as a counter in my for-loop is working just
> fine" -- I think that the compiler just optimizes out loop variable and
> does not use stack for it. There are a few things here to try here. You can
> try to add "static" to your strings that you are trying to print, so those
> will be not placed on stack (not sure if they are now). If you confirm that
> the issue is with printing strings located on stack then you can focus on
> making sure that stack pointer is pointing to valid memory.
>
> Best regards,
> Petro
>
> чт, 1 серп. 2024 р. о 03:13 Matteo Golin <ma...@gmail.com> пише:
>
> > Hello all,
> >
> > I've been working on this problem for a while now and I'm not quite sure
> > how to progress in my troubleshooting.
> >
> > I am trying to port NuttX to the Raspberry Pi 4B, which has been going
> well
> > thanks to the documentation suggestions from my previous email
> (especially
> > Lup's blogs which have been invaluable).
> >
> > There is an option for aarch64 to enable early printing within the NuttX
> > boot sequence. This enables the use of a PRINT macro
> > <
> >
> https://github.com/apache/nuttx/blob/fe45d8aace2683b212e4ca2b9166a5df93c760fe/arch/arm64/src/common/arm64_head.S#L63
> > >
> > in the startup code which is quite useful to me as debug output to see
> how
> > far along the boot is getting.
> >
> > I have been able to implement the `lowputc` requirements to enable early
> > printing, but I am encountering what I consider to be a strange issue.
> > My `arm64_earlyprintinit`
> > function
> > <
> >
> https://github.com/apache/nuttx/blob/fe45d8aace2683b212e4ca2b9166a5df93c760fe/arch/arm64/src/common/arm64_head.S#L215
> > >
> > is being called and executed just fine, and same for the `arm64_lowputc`
> > function. I expected to see the output of the PRINT macros in my serial
> > console, but just got some garbled output followed by a crash (indicated
> by
> > the Pi LED). I thought I had configured the UART wrong, but after some
> > debugging involving adding direct calls to `arm64_lowputc` at the end of
> > the `arm64_earlyprintinit`, I saw that UART was configured just fine
> > because I was able to see the characters printed by my direct calls.
> >
> > My next step was to print a string like "Hello NuttX!", so I did that by
> > using `arm64_lowputc` calls in a for loop. This didn't work, and I got
> some
> > garbled output again. After some more tinkering, I discovered that if I
> > marked the string literal as `static`, my printing in a for-loop
> succeeded.
> > I thought maybe this was a problem with the stack initialization, but the
> > `uint8_t i` variable I use as a counter in my for-loop is working just
> > fine.
> >
> > MESS:00:00:06.225070:0: Loaded 'nuttx.img' to 0x200000 size 0x46000
> > > MESS:00:00:06.231469:0: Kernel relocated to 0x480000
> > > MESS:00:00:06.235742:0: Device tree loaded to 0x2eff2000 (size 0xdf76)
> > > MESS:00:00:06.243988:0: uart: Set PL011 baud rate to 103448.300000 Hz
> > > MESS:00:00:06.251071:0: uart: Baud rate change done...
> > > MESS:00:00:06.253089:0:Hello from NuttX!
> >
> >
> > *The debug output of the Raspberry Pi start.elf program, followed by the
> > printing of a static "Hello from NuttX!" string. Immediately after should
> > be a second printed output, but it does not show because the string is
> > non-static and this seems to crash the boot process.*
> >
> > Other bare-metal kernels for the RPi that I've seen online do not have
> this
> > issue, and can print non-static strings over UART just fine. I've even
> > tried compiling one of the tutorial kernels to try on the Pi and it does
> > boot and print the non-static string (see here
> > <https://github.com/babbleberry/rpi4-osdev/tree/master/part3-helloworld
> >).
> >
> > I cannot wrap my head around why I would be able to print a static
> string,
> > but not a non-static one. Both strings appear in the `.rodata` section of
> > the image when I inspect it with `objdump` as well. Even the assembly
> > `PRINT` macro in the arm64_head.S startup code declares the strings as
> > rodata. I'm not sure what the difference is here, but it's preventing me
> > from continuing to debug the boot process. I'm by no means very
> > experienced, so I feel a bit stuck with this issue. I thought I would
> check
> > if anyone here has encountered something similar.
> >
> > In case it's of any use, my development branch can be found here:
> > https://github.com/linguini1/nuttx/tree/bcm2711
> > You can compile the exact same image as me with `./tools/configure.sh
> > raspberrypi-4b:nsh && make` (in case you have a Pi 4B and want to try).
> >
> > Any tips or pointers of what else I can check would be really
> appreciated!
> >
>
Re: aarch64 boot crash when printing non-static string
Posted by Petro Karashchenko <pe...@gmail.com>.
Hi,
Based on what you describe it seems like a stack configuration issue. About
"but the
`uint8_t i` variable I use as a counter in my for-loop is working just
fine" -- I think that the compiler just optimizes out loop variable and
does not use stack for it. There are a few things here to try here. You can
try to add "static" to your strings that you are trying to print, so those
will be not placed on stack (not sure if they are now). If you confirm that
the issue is with printing strings located on stack then you can focus on
making sure that stack pointer is pointing to valid memory.
Best regards,
Petro
чт, 1 серп. 2024 р. о 03:13 Matteo Golin <ma...@gmail.com> пише:
> Hello all,
>
> I've been working on this problem for a while now and I'm not quite sure
> how to progress in my troubleshooting.
>
> I am trying to port NuttX to the Raspberry Pi 4B, which has been going well
> thanks to the documentation suggestions from my previous email (especially
> Lup's blogs which have been invaluable).
>
> There is an option for aarch64 to enable early printing within the NuttX
> boot sequence. This enables the use of a PRINT macro
> <
> https://github.com/apache/nuttx/blob/fe45d8aace2683b212e4ca2b9166a5df93c760fe/arch/arm64/src/common/arm64_head.S#L63
> >
> in the startup code which is quite useful to me as debug output to see how
> far along the boot is getting.
>
> I have been able to implement the `lowputc` requirements to enable early
> printing, but I am encountering what I consider to be a strange issue.
> My `arm64_earlyprintinit`
> function
> <
> https://github.com/apache/nuttx/blob/fe45d8aace2683b212e4ca2b9166a5df93c760fe/arch/arm64/src/common/arm64_head.S#L215
> >
> is being called and executed just fine, and same for the `arm64_lowputc`
> function. I expected to see the output of the PRINT macros in my serial
> console, but just got some garbled output followed by a crash (indicated by
> the Pi LED). I thought I had configured the UART wrong, but after some
> debugging involving adding direct calls to `arm64_lowputc` at the end of
> the `arm64_earlyprintinit`, I saw that UART was configured just fine
> because I was able to see the characters printed by my direct calls.
>
> My next step was to print a string like "Hello NuttX!", so I did that by
> using `arm64_lowputc` calls in a for loop. This didn't work, and I got some
> garbled output again. After some more tinkering, I discovered that if I
> marked the string literal as `static`, my printing in a for-loop succeeded.
> I thought maybe this was a problem with the stack initialization, but the
> `uint8_t i` variable I use as a counter in my for-loop is working just
> fine.
>
> MESS:00:00:06.225070:0: Loaded 'nuttx.img' to 0x200000 size 0x46000
> > MESS:00:00:06.231469:0: Kernel relocated to 0x480000
> > MESS:00:00:06.235742:0: Device tree loaded to 0x2eff2000 (size 0xdf76)
> > MESS:00:00:06.243988:0: uart: Set PL011 baud rate to 103448.300000 Hz
> > MESS:00:00:06.251071:0: uart: Baud rate change done...
> > MESS:00:00:06.253089:0:Hello from NuttX!
>
>
> *The debug output of the Raspberry Pi start.elf program, followed by the
> printing of a static "Hello from NuttX!" string. Immediately after should
> be a second printed output, but it does not show because the string is
> non-static and this seems to crash the boot process.*
>
> Other bare-metal kernels for the RPi that I've seen online do not have this
> issue, and can print non-static strings over UART just fine. I've even
> tried compiling one of the tutorial kernels to try on the Pi and it does
> boot and print the non-static string (see here
> <https://github.com/babbleberry/rpi4-osdev/tree/master/part3-helloworld>).
>
> I cannot wrap my head around why I would be able to print a static string,
> but not a non-static one. Both strings appear in the `.rodata` section of
> the image when I inspect it with `objdump` as well. Even the assembly
> `PRINT` macro in the arm64_head.S startup code declares the strings as
> rodata. I'm not sure what the difference is here, but it's preventing me
> from continuing to debug the boot process. I'm by no means very
> experienced, so I feel a bit stuck with this issue. I thought I would check
> if anyone here has encountered something similar.
>
> In case it's of any use, my development branch can be found here:
> https://github.com/linguini1/nuttx/tree/bcm2711
> You can compile the exact same image as me with `./tools/configure.sh
> raspberrypi-4b:nsh && make` (in case you have a Pi 4B and want to try).
>
> Any tips or pointers of what else I can check would be really appreciated!
>