You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Joe McDonnell (Code Review)" <ge...@cloudera.org> on 2025/09/19 00:57:56 UTC
[native-toolchain-CR] IMPALA-13472: Patch breakpad to fix minidump stacks for 64KB pages
Joe McDonnell has uploaded this change for review. ( http://gerrit.cloudera.org:8080/23443
Change subject: IMPALA-13472: Patch breakpad to fix minidump stacks for 64KB pages
......................................................................
IMPALA-13472: Patch breakpad to fix minidump stacks for 64KB pages
When resolving minidumps from Redhat8 ARM machines, we found
that the stack traces were truncated, with only the top frame
available. The minidump_stackwalk output included this error
message:
minidump.cc:1391: INFO: MinidumpMemoryRegion request out of range: 0xfffee0aadf30+8/0xfffee0aa0000+0x8000
stackwalker_arm64.cc:dc: ERROR: Unable to read caller_fp from last_fp: 0xfffee0aadf30
The 0x8000 is 32KB, so the minidump contains 32KB of stack, but
that 32KB doesn't cover the memory pointed to by the stack pointer
register.
This is due to this logic in Breakpad's LinuxDumper::GetStackInfo():
// Move the stack pointer to the bottom of the page that it's in.
const uintptr_t page_size = getpagesize();
uint8_t* const stack_pointer =
reinterpret_cast<uint8_t*>(int_stack_pointer & ~(page_size - 1));
When Breakpad moves the stack pointer to the bottom of the page,
it is including unused stack memory (as the stack is top to bottom).
With a 64KB page size, since it only copies 32KB of the stack memory,
this memory usually doesn't contain any used stack memory. This
lines up with the observations from trying to process the minidump.
It is unclear why Breakpad moves the stack pointer to the bottom
of the page to include unused stack memory. There is a 128 byte
red zone on x86_64 which leaf functions can use without changing
the stack pointer, but that does not require the whole page (and
the rounding logic would not include it when near the boundary).
This changes the logic to explicitly include the 128 byte red
zone (respecting the boundary of the memory region). It then
rounds off the stack pointer to a 1KB boundary rather than a
whole OS page. This may not be necessary, but it is mostly
harmless and can be optimized more later.
Testing:
- Linked Impala with the modified Breakpad, sent SIGUSR1 to produce
a minidump, then resolved the minidump. ARM64 with 64KB pages
produces stack traces. x86_64 also continues to work.
Change-Id: Iaf19523e69628e78071636292b3cf464a7b626b8
---
M buildall.sh
A source/breakpad/breakpad-e09741c609dcd5f5274d40182c5e2cc9a002d5ba-patches/0003-Fix-stack-collection-for-64KB-page-size.patch
2 files changed, 44 insertions(+), 1 deletion(-)
git pull ssh://gerrit.cloudera.org:29418/native-toolchain refs/changes/43/23443/1
--
To view, visit http://gerrit.cloudera.org:8080/23443
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: native-toolchain
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: Iaf19523e69628e78071636292b3cf464a7b626b8
Gerrit-Change-Number: 23443
Gerrit-PatchSet: 1
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
[native-toolchain-CR] IMPALA-13472: Patch breakpad to fix minidump stacks for 64KB pages
Posted by "Joe McDonnell (Code Review)" <ge...@cloudera.org>.
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/23443 )
Change subject: IMPALA-13472: Patch breakpad to fix minidump stacks for 64KB pages
......................................................................
Patch Set 2:
I tested Redhat8 ARM64 on a couple builds (release, debug, ASAN) and minidump stacks are fine.
--
To view, visit http://gerrit.cloudera.org:8080/23443
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: native-toolchain
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaf19523e69628e78071636292b3cf464a7b626b8
Gerrit-Change-Number: 23443
Gerrit-PatchSet: 2
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Zoltan Martonka <zm...@cloudera.com>
Gerrit-Comment-Date: Wed, 24 Sep 2025 18:12:15 +0000
Gerrit-HasComments: No
[native-toolchain-CR] IMPALA-13472: Patch breakpad to fix minidump stacks for 64KB pages
Posted by "Joe McDonnell (Code Review)" <ge...@cloudera.org>.
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/23443 )
Change subject: IMPALA-13472: Patch breakpad to fix minidump stacks for 64KB pages
......................................................................
Patch Set 1:
> To my best understanding, 2 things happen here:
> + We round the pointer to whole pages, so we look up the starting
> address in FindMapping (which should not make a difference), and we
> also copy whole memory pages.
> + We maximize the copy to 32k to avoid overly large stack traces.
>
> I do not understand from the code or the git log why we need whole
> pages, but wouldn't setting the upper limit to kStackToCapture =
> max(32 * 1024, page_size), to ensure that we still copy whole
> pages, make more sense? (Because that is what the code appears to
> be doing, although there is no reasoning given for it).
I tried that approach and it functions properly, but it more than doubles the size of the minidump in most cases. That's what motivated me to pursue a more complicated approach.
Breakpad does have logic to limit the size of a minidump, but it doesn't kick in until there are a lot of threads. Here is how it works:
1. It calculates the expected size of the stacks based on how many threads there are. It does this by multiplying the number of threads by 8KB (a hard coded constant).
2. If that expected size exceeds the limit, it collects the stack regularly for 20 threads (and always the crashing thread). Then, it limits the stack to 2KB for the remaining threads.
3. The logic that limits the stack size to 2KB does understand that the leading memory before the stack pointer is useless and skips it (i.e. it advances the stack pointer by 2KB chunks until it is at the 2KB boundary before the stack pointer).
For Impala and Kudu, the limit is 20480KB, so the logic kicks in at 20480 / 8 = 2560 threads. I think a lot of minidumps have fewer than 2560 threads.
It's not a big deal if we want a simpler approach, but I don't think any of that extra memory is useful (other than the red zone). From what I can tell, Crashpad, the successor to Breakpad, doesn't include extra stack memory other than the red zone.
--
To view, visit http://gerrit.cloudera.org:8080/23443
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: native-toolchain
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaf19523e69628e78071636292b3cf464a7b626b8
Gerrit-Change-Number: 23443
Gerrit-PatchSet: 1
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Zoltan Martonka <zm...@cloudera.com>
Gerrit-Comment-Date: Mon, 22 Sep 2025 18:45:28 +0000
Gerrit-HasComments: No
[native-toolchain-CR] IMPALA-13472: Patch breakpad to fix minidump stacks for 64KB pages
Posted by "Joe McDonnell (Code Review)" <ge...@cloudera.org>.
Joe McDonnell has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/23443 )
Change subject: IMPALA-13472: Patch breakpad to fix minidump stacks for 64KB pages
......................................................................
IMPALA-13472: Patch breakpad to fix minidump stacks for 64KB pages
When resolving minidumps from Redhat8 ARM machines, we found
that the stack traces were truncated, with only the top frame
available. The minidump_stackwalk output included this error
message:
minidump.cc:1391: INFO: MinidumpMemoryRegion request out of range: 0xfffee0aadf30+8/0xfffee0aa0000+0x8000
stackwalker_arm64.cc:dc: ERROR: Unable to read caller_fp from last_fp: 0xfffee0aadf30
The 0x8000 is 32KB, so the minidump contains 32KB of stack, but
that 32KB doesn't cover the memory pointed to by the stack pointer
register.
This is due to this logic in Breakpad's LinuxDumper::GetStackInfo():
// Move the stack pointer to the bottom of the page that it's in.
const uintptr_t page_size = getpagesize();
uint8_t* const stack_pointer =
reinterpret_cast<uint8_t*>(int_stack_pointer & ~(page_size - 1));
When Breakpad moves the stack pointer to the bottom of the page,
it is including unused stack memory (as the stack is top to bottom).
With a 64KB page size, since it only copies 32KB of the stack memory,
this memory usually doesn't contain any used stack memory. This
lines up with the observations from trying to process the minidump.
It is unclear why Breakpad moves the stack pointer to the bottom
of the page to include unused stack memory. There is a 128 byte
red zone on x86_64 which leaf functions can use without changing
the stack pointer, but that does not require the whole page (and
the rounding logic would not include it when near the boundary).
This changes the logic to explicitly include the 128 byte red
zone (respecting the boundary of the memory region). It then
rounds off the stack pointer to a 1KB boundary rather than a
whole OS page. This may not be necessary, but it is mostly
harmless and can be optimized more later.
Testing:
- Linked Impala with the modified Breakpad, sent SIGUSR1 to produce
a minidump, then resolved the minidump. ARM64 with 64KB pages
produces stack traces. x86_64 also continues to work.
Change-Id: Iaf19523e69628e78071636292b3cf464a7b626b8
Reviewed-on: http://gerrit.cloudera.org:8080/23443
Reviewed-by: Michael Smith <mi...@cloudera.com>
Tested-by: Joe McDonnell <jo...@cloudera.com>
---
M buildall.sh
A source/breakpad/breakpad-e09741c609dcd5f5274d40182c5e2cc9a002d5ba-patches/0003-Fix-stack-collection-for-64KB-page-size.patch
2 files changed, 44 insertions(+), 1 deletion(-)
Approvals:
Michael Smith: Looks good to me, approved
Joe McDonnell: Verified
--
To view, visit http://gerrit.cloudera.org:8080/23443
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: native-toolchain
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: Iaf19523e69628e78071636292b3cf464a7b626b8
Gerrit-Change-Number: 23443
Gerrit-PatchSet: 3
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Zoltan Martonka <zm...@cloudera.com>
[native-toolchain-CR] IMPALA-13472: Patch breakpad to fix minidump stacks for 64KB pages
Posted by "Joe McDonnell (Code Review)" <ge...@cloudera.org>.
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/23443 )
Change subject: IMPALA-13472: Patch breakpad to fix minidump stacks for 64KB pages
......................................................................
Patch Set 2: Verified+1
This has built successfully and works with Impala, so merging
--
To view, visit http://gerrit.cloudera.org:8080/23443
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: native-toolchain
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaf19523e69628e78071636292b3cf464a7b626b8
Gerrit-Change-Number: 23443
Gerrit-PatchSet: 2
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Zoltan Martonka <zm...@cloudera.com>
Gerrit-Comment-Date: Wed, 24 Sep 2025 18:25:54 +0000
Gerrit-HasComments: No
[native-toolchain-CR] IMPALA-13472: Patch breakpad to fix minidump stacks for 64KB pages
Posted by "Zoltan Martonka (Code Review)" <ge...@cloudera.org>.
Zoltan Martonka has posted comments on this change. ( http://gerrit.cloudera.org:8080/23443 )
Change subject: IMPALA-13472: Patch breakpad to fix minidump stacks for 64KB pages
......................................................................
Patch Set 1:
To my best understanding, 2 things happen here:
+ We round the pointer to whole pages, so we look up the starting address in FindMapping (which should not make a difference), and we also copy whole memory pages.
+ We maximize the copy to 32k to avoid overly large stack traces.
I do not understand from the code or the git log why we need whole pages, but wouldn't setting the upper limit to kStackToCapture = max(32 * 1024, page_size), to ensure that we still copy whole pages, make more sense? (Because that is what the code appears to be doing, although there is no reasoning given for it).
--
To view, visit http://gerrit.cloudera.org:8080/23443
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: native-toolchain
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaf19523e69628e78071636292b3cf464a7b626b8
Gerrit-Change-Number: 23443
Gerrit-PatchSet: 1
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Zoltan Martonka <zm...@cloudera.com>
Gerrit-Comment-Date: Mon, 22 Sep 2025 11:44:05 +0000
Gerrit-HasComments: No
[native-toolchain-CR] IMPALA-13472: Patch breakpad to fix minidump stacks for 64KB pages
Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/23443 )
Change subject: IMPALA-13472: Patch breakpad to fix minidump stacks for 64KB pages
......................................................................
Patch Set 2: Code-Review+1
--
To view, visit http://gerrit.cloudera.org:8080/23443
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: native-toolchain
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaf19523e69628e78071636292b3cf464a7b626b8
Gerrit-Change-Number: 23443
Gerrit-PatchSet: 2
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Zoltan Martonka <zm...@cloudera.com>
Gerrit-Comment-Date: Wed, 24 Sep 2025 17:42:51 +0000
Gerrit-HasComments: No
[native-toolchain-CR] IMPALA-13472: Patch breakpad to fix minidump stacks for 64KB pages
Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/23443 )
Change subject: IMPALA-13472: Patch breakpad to fix minidump stacks for 64KB pages
......................................................................
Patch Set 2: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/23443
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: native-toolchain
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaf19523e69628e78071636292b3cf464a7b626b8
Gerrit-Change-Number: 23443
Gerrit-PatchSet: 2
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Zoltan Martonka <zm...@cloudera.com>
Gerrit-Comment-Date: Wed, 24 Sep 2025 18:13:19 +0000
Gerrit-HasComments: No
[native-toolchain-CR] IMPALA-13472: Patch breakpad to fix minidump stacks for 64KB pages
Posted by "Joe McDonnell (Code Review)" <ge...@cloudera.org>.
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/23443 )
Change subject: IMPALA-13472: Patch breakpad to fix minidump stacks for 64KB pages
......................................................................
Patch Set 2:
I built Impala on Ubuntu 20 x86_64 with a toolchain build from the previous upload and tested resolving minidumps on various builds (debug, release, ASAN, UBSAN). I'll do the same for Redhat8 ARM64 (but the logic is basically the same). If things are looking ok, I plan to submit this and get the Impala patch up today.
--
To view, visit http://gerrit.cloudera.org:8080/23443
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: native-toolchain
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaf19523e69628e78071636292b3cf464a7b626b8
Gerrit-Change-Number: 23443
Gerrit-PatchSet: 2
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Zoltan Martonka <zm...@cloudera.com>
Gerrit-Comment-Date: Wed, 24 Sep 2025 17:31:04 +0000
Gerrit-HasComments: No
[native-toolchain-CR] IMPALA-13472: Patch breakpad to fix minidump stacks for 64KB pages
Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/23443 )
Change subject: IMPALA-13472: Patch breakpad to fix minidump stacks for 64KB pages
......................................................................
Patch Set 1: Code-Review+1
(1 comment)
http://gerrit.cloudera.org:8080/#/c/23443/1//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/23443/1//COMMIT_MSG@31
PS1, Line 31: It is unclear why Breakpad moves the stack pointer to the bottom
According to https://chromium.googlesource.com/breakpad/breakpad/+/master/docs/getting_started_with_breakpad.md, the minidump format may have originated on Windows, and Windows doesn't have the 128 byte red zone according to https://en.wikipedia.org/wiki/Red_zone_(computing).
--
To view, visit http://gerrit.cloudera.org:8080/23443
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: native-toolchain
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaf19523e69628e78071636292b3cf464a7b626b8
Gerrit-Change-Number: 23443
Gerrit-PatchSet: 1
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Zoltan Martonka <zm...@cloudera.com>
Gerrit-Comment-Date: Mon, 22 Sep 2025 22:52:07 +0000
Gerrit-HasComments: Yes
[native-toolchain-CR] IMPALA-13472: Patch breakpad to fix minidump stacks for 64KB pages
Posted by "Zoltan Martonka (Code Review)" <ge...@cloudera.org>.
Zoltan Martonka has posted comments on this change. ( http://gerrit.cloudera.org:8080/23443 )
Change subject: IMPALA-13472: Patch breakpad to fix minidump stacks for 64KB pages
......................................................................
Patch Set 1: Code-Review+1
> > To my best understanding, 2 things happen here:
> > + We round the pointer to whole pages, so we look up the starting
> > address in FindMapping (which should not make a difference), and
> we
> > also copy whole memory pages.
> > + We maximize the copy to 32k to avoid overly large stack traces.
> >
> > I do not understand from the code or the git log why we need
> whole
> > pages, but wouldn't setting the upper limit to kStackToCapture =
> > max(32 * 1024, page_size), to ensure that we still copy whole
> > pages, make more sense? (Because that is what the code appears to
> > be doing, although there is no reasoning given for it).
>
> I tried that approach and it functions properly, but it more than
> doubles the size of the minidump in most cases. That's what
> motivated me to pursue a more complicated approach.
>
> Breakpad does have logic to limit the size of a minidump, but it
> doesn't kick in until there are a lot of threads. Here is how it
> works:
> 1. It calculates the expected size of the stacks based on how many
> threads there are. It does this by multiplying the number of
> threads by 8KB (a hard coded constant).
> 2. If that expected size exceeds the limit, it collects the stack
> regularly for 20 threads (and always the crashing thread). Then, it
> limits the stack to 2KB for the remaining threads.
> 3. The logic that limits the stack size to 2KB does understand that
> the leading memory before the stack pointer is useless and skips it
> (i.e. it advances the stack pointer by 2KB chunks until it is at
> the 2KB boundary before the stack pointer).
>
> For Impala and Kudu, the limit is 20480KB, so the logic kicks in at
> 20480 / 8 = 2560 threads. I think a lot of minidumps have fewer
> than 2560 threads.
>
> It's not a big deal if we want a simpler approach, but I don't
> think any of that extra memory is useful (other than the red zone).
> From what I can tell, Crashpad, the successor to Breakpad, doesn't
> include extra stack memory other than the red zone.
Thank you
--
To view, visit http://gerrit.cloudera.org:8080/23443
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: native-toolchain
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaf19523e69628e78071636292b3cf464a7b626b8
Gerrit-Change-Number: 23443
Gerrit-PatchSet: 1
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Zoltan Martonka <zm...@cloudera.com>
Gerrit-Comment-Date: Tue, 23 Sep 2025 11:12:19 +0000
Gerrit-HasComments: No