You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@opendal.apache.org by "aapoalas (via GitHub)" <gi...@apache.org> on 2025/08/29 13:40:32 UTC
Possible problem with experimental USDT / DTrace probes
GitHub user aapoalas created a discussion: Possible problem with experimental USDT / DTrace probes
Hello
It seems to me like the USDTs that you're generating might be turning up unsafe/unsound to trace in some circumstances. I just spent one and a half days wrestling with this in another project (and using a different crate than probe-rs to create the USDTs), and I noticed opendal's documentation shows similar (possible?) issues and figured I'd come in for a checkup.
Basically, the output `readelf` output given in the `DtraceLayer`'s documentation looks like this:
```text
stapsdt 0x00000039 NT_STAPSDT (SystemTap probe descriptors)
Provider: opendal
Name: create_dir_start
Location: 0x00000000000f8f05, Base: 0x0000000000000000, Semaphore: 0x00000000003649f8
Arguments: -8@%rax
```
The "Address" is the address of the `nop` instruction (the probe), and "Semaphore" is the address of the 16 byte value used to guard entry to the probe. But what about the "Base"? That is supposed to be the address of the `_.stapsdt.base` section, which is used ["to detect prelink address adjustments"](https://sourceware.org/systemtap/wiki/UserSpaceProbeImplementation) (see [prelink address adjustments](https://lwn.net/Articles/190139/) if you want to understand this fully; I do not): it being zero means that someone (the linker, I believe) decided that the section is unused, removed it, and replaced references to it with a zero. This then means that at tracing time prelink address adjustments cannot be detected and if such things have happened, then the probe and/or semaphore addresses will remain unadjusted. This (presumably; I'm not 100% sure on this) means that instead of changing the `nop` to an interrupt, some other location in the loaded executable gets turned into an interrup
t unexpectedly. Similarly, I believe with more conviction, the semaphore increment and decrement will write to some other part of the program.
I was having crazy kind of segmentation faults in my own project during tracing, and at the end of the day I noticed that my linker (mold) was GC'ing the base section away and leaving me without base addresses. Changing to the standard linker left the base section in the executable, retaining my base addresses, and fixing the segmentation faults.
Now: even if you have these issues in opendal's `DtraceLayer`, it's not a bug in the library. You cannot reasonably do much about this (unless you want to introduce a spurious read of `_.stapsdt.base` or otherwise usage of its address into the code somewhere), but I figured it might be worth a warning since your documentation does show this pattern. At most you'll possibly want to check your readelf output and linker usage, change the documentation, and possibly warn users that they should make sure that their executables have base addresses set.
GitHub link: https://github.com/apache/opendal/discussions/6537
----
This is an automatically sent email for dev@opendal.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@opendal.apache.org
Re: Possible problem with experimental USDT / DTrace probes
Posted by "aapoalas (via GitHub)" <gi...@apache.org>.
GitHub user aapoalas added a comment to the discussion: Possible problem with experimental USDT / DTrace probes
Ah, there actually may also be something that you could do (or probe-rs) could do; currently in talks with mold linker developer on how to properly do this: https://github.com/rui314/mold/issues/1510
GitHub link: https://github.com/apache/opendal/discussions/6537#discussioncomment-14256430
----
This is an automatically sent email for dev@opendal.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@opendal.apache.org
Re: Possible problem with experimental USDT / DTrace probes
Posted by "yihong0618 (via GitHub)" <gi...@apache.org>.
GitHub user yihong0618 added a comment to the discussion: Possible problem with experimental USDT / DTrace probes
cc @Zheaoli you may have interest about this
GitHub link: https://github.com/apache/opendal/discussions/6537#discussioncomment-14256335
----
This is an automatically sent email for dev@opendal.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@opendal.apache.org
Re: Possible problem with experimental USDT / DTrace probes
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
GitHub user Xuanwo added a comment to the discussion: Possible problem with experimental USDT / DTrace probes
Wow, thanks a lot for raising this. I will take a look along with @Zheaoli who is the code owner of this layer.
GitHub link: https://github.com/apache/opendal/discussions/6537#discussioncomment-14256400
----
This is an automatically sent email for dev@opendal.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@opendal.apache.org