You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@celix.apache.org by "pnoltes (via GitHub)" <gi...@apache.org> on 2025/08/14 14:03:51 UTC
[PR] Feature/add fuzzing (celix)
pnoltes opened a new pull request, #799:
URL: https://github.com/apache/celix/pull/799
This PR introduces some fuzz tests for celix utils.
Also:
- Aligns enable_fuzzing and enable_benchmarks option
- Updates conan package id so that test, benchmarks, etc options are not used for the package id.
- Fixes an issues where the jar command was chosen to make bundles if jar and zip is available, even if the "use zip" option was enable
- Update building document for testing, benchmarking and fuzzing
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@celix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Feature/add fuzzing (celix)
Posted by "pnoltes (via GitHub)" <gi...@apache.org>.
pnoltes merged PR #799:
URL: https://github.com/apache/celix/pull/799
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@celix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Feature/add fuzzing (celix)
Posted by "PengZheng (via GitHub)" <gi...@apache.org>.
PengZheng commented on code in PR #799:
URL: https://github.com/apache/celix/pull/799#discussion_r2619044291
##########
.github/workflows/fuzzing.yml:
##########
@@ -0,0 +1,59 @@
+name: Celix Fuzzing
+
+on:
+ push:
+ pull_request:
+ schedule:
+ - cron: '0 3 * * *'
+
+jobs:
+ fuzz-utils:
+ runs-on: ubuntu-22.04
+ timeout-minutes: 30
+ steps:
+ - name: Checkout source code
+ uses: actions/checkout@ac593985615ec2ede58e132d2e21d2b1cbd6127c #v3.3.0
+ - name: Set up Python
+ uses: actions/setup-python@7f4fc3e22c37d6ff65e88745f38bd3157c663f7c #v4.9.1
+ with:
+ python-version: '3.x'
+ - name: Set Compiler Environment Variables
+ run: |
+ echo "CC=clang" >> $GITHUB_ENV
+ echo "CXX=clang++" >> $GITHUB_ENV
+ - name: Install Conan
+ run: pip install conan
+ - name: Cache Conan
+ uses: actions/cache@v3
+ with:
+ path: ~/.conan2/p
+ key: ${{ runner.os }}-conan-${{ hashFiles('conanfile.py', 'libs/utils/**') }}
+ restore-keys: |
+ ${{ runner.os }}-conan-
+ - name: Setup Conan Profile
+ run: |
+ conan profile detect
+ - name: Conan install
+ run: conan install . --output-folder=build --build=missing -o "celix/*:build_utils=True" -o "celix/*:enable_fuzzing=True"
+ - name: Conan build
+ run: conan build . --output-folder=build -o "celix/*:build_utils=True" -o "celix/*:enable_fuzzing=True" -o "celix/*:celix_err_buffer_size=5120"
Review Comment:
Here is a minimal reproducer:
```C++
// myfuzzer.cpp
#include <stddef.h>
#include <stdint.h>
#include <stdio.h>
#include <stdlib.h>
// Function to test (example: a simple vulnerable function)
void my_api_function(const uint8_t* data, size_t size) {
if (size > 0 && data[0] == 'B') {
if (size > 1 && data[1] == 'U') {
if (size > 2 && data[2] == 'G') {
char* p = NULL;
*p = 1; // This will cause a crash (dereferencing NULL)
}
}
}
}
// The libFuzzer entry point
extern "C" int LLVMFuzzerTestOneInput(const uint8_t* data, size_t size) {
malloc(0);
my_api_function(data, size);
return 0;
}
```
```Bash
$ clang++ -g -shared-libasan -fsanitize=address,fuzzer myfuzzer.cpp -o myfuzzer_asan -Wl,-rpath,/usr/lib/llvm-18/lib/clang/18/lib/linux/
$ ./myfuzzer_asan
==79772==ASan runtime does not come first in initial library list; you should either link runtime to your application or manually preload it with LD_PRELOAD.
```
Removing `-shared-libasan` fixes the issue:
```Bash
$ clang++ -g -fsanitize=address,fuzzer myfuzzer.cpp -o myfuzzer_asan -Wl,-rpath,/usr/lib/llvm-18/lib/clang/18/lib/linux/
$ ./myfuzzer_asan
INFO: Running with entropic power schedule (0xFF, 100).
INFO: Seed: 3170674705
INFO: Loaded 1 modules (9 inline 8-bit counters): 9 [0x57509822be88, 0x57509822be91),
INFO: Loaded 1 PC tables (9 PCs): 9 [0x57509822be98,0x57509822bf28),
INFO: -max_len is not provided; libFuzzer will not generate inputs larger than 4096 bytes
INFO: A corpus is not provided, starting from an empty corpus
#2 INITED cov: 3 ft: 3 corp: 1/1b exec/s: 0 rss: 31Mb
=================================================================
==79832==ERROR: LeakSanitizer: detected memory leaks
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@celix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Feature/add fuzzing (celix)
Posted by "PengZheng (via GitHub)" <gi...@apache.org>.
PengZheng commented on code in PR #799:
URL: https://github.com/apache/celix/pull/799#discussion_r2618361589
##########
.github/workflows/fuzzing.yml:
##########
@@ -0,0 +1,59 @@
+name: Celix Fuzzing
+
+on:
+ push:
+ pull_request:
+ schedule:
+ - cron: '0 3 * * *'
+
+jobs:
+ fuzz-utils:
+ runs-on: ubuntu-22.04
+ timeout-minutes: 30
+ steps:
+ - name: Checkout source code
+ uses: actions/checkout@ac593985615ec2ede58e132d2e21d2b1cbd6127c #v3.3.0
+ - name: Set up Python
+ uses: actions/setup-python@7f4fc3e22c37d6ff65e88745f38bd3157c663f7c #v4.9.1
+ with:
+ python-version: '3.x'
+ - name: Set Compiler Environment Variables
+ run: |
+ echo "CC=clang" >> $GITHUB_ENV
+ echo "CXX=clang++" >> $GITHUB_ENV
+ - name: Install Conan
+ run: pip install conan
+ - name: Cache Conan
+ uses: actions/cache@v3
+ with:
+ path: ~/.conan2/p
+ key: ${{ runner.os }}-conan-${{ hashFiles('conanfile.py', 'libs/utils/**') }}
+ restore-keys: |
+ ${{ runner.os }}-conan-
+ - name: Setup Conan Profile
+ run: |
+ conan profile detect
+ - name: Conan install
+ run: conan install . --output-folder=build --build=missing -o "celix/*:build_utils=True" -o "celix/*:enable_fuzzing=True"
+ - name: Conan build
+ run: conan build . --output-folder=build -o "celix/*:build_utils=True" -o "celix/*:enable_fuzzing=True" -o "celix/*:celix_err_buffer_size=5120"
Review Comment:
Hmm, `RPATH` should be enough. I will have a look.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@celix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Feature/add fuzzing (celix)
Posted by "PengZheng (via GitHub)" <gi...@apache.org>.
PengZheng commented on code in PR #799:
URL: https://github.com/apache/celix/pull/799#discussion_r2619044291
##########
.github/workflows/fuzzing.yml:
##########
@@ -0,0 +1,59 @@
+name: Celix Fuzzing
+
+on:
+ push:
+ pull_request:
+ schedule:
+ - cron: '0 3 * * *'
+
+jobs:
+ fuzz-utils:
+ runs-on: ubuntu-22.04
+ timeout-minutes: 30
+ steps:
+ - name: Checkout source code
+ uses: actions/checkout@ac593985615ec2ede58e132d2e21d2b1cbd6127c #v3.3.0
+ - name: Set up Python
+ uses: actions/setup-python@7f4fc3e22c37d6ff65e88745f38bd3157c663f7c #v4.9.1
+ with:
+ python-version: '3.x'
+ - name: Set Compiler Environment Variables
+ run: |
+ echo "CC=clang" >> $GITHUB_ENV
+ echo "CXX=clang++" >> $GITHUB_ENV
+ - name: Install Conan
+ run: pip install conan
+ - name: Cache Conan
+ uses: actions/cache@v3
+ with:
+ path: ~/.conan2/p
+ key: ${{ runner.os }}-conan-${{ hashFiles('conanfile.py', 'libs/utils/**') }}
+ restore-keys: |
+ ${{ runner.os }}-conan-
+ - name: Setup Conan Profile
+ run: |
+ conan profile detect
+ - name: Conan install
+ run: conan install . --output-folder=build --build=missing -o "celix/*:build_utils=True" -o "celix/*:enable_fuzzing=True"
+ - name: Conan build
+ run: conan build . --output-folder=build -o "celix/*:build_utils=True" -o "celix/*:enable_fuzzing=True" -o "celix/*:celix_err_buffer_size=5120"
Review Comment:
Here is a minimal reproducer:
```C++
// myfuzzer.cpp
#include <stddef.h>
#include <stdint.h>
#include <stdio.h>
#include <stdlib.h>
// Function to test (example: a simple vulnerable function)
void my_api_function(const uint8_t* data, size_t size) {
if (size > 0 && data[0] == 'B') {
if (size > 1 && data[1] == 'U') {
if (size > 2 && data[2] == 'G') {
char* p = NULL;
*p = 1; // This will cause a crash (dereferencing NULL)
}
}
}
}
// The libFuzzer entry point
extern "C" int LLVMFuzzerTestOneInput(const uint8_t* data, size_t size) {
malloc(0);
my_api_function(data, size);
return 0;
}
```
```Bash
$ clang++ -g -shared-libasan -fsanitize=address,fuzzer myfuzzer.cpp -o myfuzzer_asan -Wl,-rpath,/usr/lib/llvm-18/lib/clang/18/lib/linux/
$ ./myfuzzer_asan
==79772==ASan runtime does not come first in initial library list; you should either link runtime to your application or manually preload it with LD_PRELOAD.
```
Removing `-shared-libasan` fixes the issue:
```Bash
$ clang++ -g -fsanitize=address,fuzzer myfuzzer.cpp -o myfuzzer_asan -Wl,-rpath,/usr/lib/llvm-18/lib/clang/18/lib/linux/
$ ./myfuzzer_asan
INFO: Running with entropic power schedule (0xFF, 100).
INFO: Seed: 3170674705
INFO: Loaded 1 modules (9 inline 8-bit counters): 9 [0x57509822be88, 0x57509822be91),
INFO: Loaded 1 PC tables (9 PCs): 9 [0x57509822be98,0x57509822bf28),
INFO: -max_len is not provided; libFuzzer will not generate inputs larger than 4096 bytes
INFO: A corpus is not provided, starting from an empty corpus
#2 INITED cov: 3 ft: 3 corp: 1/1b exec/s: 0 rss: 31Mb
=================================================================
==79832==ERROR: LeakSanitizer: detected memory leaks
```
IIRC, `-shared-libasan` is needed for error injector.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@celix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Feature/add fuzzing (celix)
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #799:
URL: https://github.com/apache/celix/pull/799#issuecomment-3188614000
## [Codecov](https://app.codecov.io/gh/apache/celix/pull/799?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:white_check_mark: All modified and coverable lines are covered by tests.
:white_check_mark: Project coverage is 91.45%. Comparing base ([`d1ad26b`](https://app.codecov.io/gh/apache/celix/commit/d1ad26b221545d60139b85a0eaa8ef02ac6141d2?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`1273863`](https://app.codecov.io/gh/apache/celix/commit/12738634e522390910da7bf0b267287400d1de70?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #799 +/- ##
==========================================
- Coverage 91.46% 91.45% -0.02%
==========================================
Files 236 236
Lines 29544 29544
==========================================
- Hits 27023 27019 -4
- Misses 2521 2525 +4
```
</details>
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/celix/pull/799?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@celix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Feature/add fuzzing (celix)
Posted by "pnoltes (via GitHub)" <gi...@apache.org>.
pnoltes commented on code in PR #799:
URL: https://github.com/apache/celix/pull/799#discussion_r2373031315
##########
libs/utils/fuzzing/src/FilterFuzz.cc:
##########
@@ -0,0 +1,23 @@
+#include <celix_filter.h>
+#include <celix_err.h>
+#include <cstdint>
+#include <cstdlib>
+#include <cstring>
+
+int filterParseFuzzOneInput(const uint8_t* data, size_t size) {
+ char* buffer = static_cast<char*>(malloc(size + 1));
+ if (buffer == nullptr) {
+ return 0;
+ }
+ memcpy(buffer, data, size);
+ buffer[size] = '\0';
+
+ celix_filter_t* filter = celix_filter_create(buffer);
Review Comment:
Good point, I will look into add filter match as fuzz test entry.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@celix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Feature/add fuzzing (celix)
Posted by "PengZheng (via GitHub)" <gi...@apache.org>.
PengZheng commented on code in PR #799:
URL: https://github.com/apache/celix/pull/799#discussion_r2358889819
##########
.github/workflows/fuzzing.yml:
##########
@@ -0,0 +1,59 @@
+name: Celix Fuzzing
+
+on:
+ push:
+ pull_request:
+ schedule:
+ - cron: '0 3 * * *'
+
+jobs:
+ fuzz-utils:
+ runs-on: ubuntu-22.04
+ timeout-minutes: 30
+ steps:
+ - name: Checkout source code
+ uses: actions/checkout@ac593985615ec2ede58e132d2e21d2b1cbd6127c #v3.3.0
+ - name: Set up Python
+ uses: actions/setup-python@7f4fc3e22c37d6ff65e88745f38bd3157c663f7c #v4.9.1
+ with:
+ python-version: '3.x'
+ - name: Set Compiler Environment Variables
+ run: |
+ echo "CC=clang" >> $GITHUB_ENV
+ echo "CXX=clang++" >> $GITHUB_ENV
+ - name: Install Conan
+ run: pip install conan
+ - name: Cache Conan
+ uses: actions/cache@v3
+ with:
+ path: ~/.conan2/p
+ key: ${{ runner.os }}-conan-${{ hashFiles('conanfile.py', 'libs/utils/**') }}
+ restore-keys: |
+ ${{ runner.os }}-conan-
+ - name: Setup Conan Profile
+ run: |
+ conan profile detect
+ - name: Conan install
+ run: conan install . --output-folder=build --build=missing -o "celix/*:build_utils=True" -o "celix/*:enable_fuzzing=True"
+ - name: Conan build
+ run: conan build . --output-folder=build -o "celix/*:build_utils=True" -o "celix/*:enable_fuzzing=True" -o "celix/*:celix_err_buffer_size=5120"
+ - name: Set fuzzer run time
+ id: set-runtime
+ run: |
+ if [[ "${{ github.event_name }}" == "schedule" ]]; then
+ echo "FUZZ_TIME=600" >> ${GITHUB_ENV}
+ else
+ echo "FUZZ_TIME=30" >> ${GITHUB_ENV}
+ fi
+ - name: Run properties fuzzer
+ run: |
+ source build/conanrun.sh
+ ./build/libs/utils/fuzzing/celix_properties_fuzzer -max_total_time=$FUZZ_TIME ./build/libs/utils/fuzzing/properties_corpus
Review Comment:
Shall we let corpus to be maintained by CI?
1. Commit CI's addition
2. Add `-merge=1`
##########
.github/workflows/fuzzing.yml:
##########
@@ -0,0 +1,59 @@
+name: Celix Fuzzing
+
+on:
+ push:
+ pull_request:
+ schedule:
+ - cron: '0 3 * * *'
+
+jobs:
+ fuzz-utils:
+ runs-on: ubuntu-22.04
+ timeout-minutes: 30
+ steps:
+ - name: Checkout source code
+ uses: actions/checkout@ac593985615ec2ede58e132d2e21d2b1cbd6127c #v3.3.0
+ - name: Set up Python
+ uses: actions/setup-python@7f4fc3e22c37d6ff65e88745f38bd3157c663f7c #v4.9.1
+ with:
+ python-version: '3.x'
+ - name: Set Compiler Environment Variables
+ run: |
+ echo "CC=clang" >> $GITHUB_ENV
+ echo "CXX=clang++" >> $GITHUB_ENV
+ - name: Install Conan
+ run: pip install conan
+ - name: Cache Conan
+ uses: actions/cache@v3
+ with:
+ path: ~/.conan2/p
+ key: ${{ runner.os }}-conan-${{ hashFiles('conanfile.py', 'libs/utils/**') }}
+ restore-keys: |
+ ${{ runner.os }}-conan-
+ - name: Setup Conan Profile
+ run: |
+ conan profile detect
+ - name: Conan install
+ run: conan install . --output-folder=build --build=missing -o "celix/*:build_utils=True" -o "celix/*:enable_fuzzing=True"
Review Comment:
IIRC, for conan 2, this step in superfluous.
If options appear in install step, they don't need to be repeated in the build step.
##########
libs/utils/fuzzing/src/FilterFuzz.cc:
##########
@@ -0,0 +1,23 @@
+#include <celix_filter.h>
+#include <celix_err.h>
+#include <cstdint>
+#include <cstdlib>
+#include <cstring>
+
+int filterParseFuzzOneInput(const uint8_t* data, size_t size) {
+ char* buffer = static_cast<char*>(malloc(size + 1));
+ if (buffer == nullptr) {
+ return 0;
+ }
+ memcpy(buffer, data, size);
+ buffer[size] = '\0';
+
+ celix_filter_t* filter = celix_filter_create(buffer);
Review Comment:
`celix_filter_match` seems another interesting fuzzing target.
##########
.github/workflows/fuzzing.yml:
##########
@@ -0,0 +1,59 @@
+name: Celix Fuzzing
+
+on:
+ push:
+ pull_request:
+ schedule:
+ - cron: '0 3 * * *'
+
+jobs:
+ fuzz-utils:
+ runs-on: ubuntu-22.04
+ timeout-minutes: 30
+ steps:
+ - name: Checkout source code
+ uses: actions/checkout@ac593985615ec2ede58e132d2e21d2b1cbd6127c #v3.3.0
+ - name: Set up Python
+ uses: actions/setup-python@7f4fc3e22c37d6ff65e88745f38bd3157c663f7c #v4.9.1
+ with:
+ python-version: '3.x'
+ - name: Set Compiler Environment Variables
+ run: |
+ echo "CC=clang" >> $GITHUB_ENV
+ echo "CXX=clang++" >> $GITHUB_ENV
+ - name: Install Conan
+ run: pip install conan
+ - name: Cache Conan
+ uses: actions/cache@v3
+ with:
+ path: ~/.conan2/p
+ key: ${{ runner.os }}-conan-${{ hashFiles('conanfile.py', 'libs/utils/**') }}
+ restore-keys: |
+ ${{ runner.os }}-conan-
+ - name: Setup Conan Profile
+ run: |
+ conan profile detect
+ - name: Conan install
+ run: conan install . --output-folder=build --build=missing -o "celix/*:build_utils=True" -o "celix/*:enable_fuzzing=True"
+ - name: Conan build
+ run: conan build . --output-folder=build -o "celix/*:build_utils=True" -o "celix/*:enable_fuzzing=True" -o "celix/*:celix_err_buffer_size=5120"
Review Comment:
How about enabling ASAN and UBSAN altogether?
##########
libs/utils/CMakeLists.txt:
##########
@@ -126,5 +126,24 @@ if (UTILS)
add_subdirectory(gtest)
endif ()
+
+ if (ENABLE_FUZZING)
+ add_library(utils_cuf STATIC ${UTILS_SRC})
+ target_compile_definitions(utils_cuf PRIVATE CELIX_UTILS_STATIC_DEFINE)
+ target_include_directories(utils_cuf PUBLIC
+ ${CMAKE_CURRENT_LIST_DIR}/include
+ ${CMAKE_CURRENT_LIST_DIR}/include_internal
+ ${CMAKE_BINARY_DIR}/celix/gen/includes/utils
+ ${CMAKE_BINARY_DIR}/celix/gen/src/utils
+ include_deprecated
+ )
+ target_link_libraries(utils_cuf PUBLIC ${UTILS_PUBLIC_DEPS} ${UTILS_PRIVATE_DEPS})
+ target_compile_options(utils_cuf PRIVATE -fsanitize=fuzzer-no-link,address)
Review Comment:
Oh, I see ASAN is enabled separately. Note that we have `enable_xxx_sanitizer` conan options.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@celix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Feature/add fuzzing (celix)
Posted by "PengZheng (via GitHub)" <gi...@apache.org>.
PengZheng commented on code in PR #799:
URL: https://github.com/apache/celix/pull/799#discussion_r2347816217
##########
documents/README.md:
##########
@@ -81,8 +81,7 @@ bundles contains binaries depending on the stdlibc++ library.
* Building
* [Building and Installing Apache Celix](building/README.md)
- * [Building and Developing Apache Celix with CLion](building/dev_celix_with_clion.md)
-* C Patterns
+≈* C Patterns
Review Comment:
So shall we stop blessing CLion?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@celix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Feature/add fuzzing (celix)
Posted by "pnoltes (via GitHub)" <gi...@apache.org>.
pnoltes commented on code in PR #799:
URL: https://github.com/apache/celix/pull/799#discussion_r2373025899
##########
.github/workflows/fuzzing.yml:
##########
@@ -0,0 +1,59 @@
+name: Celix Fuzzing
+
+on:
+ push:
+ pull_request:
+ schedule:
+ - cron: '0 3 * * *'
+
+jobs:
+ fuzz-utils:
+ runs-on: ubuntu-22.04
+ timeout-minutes: 30
+ steps:
+ - name: Checkout source code
+ uses: actions/checkout@ac593985615ec2ede58e132d2e21d2b1cbd6127c #v3.3.0
+ - name: Set up Python
+ uses: actions/setup-python@7f4fc3e22c37d6ff65e88745f38bd3157c663f7c #v4.9.1
+ with:
+ python-version: '3.x'
+ - name: Set Compiler Environment Variables
+ run: |
+ echo "CC=clang" >> $GITHUB_ENV
+ echo "CXX=clang++" >> $GITHUB_ENV
+ - name: Install Conan
+ run: pip install conan
+ - name: Cache Conan
+ uses: actions/cache@v3
+ with:
+ path: ~/.conan2/p
+ key: ${{ runner.os }}-conan-${{ hashFiles('conanfile.py', 'libs/utils/**') }}
+ restore-keys: |
+ ${{ runner.os }}-conan-
+ - name: Setup Conan Profile
+ run: |
+ conan profile detect
+ - name: Conan install
+ run: conan install . --output-folder=build --build=missing -o "celix/*:build_utils=True" -o "celix/*:enable_fuzzing=True"
Review Comment:
Ok, good to know. I was not sure and I though that it did not work for me, but I will recheck this. If possible I would like to align the conan builds a bit more (In another PR).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@celix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Feature/add fuzzing (celix)
Posted by "PengZheng (via GitHub)" <gi...@apache.org>.
PengZheng commented on code in PR #799:
URL: https://github.com/apache/celix/pull/799#discussion_r2347816217
##########
documents/README.md:
##########
@@ -81,8 +81,7 @@ bundles contains binaries depending on the stdlibc++ library.
* Building
* [Building and Installing Apache Celix](building/README.md)
- * [Building and Developing Apache Celix with CLion](building/dev_celix_with_clion.md)
-* C Patterns
+≈* C Patterns
Review Comment:
So shall we stop blessing CLion?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@celix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Feature/add fuzzing (celix)
Posted by "PengZheng (via GitHub)" <gi...@apache.org>.
PengZheng commented on code in PR #799:
URL: https://github.com/apache/celix/pull/799#discussion_r2618500374
##########
.github/workflows/fuzzing.yml:
##########
@@ -0,0 +1,59 @@
+name: Celix Fuzzing
+
+on:
+ push:
+ pull_request:
+ schedule:
+ - cron: '0 3 * * *'
+
+jobs:
+ fuzz-utils:
+ runs-on: ubuntu-22.04
+ timeout-minutes: 30
+ steps:
+ - name: Checkout source code
+ uses: actions/checkout@ac593985615ec2ede58e132d2e21d2b1cbd6127c #v3.3.0
+ - name: Set up Python
+ uses: actions/setup-python@7f4fc3e22c37d6ff65e88745f38bd3157c663f7c #v4.9.1
+ with:
+ python-version: '3.x'
+ - name: Set Compiler Environment Variables
+ run: |
+ echo "CC=clang" >> $GITHUB_ENV
+ echo "CXX=clang++" >> $GITHUB_ENV
+ - name: Install Conan
+ run: pip install conan
+ - name: Cache Conan
+ uses: actions/cache@v3
+ with:
+ path: ~/.conan2/p
+ key: ${{ runner.os }}-conan-${{ hashFiles('conanfile.py', 'libs/utils/**') }}
+ restore-keys: |
+ ${{ runner.os }}-conan-
+ - name: Setup Conan Profile
+ run: |
+ conan profile detect
+ - name: Conan install
+ run: conan install . --output-folder=build --build=missing -o "celix/*:build_utils=True" -o "celix/*:enable_fuzzing=True"
+ - name: Conan build
+ run: conan build . --output-folder=build -o "celix/*:build_utils=True" -o "celix/*:enable_fuzzing=True" -o "celix/*:celix_err_buffer_size=5120"
Review Comment:
With ASAN enabled for the clang toolchain, `test_utils` works as expected without `LD_PRELOAD`.
```
(env3) peng@hackerlife2:~/Downloads/git/celix_review/cmake-build-clang/libs/utils/gtest$ readelf -d test_utils
Dynamic section at offset 0x489c80 contains 38 entries:
Tag Type Name/Value
0x0000000000000001 (NEEDED) Shared library: [libclang_rt.asan-x86_64.so]
0x0000000000000001 (NEEDED) Shared library: [libgtest_main.so]
0x0000000000000001 (NEEDED) Shared library: [libzip.so.5]
0x0000000000000001 (NEEDED) Shared library: [libjansson.so.4]
0x0000000000000001 (NEEDED) Shared library: [libcelix_utils.so.3]
0x0000000000000001 (NEEDED) Shared library: [libgtest.so]
0x0000000000000001 (NEEDED) Shared library: [libm.so.6]
0x0000000000000001 (NEEDED) Shared library: [libstdc++.so.6]
0x0000000000000001 (NEEDED) Shared library: [libgcc_s.so.1]
0x0000000000000001 (NEEDED) Shared library: [libc.so.6]
0x000000000000001d (RUNPATH) Library runpath: [/usr/lib/llvm-18/lib/clang/18/lib/linux:/home/peng/.conan2/p/b/libzic7db358af7f13/p/lib:/home/peng/.conan2/p/b/janssdd7698f9ed975/p/lib:/home/peng/.conan2/p/b/gtest070ee13ce63c1/p/lib:/home/peng/Downloads/git/celix_review/cmake-build-clang/libs/utils]
```
`libclang_rt.asan-x86_64.so` comes as the first NEEDED entry. However for `celix_filter_fuzzer`, `libstdc++.so.6` comes before `libclang_rt.asan-x86_64.so`:
```
readelf -d celix_filter_fuzzer
Dynamic section at offset 0xcac00 contains 35 entries:
Tag Type Name/Value
0x0000000000000001 (NEEDED) Shared library: [libstdc++.so.6]
0x0000000000000001 (NEEDED) Shared library: [libclang_rt.asan-x86_64.so]
0x0000000000000001 (NEEDED) Shared library: [libzip.so.5]
0x0000000000000001 (NEEDED) Shared library: [libjansson.so.4]
0x0000000000000001 (NEEDED) Shared library: [libm.so.6]
0x0000000000000001 (NEEDED) Shared library: [libgcc_s.so.1]
0x0000000000000001 (NEEDED) Shared library: [libc.so.6]
0x000000000000001d (RUNPATH) Library runpath: [/usr/lib/llvm-18/lib/clang/18/lib/linux:/home/peng/.conan2/p/b/libzic7db358af7f13/p/lib:/home/peng/.conan2/p/b/janssdd7698f9ed975/p/lib]
```
Double-check with `LD_DEBUG=libs`:
```
LD_DEBUG=libs ./celix_filter_fuzzer
50646: find library=libstdc++.so.6 [0]; searching
50646: search path=/usr/lib/llvm-18/lib/clang/18/lib/linux/glibc-hwcaps/x86-64-v4:/usr/lib/llvm-18/lib/clang/18/lib/linux/glibc-hwcaps/x86-64-v3:/usr/lib/llvm-18/lib/clang/18/lib/linux/glibc-hwcaps/x86-64-v2:/usr/lib/llvm-18/lib/clang/18/lib/linux:/home/peng/.conan2/p/b/libzic7db358af7f13/p/lib/glibc-hwcaps/x86-64-v4:/home/peng/.conan2/p/b/libzic7db358af7f13/p/lib/glibc-hwcaps/x86-64-v3:/home/peng/.conan2/p/b/libzic7db358af7f13/p/lib/glibc-hwcaps/x86-64-v2:/home/peng/.conan2/p/b/libzic7db358af7f13/p/lib:/home/peng/.conan2/p/b/janssdd7698f9ed975/p/lib/glibc-hwcaps/x86-64-v4:/home/peng/.conan2/p/b/janssdd7698f9ed975/p/lib/glibc-hwcaps/x86-64-v3:/home/peng/.conan2/p/b/janssdd7698f9ed975/p/lib/glibc-hwcaps/x86-64-v2:/home/peng/.conan2/p/b/janssdd7698f9ed975/p/lib (RUNPATH from file ./celix_filter_fuzzer)
...
==50646==ASan runtime does not come first in initial library list; you should either link runtime to your application or manually preload it with LD_PRELOAD.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@celix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Feature/add fuzzing (celix)
Posted by "pnoltes (via GitHub)" <gi...@apache.org>.
pnoltes commented on code in PR #799:
URL: https://github.com/apache/celix/pull/799#discussion_r2617383419
##########
.github/workflows/fuzzing.yml:
##########
@@ -0,0 +1,59 @@
+name: Celix Fuzzing
+
+on:
+ push:
+ pull_request:
+ schedule:
+ - cron: '0 3 * * *'
+
+jobs:
+ fuzz-utils:
+ runs-on: ubuntu-22.04
+ timeout-minutes: 30
+ steps:
+ - name: Checkout source code
+ uses: actions/checkout@ac593985615ec2ede58e132d2e21d2b1cbd6127c #v3.3.0
+ - name: Set up Python
+ uses: actions/setup-python@7f4fc3e22c37d6ff65e88745f38bd3157c663f7c #v4.9.1
+ with:
+ python-version: '3.x'
+ - name: Set Compiler Environment Variables
+ run: |
+ echo "CC=clang" >> $GITHUB_ENV
+ echo "CXX=clang++" >> $GITHUB_ENV
+ - name: Install Conan
+ run: pip install conan
+ - name: Cache Conan
+ uses: actions/cache@v3
+ with:
+ path: ~/.conan2/p
+ key: ${{ runner.os }}-conan-${{ hashFiles('conanfile.py', 'libs/utils/**') }}
+ restore-keys: |
+ ${{ runner.os }}-conan-
+ - name: Setup Conan Profile
+ run: |
+ conan profile detect
+ - name: Conan install
+ run: conan install . --output-folder=build --build=missing -o "celix/*:build_utils=True" -o "celix/*:enable_fuzzing=True"
+ - name: Conan build
+ run: conan build . --output-folder=build -o "celix/*:build_utils=True" -o "celix/*:enable_fuzzing=True" -o "celix/*:celix_err_buffer_size=5120"
Review Comment:
I updated the fuzz test building to use ubsan and asan.
I also configured LD_PRELOAD for the fuzz testing ci. I would prefer this to be integrated with the enable asan and ubsan cmake options, but could not get that working.
Lastly I changed the handling of the enable asan and ubsan options so that both asan and ubsan can be active.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@celix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Feature/add fuzzing (celix)
Posted by "PengZheng (via GitHub)" <gi...@apache.org>.
PengZheng commented on PR #799:
URL: https://github.com/apache/celix/pull/799#issuecomment-3239913707
I'll finish reviewing it this week.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@celix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Feature/add fuzzing (celix)
Posted by "pnoltes (via GitHub)" <gi...@apache.org>.
pnoltes commented on code in PR #799:
URL: https://github.com/apache/celix/pull/799#discussion_r2373021807
##########
.github/workflows/fuzzing.yml:
##########
@@ -0,0 +1,59 @@
+name: Celix Fuzzing
+
+on:
+ push:
+ pull_request:
+ schedule:
+ - cron: '0 3 * * *'
+
+jobs:
+ fuzz-utils:
+ runs-on: ubuntu-22.04
+ timeout-minutes: 30
+ steps:
+ - name: Checkout source code
+ uses: actions/checkout@ac593985615ec2ede58e132d2e21d2b1cbd6127c #v3.3.0
+ - name: Set up Python
+ uses: actions/setup-python@7f4fc3e22c37d6ff65e88745f38bd3157c663f7c #v4.9.1
+ with:
+ python-version: '3.x'
+ - name: Set Compiler Environment Variables
+ run: |
+ echo "CC=clang" >> $GITHUB_ENV
+ echo "CXX=clang++" >> $GITHUB_ENV
+ - name: Install Conan
+ run: pip install conan
+ - name: Cache Conan
+ uses: actions/cache@v3
+ with:
+ path: ~/.conan2/p
+ key: ${{ runner.os }}-conan-${{ hashFiles('conanfile.py', 'libs/utils/**') }}
+ restore-keys: |
+ ${{ runner.os }}-conan-
+ - name: Setup Conan Profile
+ run: |
+ conan profile detect
+ - name: Conan install
+ run: conan install . --output-folder=build --build=missing -o "celix/*:build_utils=True" -o "celix/*:enable_fuzzing=True"
+ - name: Conan build
+ run: conan build . --output-folder=build -o "celix/*:build_utils=True" -o "celix/*:enable_fuzzing=True" -o "celix/*:celix_err_buffer_size=5120"
+ - name: Set fuzzer run time
+ id: set-runtime
+ run: |
+ if [[ "${{ github.event_name }}" == "schedule" ]]; then
+ echo "FUZZ_TIME=600" >> ${GITHUB_ENV}
+ else
+ echo "FUZZ_TIME=30" >> ${GITHUB_ENV}
+ fi
+ - name: Run properties fuzzer
+ run: |
+ source build/conanrun.sh
+ ./build/libs/utils/fuzzing/celix_properties_fuzzer -max_total_time=$FUZZ_TIME ./build/libs/utils/fuzzing/properties_corpus
Review Comment:
I am not sure yet. My idea was maybe to check whether we can use this together with OSS-Fuzz, for OSS-Fuzz I think the updated corpus will then be stored in the project setup at OSS-Fuzz.
If OSS-Fuzz is not useable, the some nightly or weekly runs that updates the corpus wat be good.
For now this PR add the fuzz functionality and does a few small runs to see if the fuzz tests still works.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@celix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Feature/add fuzzing (celix)
Posted by "pnoltes (via GitHub)" <gi...@apache.org>.
pnoltes commented on code in PR #799:
URL: https://github.com/apache/celix/pull/799#discussion_r2373027856
##########
libs/utils/CMakeLists.txt:
##########
@@ -126,5 +126,24 @@ if (UTILS)
add_subdirectory(gtest)
endif ()
+
+ if (ENABLE_FUZZING)
+ add_library(utils_cuf STATIC ${UTILS_SRC})
+ target_compile_definitions(utils_cuf PRIVATE CELIX_UTILS_STATIC_DEFINE)
+ target_include_directories(utils_cuf PUBLIC
+ ${CMAKE_CURRENT_LIST_DIR}/include
+ ${CMAKE_CURRENT_LIST_DIR}/include_internal
+ ${CMAKE_BINARY_DIR}/celix/gen/includes/utils
+ ${CMAKE_BINARY_DIR}/celix/gen/src/utils
+ include_deprecated
+ )
+ target_link_libraries(utils_cuf PUBLIC ${UTILS_PUBLIC_DEPS} ${UTILS_PRIVATE_DEPS})
+ target_compile_options(utils_cuf PRIVATE -fsanitize=fuzzer-no-link,address)
Review Comment:
Correct, I was not sure whether to allow (en)disabling asan for fuzz testing. I think for fuzz testing it would be logical to always have asan on.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@celix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Feature/add fuzzing (celix)
Posted by "pnoltes (via GitHub)" <gi...@apache.org>.
pnoltes commented on code in PR #799:
URL: https://github.com/apache/celix/pull/799#discussion_r2617384277
##########
libs/utils/fuzzing/src/FilterFuzz.cc:
##########
@@ -0,0 +1,23 @@
+#include <celix_filter.h>
+#include <celix_err.h>
+#include <cstdint>
+#include <cstdlib>
+#include <cstring>
+
+int filterParseFuzzOneInput(const uint8_t* data, size_t size) {
+ char* buffer = static_cast<char*>(malloc(size + 1));
+ if (buffer == nullptr) {
+ return 0;
+ }
+ memcpy(buffer, data, size);
+ buffer[size] = '\0';
+
+ celix_filter_t* filter = celix_filter_create(buffer);
Review Comment:
Look into this, but I think this is not really feasible. The fuzz testing is focused on fuzzing a single input buffer and for filter match you need two (properties and a filter).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@celix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Feature/add fuzzing (celix)
Posted by "pnoltes (via GitHub)" <gi...@apache.org>.
pnoltes commented on PR #799:
URL: https://github.com/apache/celix/pull/799#issuecomment-3671859873
> I have fixed the LD_PRELOAD issue.
:+1: , Thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@celix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org