You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@nifi.apache.org by fg...@apache.org on 2023/09/20 17:59:47 UTC
[nifi-minifi-cpp] branch main updated (7c6464707 -> 73ccd5eef)
This is an automated email from the ASF dual-hosted git repository.
fgerlits pushed a change to branch main
in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git
from 7c6464707 MINIFICPP-2216 Fix failing proxy setup in docker tests
new a328a01d2 MINIFICPP-2174 Send all cached compressed log files through C2
new 6ac5cb94d MINIFICPP-2209 Small HTTPClient refactor: preconditions and exception safety
new c44ea7c8f MINIFICPP-2214 Fix extension list inconsistencies in build scripts
new d9537edf0 MINIFICPP-2222 Update CONTRIB.md file with new contributor access request process
new 73ccd5eef MINIFICPP-2223 Add .asf.yaml file to configure repository labels
The 5 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.asf.yaml | 36 ++++
CMakeLists.txt | 1 +
CONTRIB.md => CONTRIBUTING.md | 16 +-
README.md | 20 +-
Windows.md | 76 ++++---
bootstrap.py | 234 ---------------------
bootstrap.sh | 16 +-
extensions/http-curl/client/HTTPClient.cpp | 30 ++-
extensions/http-curl/client/HTTPClient.h | 6 +-
.../include/core/logging/LoggerConfiguration.h | 8 +-
.../core/logging/internal/CompressionManager.h | 7 +-
.../core/logging/internal/LogCompressorSink.h | 33 ++-
libminifi/include/utils/StagingQueue.h | 8 +
libminifi/src/FlowController.cpp | 6 +-
.../core/logging/internal/CompressionManager.cpp | 3 +-
libminifi/test/unit/LoggerTests.cpp | 78 +++++--
win_build_vs.bat | 90 ++++----
17 files changed, 267 insertions(+), 401 deletions(-)
create mode 100644 .asf.yaml
rename CONTRIB.md => CONTRIBUTING.md (90%)
delete mode 100755 bootstrap.py
[nifi-minifi-cpp] 03/05: MINIFICPP-2214 Fix extension list inconsistencies in build scripts
Posted by fg...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
fgerlits pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git
commit c44ea7c8ffb5013f48e587d1f32aca1afae97084
Author: Marton Szasz <sz...@apache.org>
AuthorDate: Mon Sep 11 14:49:20 2023 +0200
MINIFICPP-2214 Fix extension list inconsistencies in build scripts
Signed-off-by: Ferenc Gerlits <fg...@gmail.com>
This closes #1654
---
CMakeLists.txt | 1 +
README.md | 20 ++---
Windows.md | 76 ++++++++++--------
bootstrap.py | 234 -------------------------------------------------------
bootstrap.sh | 16 ++--
win_build_vs.bat | 90 ++++++++++-----------
6 files changed, 103 insertions(+), 334 deletions(-)
diff --git a/CMakeLists.txt b/CMakeLists.txt
index bb20647ac..f602449c3 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -601,6 +601,7 @@ set(EXTENSIONS_ENABLED_BY_DEFAULT (
minifi-gcp
minifi-http-curl
minifi-archive-extensions
+ minifi-mqtt-extensions
minifi-rdkafka-extensions
minifi-pdh
minifi-prometheus
diff --git a/README.md b/README.md
index eeffe9cda..b46bb3452 100644
--- a/README.md
+++ b/README.md
@@ -106,7 +106,7 @@ Through JNI extensions you can run NiFi processors using NARs. The JNI extension
## Caveats
* We follow semver with regards to API compatibility, but no ABI compatibility is provided. See [semver's website](https://semver.org/) for more information
-* Build and usage currently only supports Windows, Linux and OS X environments. MiNiFi C++ can be built and run through the Windows Subsystem for Linux but we provide no support for this platform.
+* Build and usage currently only supports Windows, Linux and macOS environments. MiNiFi C++ can be built and run through the Windows Subsystem for Linux but we provide no support for this platform.
* Provenance events generation is supported and are persisted using RocksDB. Volatile repositories can be used on systems without persistent storage.
* If MiNiFi C++ is built with the OPC-UA extension enabled, it bundles [open62541](https://open62541.org/), which is available under the Mozilla Public License Version 2.0, a Category B license under [ASF 3rd party license policy](https://www.apache.org/legal/resolved.html#category-b).
* If MiNiFi C++ packaged on Windows, the resulting MSI may not be publicly redistributed under the Apache license, because it contains Microsoft redistributable DLLs, which fall under Category X of the [ASF 3rd party license policy](https://www.apache.org/legal/resolved.html#category-x).
@@ -131,7 +131,7 @@ versions of OpenSSL, cURL, or zlib are used:
* automake
* libtool
-**NOTE** if Expression Language support is enabled, FlexLexer must be in the include path and the version must be compatible with the version of flex used when generating lexer sources. On Mac OS X and FreeBSD, make sure that the system version is overridden with the correct flex version, which may require merging your CPPFLAGS into CFLAGS and CXXFLAGS. Lexer source generation is automatically performed during CMake builds. To re-generate the sources, remove:
+**NOTE** if Expression Language support is enabled, FlexLexer must be in the include path and the version must be compatible with the version of flex used when generating lexer sources. On macOS and FreeBSD, make sure that the system version is overridden with the correct flex version, which may require merging your CPPFLAGS into CFLAGS and CXXFLAGS. Lexer source generation is automatically performed during CMake builds. To re-generate the sources, remove:
* extensions/expression-language/Parser.cpp
* extensions/expression-language/Parser.hpp
@@ -267,7 +267,7 @@ apt install libgps-dev
apt install libpcap-dev
```
-##### OS X Using Homebrew (with XCode Command Line Tools installed)
+##### macOS Using Homebrew (with XCode Command Line Tools installed)
```
# ~/Development/code/apache/nifi-minifi-cpp on git:master
brew install cmake \
@@ -326,31 +326,31 @@ sudo brew install libpcap
A. Persistent Repositories .....Enabled
B. libcurl features ............Enabled
C. libarchive features .........Enabled
- D. Python Scripting support ....Disabled
+ D. Python Scripting support ....Enabled
E. Expression Language support .Enabled
F. Kafka support ...............Enabled
G. PCAP support ................Disabled
H. USB Camera support ..........Disabled
I. GPS support .................Disabled
K. Bustache Support ............Disabled
- L. Lua Scripting Support .......Disabled
- M. MQTT Support ................Disabled
+ L. Lua Scripting Support .......Enabled
+ M. MQTT Support ................Enabled
N. COAP Support ................Disabled
O. SFTP Support ................Disabled
S. AWS Support .................Enabled
T. OpenCV Support ..............Disabled
- U. OPC-UA Support...............Disabled
+ U. OPC-UA Support...............Enabled
V. SQL Support..................Enabled
W. Openwsman Support ...........Disabled
X. Azure Support ...............Enabled
Y. Systemd Support .............Enabled
Z. NanoFi Support ..............Disabled
AA. Splunk Support .............Enabled
- AB. Kubernetes Support .........Disabled
+ AB. Kubernetes Support .........Enabled
AC. Google Cloud Support .......Enabled
AD. ProcFs Support .............Enabled
- AE. Prometheus Support .........Disabled
- AF. Elasticsearch Support ......Disabled
+ AE. Prometheus Support .........Enabled
+ AF. Elasticsearch Support ......Enabled
****************************************
Build Options.
****************************************
diff --git a/Windows.md b/Windows.md
index 3502594ff..6050f375d 100644
--- a/Windows.md
+++ b/Windows.md
@@ -26,6 +26,8 @@ The project previously required OpenSSL to be installed. If you follow our build
- Visual Studio 2019
- [CMake](https://cmake.org/download/)
- [Git](https://git-scm.com/download/win) (the build process requires the bash.exe and patch.exe tools packaged with Git)
+ - [Perl](https://strawberryperl.com/)
+ - [NASM](https://nasm.us)
- (Optional) [WiX Toolset](https://wixtoolset.org/releases/) (only for building the MSI)
- (Optional) JDK (only for JNI support)
@@ -53,40 +55,46 @@ The preferred way of building the project is via the `win_build_vs.bat` script f
After the build directory it will take optional parameters modifying the CMake configuration used in the build:
-| Argument | Effect |
-|-------------------|--------------------------------------------------------------------------------|
-| /T | Disables building tests |
-| /R | Disables automatic test running after build |
-| /P | Enables MSI creation |
-| /K | Enables Kafka extension |
-| /J | Enables JNI |
-| /S | Enables SQL extension |
-| /C | Enables CoAP extension |
-| /A | Enables AWS extension |
-| /SFTP | Enables SFTP extension |
-| /PDH | Enables Performance Monitor extension |
-| /SPLUNK | Enables Splunk extension |
-| /GCP | Enables Google cloud storage extension |
-| /ELASTIC | Enables Elastic extension |
-| /Z | Enables Azure extension |
-| /MQTT | Enables MQTT extension |
-| /N | Enables Nanofi |
-| /LUA_SCRIPTING | Enables Lua scripting extension |
-| /PYTHON_SCRIPTING | Enables Python scripting extension |
-| /SENSORS | Enables the Sensors package |
-| /USB_CAMERA | Enables USB camera support |
-| /L | Enables Linter |
-| /O | Enables OpenCV |
-| /PR | Enables Prometheus |
-| /RO | Use real ODBC driver in tests instead of mock SQL driver |
-| /M | Creates installer with merge modules |
-| /64 | Creates 64-bit build instead of a 32-bit one |
-| /D | Builds RelWithDebInfo build instead of Release |
-| /DD | Builds Debug build instead of Release |
-| /CI | Sets STRICT_GSL_CHECKS to AUDIT |
-| /RO | Use SQLite ODBC driver in SQL extenstion unit tests instead of a mock database |
-| /NINJA | Uses Ninja build system instead of MSBuild |
-| /ENCRYPT_CONFIG | Enables build of encrypt-config binary |
+| Argument | Effect |
+|----------------------|-------------------------------------------------------------------------------------|
+| /T | Disables building tests |
+| /R | Disables automatic test running after build |
+| /P | Enables MSI creation |
+| /NO_KAFKA | Disables Kafka extension |
+| /J | Enables JNI |
+| /NO_SQL | Disables SQL extension |
+| /C | Enables CoAP extension |
+| /NO_AWS | Disables AWS extension |
+| /SFTP | Enables SFTP extension |
+| /PDH | Enables Performance Monitor extension |
+| /NO_SPLUNK | Disables Splunk extension |
+| /NO_GCP | Disables Google cloud storage extension |
+| /NO_ELASTIC | Disables Elastic extension |
+| /NO_AZURE | Disables Azure extension |
+| /NO_MQTT | Disables MQTT extension |
+| /N | Enables Nanofi |
+| /NO_LUA_SCRIPTING | Disables Lua scripting extension |
+| /NO_PYTHON_SCRIPTING | Disables Python scripting extension |
+| /SENSORS | Enables the Sensors package |
+| /USB_CAMERA | Enables USB camera support |
+| /O | Enables OpenCV |
+| /NO_PROMETHEUS | Disables Prometheus |
+| /RO | Use real ODBC driver in tests instead of mock SQL driver |
+| /M | Creates installer with merge modules |
+| /32 | Creates 32-bit build instead of a 64-bit one |
+| /D | Builds RelWithDebInfo build instead of Release |
+| /DD | Builds Debug build instead of Release |
+| /CI | Sets STRICT_GSL_CHECKS to AUDIT |
+| /RO | Use SQLite ODBC driver in SQL extenstion unit tests instead of a mock database |
+| /NINJA | Uses Ninja build system instead of MSBuild |
+| /NO_ENCRYPT_CONFIG | Disables build of encrypt-config binary |
+| /SCCACHE | Uses sccache build caching |
+| /BUSTACHE | Enables Bustache templating support |
+| /NO_OPC | Disables OPC extension |
+| /OPENWSMAN | Enables OpenWSMAN extension |
+| /NO_OPS | Disables OPS extension |
+| /PCAP | Enables PCAP extension |
+| /NONFREEUCRT | Enables inclusion of non-free UCRT libraries in the installer (not redistributable) |
Examples:
- 32-bit build with kafka, disabling tests, enabling MSI creation: `win_build_vs.bat build32 /T /K /P`
diff --git a/bootstrap.py b/bootstrap.py
deleted file mode 100755
index 6a7d72d16..000000000
--- a/bootstrap.py
+++ /dev/null
@@ -1,234 +0,0 @@
-#!/usr/bin/python
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations
-# under the License.
-
-from __future__ import print_function
-
-import argparse
-import sys
-import json
-import os.path
-import platform
-import tarfile
-
-from distutils.util import strtobool
-from ftplib import FTP
-
-if sys.version_info[0] < 3:
- from urllib2 import urlopen
- input = raw_input
-else:
- from urllib.request import urlopen
-
-MINIFI_SUBFOLDER = '/nifi/nifi-minifi-cpp/'
-APACHE_CLOSER_REPO_JSON_URL = 'https://www.apache.org/dyn/closer.cgi?as_json=1&path=/nifi/nifi-minifi-cpp'
-APACHE_MIRROR_LIST = "http://www.apache.org/mirrors/"
-
-
-def install_package(package_name):
- try:
- import pip
- if hasattr(pip, 'main'):
- pipcode = pip.main(['install', package_name])
- else:
- pipcode = pip._internal.main(['install', package_name])
- return pipcode == 0
- except ImportError:
- return False
-
-
-distro_available = False
-
-try:
- import distro
-
- distro_available = True
-except ImportError:
- distro_available = install_package("distro")
-
-
-def get_distro():
- if is_mac():
- return ["osx", "", "darwin"]
- try:
- if distro_available:
- return distro.linux_distribution(full_distribution_name=False)
- else:
- return platform.linux_distribution()
- except Exception:
- return ["N/A", "N/A", "N/A"]
-
-
-def is_mac():
- return platform.system() == "Darwin"
-
-
-def mapped_distro():
- distro_info = get_distro()
- distro = distro_info[0].lower()
- release = distro_info[2].lower()
- if any(d in distro for d in ["rhel", "red hat", "centos"]):
- return "rhel", release
- else:
- return distro, release
-
-
-def find_closest_mirror():
- try:
- url = urlopen(APACHE_CLOSER_REPO_JSON_URL)
- data = json.loads(url.read().decode())
-
- return data['ftp'][0]
- except Exception:
- print("Failed to find closest mirror, please specify one!")
- return ""
-
-
-def get_release_and_binaries_from_ftp(host, apache_dir, version=None):
- ftp = FTP(host)
- ftp.login()
- ftp.cwd(apache_dir + MINIFI_SUBFOLDER)
- # list files with ftplib
- file_list = list(filter(lambda x: any(char.isdigit() for char in x),
- ftp.nlst(""))) # to filter "." and ".." - relese names contain number
- file_list.sort(reverse=True)
- if not version:
- latest_release = file_list[0]
- else:
- if version not in file_list:
- print("The specified version (" + version + ") doesn't exist. Please use one of the following: " + ", ".join(file_list))
- exit(-1)
- latest_release = version
-
- ftp.cwd("./" + latest_release)
- binaries = list(filter(lambda x: any(char.isdigit() for char in x), ftp.nlst("")))
-
- ftp.quit()
-
- return latest_release, binaries
-
-
-def download_binary_from_ftp(host, apache_dir, release, binary):
- successful_download = False
-
- try:
- ftp = FTP(host)
- ftp.login()
- ftp.cwd(apache_dir + MINIFI_SUBFOLDER + release)
-
- print("Downloading: ftp://" + host + "/" + MINIFI_SUBFOLDER + release + "/" + binary)
-
- with open(os.path.join(os.getcwd(), binary), "wb") as targetfile:
- ftp.retrbinary("RETR " + binary, targetfile.write)
- successful_download = True
- except Exception:
- print("Failed to download binary")
- finally:
- ftp.quit()
-
- return successful_download
-
-
-def main(args):
- print(get_distro())
- binaries = []
-
- try:
- local_repo = args.mirror if args.mirror else find_closest_mirror()
-
- print(local_repo)
-
- host, dir = local_repo.replace('ftp://', '').split('/', 1)
- latest_release, binaries = get_release_and_binaries_from_ftp(host, dir, args.version if args.version else None)
-
- except Exception:
- print("Failed to get binaries from Apache mirror")
- return -1
-
- matching_binaries = []
-
- for binary in binaries:
- distro, release = mapped_distro()
- if release and release in binary:
- matching_binaries.append(binary)
- elif distro and distro in binary:
- matching_binaries.append(binary)
-
- if not matching_binaries:
- print("No compatible binary found, MiNiFi needs to be compiled locally")
- return 1
-
- invalid_input = True
- download = None
- selected_binary = None
-
- if len(matching_binaries) == 1:
- print("A binary in Apache repo seems to match your system: " + matching_binaries[0])
- while invalid_input:
- try:
- download = strtobool(input("Would you like to download? [y/n]"))
- invalid_input = False
- if download:
- selected_binary = matching_binaries[0]
- except Exception:
- pass
-
- else:
- print("The following binaries in Apache repo seem to match your system: ")
- for i, item in enumerate(matching_binaries):
- print(str(i + 1) + " - " + item)
- print()
- while invalid_input:
- try:
- user_input = input("Please select one to download (1 to " + str(len(matching_binaries)) + ") or \"s\" to skip and compile locally\n")
- user_input.lower()
- if user_input == "s":
- invalid_input = False
- download = False
- break
- idx = int(user_input) - 1
- if (idx < 0):
- continue
- selected_binary = matching_binaries[idx]
- download = True
- invalid_input = False
- except Exception:
- pass
-
- if not download:
- return 1
-
- if not download_binary_from_ftp(host, dir, latest_release, selected_binary):
- return -1
-
- try:
- with tarfile.open(os.path.join(os.getcwd(), selected_binary), "r:gz") as tar:
- tar.extractall()
- except Exception:
- print("Failed to extract tar file")
- return -1
-
- print("Successfully downloaded and extracted MiNiFi")
- return 0
-
-
-if __name__ == '__main__':
- parser = argparse.ArgumentParser(description="Download latest MiNiFi release")
- parser.add_argument("-m", "--mirror", dest="mirror", help="user-specified apache mirror")
- parser.add_argument("-v", "--version", dest="version", help="user-specified version to be downloaded")
- args = parser.parse_args()
- main(args)
diff --git a/bootstrap.sh b/bootstrap.sh
index 466251b68..4eab6c350 100755
--- a/bootstrap.sh
+++ b/bootstrap.sh
@@ -266,9 +266,9 @@ add_option HTTP_CURL_ENABLED ${TRUE} "DISABLE_CURL"
add_option LIBARCHIVE_ENABLED ${TRUE} "DISABLE_LIBARCHIVE"
add_dependency LIBARCHIVE_ENABLED "libarchive"
-add_option PYTHON_SCRIPTING_ENABLED ${FALSE} "ENABLE_PYTHON_SCRIPTING"
+add_option PYTHON_SCRIPTING_ENABLED ${TRUE} "ENABLE_PYTHON_SCRIPTING"
add_dependency PYTHON_SCRIPTING_ENABLED "python"
-add_option LUA_SCRIPTING_ENABLED ${FALSE} "ENABLE_LUA_SCRIPTING"
+add_option LUA_SCRIPTING_ENABLED ${TRUE} "ENABLE_LUA_SCRIPTING"
add_dependency LUA_SCRIPTING_ENABLED "lua"
add_option EXPRESSION_LANGUAGE_ENABLED ${TRUE} "DISABLE_EXPRESSION_LANGUAGE"
@@ -289,9 +289,9 @@ add_option AWS_ENABLED ${TRUE} "ENABLE_AWS"
add_option KAFKA_ENABLED ${TRUE} "ENABLE_LIBRDKAFKA"
-add_option KUBERNETES_ENABLED ${FALSE} "ENABLE_KUBERNETES"
+add_option KUBERNETES_ENABLED ${TRUE} "ENABLE_KUBERNETES"
-add_option MQTT_ENABLED ${FALSE} "ENABLE_MQTT"
+add_option MQTT_ENABLED ${TRUE} "ENABLE_MQTT"
add_option COAP_ENABLED ${FALSE} "ENABLE_COAP"
add_dependency COAP_ENABLED "automake"
@@ -303,8 +303,6 @@ add_dependency JNI_ENABLED "jnibuild"
add_option OPENCV_ENABLED ${FALSE} "ENABLE_OPENCV"
-add_option OPENCV_ENABLED ${FALSE} "ENABLE_OPENCV"
-
add_option SFTP_ENABLED ${FALSE} "ENABLE_SFTP"
add_dependency SFTP_ENABLED "libssh2"
@@ -315,7 +313,7 @@ add_option OPENWSMAN_ENABLED ${FALSE} "ENABLE_OPENWSMAN"
# Since the following extensions have limitations on
add_option BUSTACHE_ENABLED ${FALSE} "ENABLE_BUSTACHE" "2.6" ${TRUE}
-add_option OPC_ENABLED ${FALSE} "ENABLE_OPC"
+add_option OPC_ENABLED ${TRUE} "ENABLE_OPC"
add_dependency OPC_ENABLED "mbedtls"
add_option AZURE_ENABLED ${TRUE} "ENABLE_AZURE"
@@ -331,12 +329,12 @@ set_dependency SPLUNK_ENABLED HTTP_CURL_ENABLED
add_option GCP_ENABLED ${TRUE} "ENABLE_GCP"
-add_option ELASTIC_ENABLED ${FALSE} "ENABLE_ELASTICSEARCH"
+add_option ELASTIC_ENABLED ${TRUE} "ENABLE_ELASTICSEARCH"
set_dependency ELASTIC_ENABLED HTTP_CURL_ENABLED
add_option PROCFS_ENABLED ${TRUE} "ENABLE_PROCFS"
-add_option PROMETHEUS_ENABLED ${FALSE} "ENABLE_PROMETHEUS"
+add_option PROMETHEUS_ENABLED ${TRUE} "ENABLE_PROMETHEUS"
add_option OPENSSL_ENABLED ${TRUE} "OPENSSL_OFF"
add_dependency OPENSSL_ENABLED "opensslbuild"
diff --git a/win_build_vs.bat b/win_build_vs.bat
index c4d98c810..fada62d85 100755
--- a/win_build_vs.bat
+++ b/win_build_vs.bat
@@ -24,27 +24,24 @@ set scriptdir=%~dp0
set skiptests=OFF
set skiptestrun=OFF
set cmake_build_type=Release
-set build_platform=Win32
-set build_kafka=OFF
-set build_coap=OFF
-set build_jni=OFF
-set build_SQL=OFF
-set build_AWS=OFF
-set build_SFTP=OFF
-set build_azure=OFF
-set enable_bustache=OFF
+set build_platform=x64
+set enable_kafka=ON
set enable_coap=OFF
-set enable_encrypt_config=OFF
-set enable_gps=OFF
-set enable_lua_scripting=OFF
-set enable_mqtt=OFF
-set enable_opc=OFF
+set enable_jni=OFF
+set enable_sql=ON
+set enable_aws=ON
+set enable_sftp=OFF
+set enable_azure=ON
+set enable_bustache=OFF
+set enable_encrypt_config=ON
+set enable_lua_scripting=ON
+set enable_mqtt=ON
+set enable_opc=ON
set enable_openwsman=OFF
-set enable_ops=OFF
+set enable_ops=ON
set enable_pcap=OFF
-set enable_python_scripting=OFF
+set enable_python_scripting=ON
set enable_sensors=OFF
-set enable_tensorflow=OFF
set enable_usb_camera=OFF
set test_custom_wel_provider=OFF
set generator="Visual Studio 16 2019"
@@ -52,9 +49,9 @@ set cpack=OFF
set installer_merge_modules=OFF
set strict_gsl_checks=
set redist=
-set build_nanofi=OFF
-set build_opencv=OFF
-set build_prometheus=OFF
+set enable_nanofi=OFF
+set enable_opencv=OFF
+set enable_prometheus=ON
set real_odbc=OFF
set sccache_arg=
@@ -65,34 +62,33 @@ for %%x in (%*) do (
if [%%~x] EQU [/T] set skiptests=ON
if [%%~x] EQU [/R] set skiptestrun=ON
if [%%~x] EQU [/P] set cpack=ON
- if [%%~x] EQU [/K] set build_kafka=ON
- if [%%~x] EQU [/J] set build_JNI=ON
- if [%%~x] EQU [/S] set build_SQL=ON
- if [%%~x] EQU [/C] set build_coap=ON
- if [%%~x] EQU [/A] set build_AWS=ON
- if [%%~x] EQU [/SFTP] set build_SFTP=ON
- if [%%~x] EQU [/PDH] set build_PDH=ON
- if [%%~x] EQU [/SPLUNK] set build_SPLUNK=ON
- if [%%~x] EQU [/GCP] set build_GCP=ON
- if [%%~x] EQU [/ELASTIC] set build_ELASTIC=ON
+ if [%%~x] EQU [/NO_KAFKA] set enable_kafka=OFF
+ if [%%~x] EQU [/J] set enable_jni=ON
+ if [%%~x] EQU [/NO_SQL] set enable_sql=OFF
+ if [%%~x] EQU [/C] set enable_coap=ON
+ if [%%~x] EQU [/NO_AWS] set enable_aws=OFF
+ if [%%~x] EQU [/SFTP] set enable_sftp=ON
+ if [%%~x] EQU [/PDH] set enable_pdh=ON
+ if [%%~x] EQU [/NO_SPLUNK] set enable_splunk=OFF
+ if [%%~x] EQU [/NO_GCP] set enable_gcp=OFF
+ if [%%~x] EQU [/NO_ELASTIC] set enable_elastic=OFF
if [%%~x] EQU [/M] set installer_merge_modules=ON
- if [%%~x] EQU [/Z] set build_azure=ON
- if [%%~x] EQU [/N] set build_nanofi=ON
- if [%%~x] EQU [/O] set build_opencv=ON
- if [%%~x] EQU [/PR] set build_prometheus=ON
+ if [%%~x] EQU [/NO_AZURE] set enable_azure=OFF
+ if [%%~x] EQU [/N] set enable_nanofi=ON
+ if [%%~x] EQU [/O] set enable_opencv=ON
+ if [%%~x] EQU [/NO_PROMETHEUS] set enable_prometheus=OFF
if [%%~x] EQU [/BUSTACHE] set enable_bustache=ON
- if [%%~x] EQU [/COAP] set enable_coap=ON
- if [%%~x] EQU [/ENCRYPT_CONFIG] set enable_encrypt_config=ON
- if [%%~x] EQU [/LUA_SCRIPTING] set enable_lua_scripting=ON
- if [%%~x] EQU [/MQTT] set enable_mqtt=ON
- if [%%~x] EQU [/OPC] set enable_opc=ON
+ if [%%~x] EQU [/NO_ENCRYPT_CONFIG] set enable_encrypt_config=OFF
+ if [%%~x] EQU [/NO_LUA_SCRIPTING] set enable_lua_scripting=OFF
+ if [%%~x] EQU [/NO_MQTT] set enable_mqtt=OFF
+ if [%%~x] EQU [/NO_OPC] set enable_opc=OFF
if [%%~x] EQU [/OPENWSMAN] set enable_openwsman=ON
- if [%%~x] EQU [/OPS] set enable_ops=ON
+ if [%%~x] EQU [/NO_OPS] set enable_ops=OFF
if [%%~x] EQU [/PCAP] set enable_pcap=ON
- if [%%~x] EQU [/PYTHON_SCRIPTING] set enable_python_scripting=ON
+ if [%%~x] EQU [/NO_PYTHON_SCRIPTING] set enable_python_scripting=OFF
if [%%~x] EQU [/SENSORS] set enable_sensors=ON
if [%%~x] EQU [/USB_CAMERA] set enable_usb_camera=ON
- if [%%~x] EQU [/64] set build_platform=x64
+ if [%%~x] EQU [/32] set build_platform=Win32
if [%%~x] EQU [/D] set cmake_build_type=RelWithDebInfo
if [%%~x] EQU [/DD] set cmake_build_type=Debug
if [%%~x] EQU [/CI] set "strict_gsl_checks=-DSTRICT_GSL_CHECKS=AUDIT" & set test_custom_wel_provider=ON
@@ -113,11 +109,11 @@ if [%generator%] EQU ["Ninja"] (
set "build_platform_cmd=-A %build_platform%"
)
echo on
-cmake -G %generator% %build_platform_cmd% -DINSTALLER_MERGE_MODULES=%installer_merge_modules% -DTEST_CUSTOM_WEL_PROVIDER=%test_custom_wel_provider% -DENABLE_SQL=%build_SQL% -DUSE_REAL_ODBC_TEST_DRIVER=%real_odbc% ^
- -DCMAKE_BUILD_TYPE_INIT=%cmake_build_type% -DCMAKE_BUILD_TYPE=%cmake_build_type% -DWIN32=WIN32 -DENABLE_LIBRDKAFKA=%build_kafka% -DENABLE_JNI=%build_jni% -DOPENSSL_OFF=OFF ^
- -DENABLE_COAP=%build_coap% -DENABLE_AWS=%build_AWS% -DENABLE_PDH=%build_PDH% -DENABLE_AZURE=%build_azure% -DENABLE_SFTP=%build_SFTP% -DENABLE_SPLUNK=%build_SPLUNK% -DENABLE_GCP=%build_GCP% ^
- -DENABLE_NANOFI=%build_nanofi% -DENABLE_OPENCV=%build_opencv% -DENABLE_PROMETHEUS=%build_prometheus% -DENABLE_ELASTICSEARCH=%build_ELASTIC% -DUSE_SHARED_LIBS=OFF -DENABLE_CONTROLLER=OFF ^
- -DENABLE_BUSTACHE=%enable_bustache% -DENABLE_COAP=%enable_coap% -DENABLE_ENCRYPT_CONFIG=%enable_encrypt_config% -DENABLE_LUA_SCRIPTING=%enable_lua_scripting% ^
+cmake -G %generator% %build_platform_cmd% -DINSTALLER_MERGE_MODULES=%installer_merge_modules% -DTEST_CUSTOM_WEL_PROVIDER=%test_custom_wel_provider% -DENABLE_SQL=%enable_sql% -DUSE_REAL_ODBC_TEST_DRIVER=%real_odbc% ^
+ -DCMAKE_BUILD_TYPE_INIT=%cmake_build_type% -DCMAKE_BUILD_TYPE=%cmake_build_type% -DWIN32=WIN32 -DENABLE_LIBRDKAFKA=%enable_kafka% -DENABLE_JNI=%enable_jni% -DOPENSSL_OFF=OFF ^
+ -DENABLE_COAP=%enable_coap% -DENABLE_AWS=%enable_aws% -DENABLE_PDH=%enable_pdh% -DENABLE_AZURE=%enable_azure% -DENABLE_SFTP=%enable_sftp% -DENABLE_SPLUNK=%enable_splunk% -DENABLE_GCP=%enable_gcp% ^
+ -DENABLE_NANOFI=%enable_nanofi% -DENABLE_OPENCV=%enable_opencv% -DENABLE_PROMETHEUS=%enable_prometheus% -DENABLE_ELASTICSEARCH=%enable_elastic% -DUSE_SHARED_LIBS=OFF -DENABLE_CONTROLLER=OFF ^
+ -DENABLE_BUSTACHE=%enable_bustache% -DENABLE_ENCRYPT_CONFIG=%enable_encrypt_config% -DENABLE_LUA_SCRIPTING=%enable_lua_scripting% ^
-DENABLE_MQTT=%enable_mqtt% -DENABLE_OPC=%enable_opc% -DENABLE_OPENWSMAN=%enable_openwsman% -DENABLE_OPS=%enable_ops% -DENABLE_PCAP=%enable_pcap% ^
-DENABLE_PYTHON_SCRIPTING=%enable_python_scripting% -DENABLE_SENSORS=%enable_sensors% -DENABLE_USB_CAMERA=%enable_usb_camera% ^
-DBUILD_ROCKSDB=ON -DFORCE_WINDOWS=ON -DUSE_SYSTEM_UUID=OFF -DDISABLE_LIBARCHIVE=OFF -DENABLE_WEL=ON -DFAIL_ON_WARNINGS=OFF -DSKIP_TESTS=%skiptests% ^
[nifi-minifi-cpp] 01/05: MINIFICPP-2174 Send all cached compressed log files through C2
Posted by fg...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
fgerlits pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git
commit a328a01d288d021a19023facd79a76b2a7d2709f
Author: Gabor Gyimesi <ga...@gmail.com>
AuthorDate: Wed Aug 9 18:24:24 2023 +0200
MINIFICPP-2174 Send all cached compressed log files through C2
Signed-off-by: Ferenc Gerlits <fg...@gmail.com>
This closes #1631
---
.../include/core/logging/LoggerConfiguration.h | 8 +--
.../core/logging/internal/CompressionManager.h | 7 +-
.../core/logging/internal/LogCompressorSink.h | 33 ++++++---
libminifi/include/utils/StagingQueue.h | 8 +++
libminifi/src/FlowController.cpp | 6 +-
.../core/logging/internal/CompressionManager.cpp | 3 +-
libminifi/test/unit/LoggerTests.cpp | 78 ++++++++++++++++------
7 files changed, 104 insertions(+), 39 deletions(-)
diff --git a/libminifi/include/core/logging/LoggerConfiguration.h b/libminifi/include/core/logging/LoggerConfiguration.h
index 6dfd7effb..c62ee74b4 100644
--- a/libminifi/include/core/logging/LoggerConfiguration.h
+++ b/libminifi/include/core/logging/LoggerConfiguration.h
@@ -98,15 +98,15 @@ class LoggerConfiguration {
*/
void initialize(const std::shared_ptr<LoggerProperties> &logger_properties);
- static std::unique_ptr<io::InputStream> getCompressedLog(bool flush = false) {
- return getCompressedLog(std::chrono::milliseconds{0}, flush);
+ static std::vector<std::unique_ptr<io::InputStream>> getCompressedLogs() {
+ return getCompressedLogs(std::chrono::milliseconds{0});
}
void initializeAlertSinks(core::controller::ControllerServiceProvider* controller, const std::shared_ptr<AgentIdentificationProvider>& agent_id);
template<class Rep, class Period>
- static std::unique_ptr<io::InputStream> getCompressedLog(const std::chrono::duration<Rep, Period>& time, bool flush = false) {
- return getConfiguration().compression_manager_.getCompressedLog(time, flush);
+ static std::vector<std::unique_ptr<io::InputStream>> getCompressedLogs(const std::chrono::duration<Rep, Period>& time) {
+ return getConfiguration().compression_manager_.getCompressedLogs(time);
}
/**
diff --git a/libminifi/include/core/logging/internal/CompressionManager.h b/libminifi/include/core/logging/internal/CompressionManager.h
index b1ea9fac9..b881a5a3f 100644
--- a/libminifi/include/core/logging/internal/CompressionManager.h
+++ b/libminifi/include/core/logging/internal/CompressionManager.h
@@ -24,6 +24,7 @@
#include <functional>
#include <utility>
#include <string>
+#include <vector>
#include "core/logging/Logger.h"
#include "LogCompressorSink.h"
@@ -50,12 +51,12 @@ class CompressionManager {
std::shared_ptr<LogCompressorSink> initialize(const std::shared_ptr<LoggerProperties>& properties, const std::shared_ptr<Logger>& error_logger, const LoggerFactory& logger_factory);
template<class Rep, class Period>
- std::unique_ptr<io::InputStream> getCompressedLog(const std::chrono::duration<Rep, Period>& time, bool flush = false) {
+ std::vector<std::unique_ptr<io::InputStream>> getCompressedLogs(const std::chrono::duration<Rep, Period>& time) {
std::shared_ptr<internal::LogCompressorSink> sink = getSink();
if (sink) {
- return sink->getContent(time, flush);
+ return sink->getContent(time);
}
- return nullptr;
+ return {};
}
static constexpr const char* compression_cached_log_max_size_ = "compression.cached.log.max.size";
diff --git a/libminifi/include/core/logging/internal/LogCompressorSink.h b/libminifi/include/core/logging/internal/LogCompressorSink.h
index fd02521ad..009fc1235 100644
--- a/libminifi/include/core/logging/internal/LogCompressorSink.h
+++ b/libminifi/include/core/logging/internal/LogCompressorSink.h
@@ -21,6 +21,7 @@
#include <memory>
#include <atomic>
#include <utility>
+#include <vector>
#include "spdlog/common.h"
#include "spdlog/details/log_msg.h"
@@ -57,26 +58,42 @@ class LogCompressorSink : public spdlog::sinks::base_sink<std::mutex> {
~LogCompressorSink() override;
template<class Rep, class Period>
- std::unique_ptr<io::InputStream> getContent(const std::chrono::duration<Rep, Period>& time, bool flush = false) {
- if (flush) {
- cached_logs_.commit();
- compress(true);
+ std::vector<std::unique_ptr<io::InputStream>> getContent(const std::chrono::duration<Rep, Period>& time) {
+ cached_logs_.commit();
+ compress(true);
+
+ std::vector<std::unique_ptr<io::InputStream>> log_segments;
+ const auto segment_count = compressed_logs_.itemCount();
+ for (size_t i = 0; i < segment_count; ++i) {
+ LogBuffer compressed;
+ if (!compressed_logs_.tryDequeue(compressed, time)) {
+ break;
+ }
+ log_segments.push_back(std::move(compressed.buffer_));
}
- LogBuffer compressed;
- if (!compressed_logs_.tryDequeue(compressed, time) && flush) {
- return createEmptyArchive();
+
+ if (log_segments.empty()) {
+ log_segments.push_back(createEmptyArchive());
}
- return std::move(compressed.buffer_);
+ return log_segments;
}
size_t getMaxCacheSize() const {
return cached_logs_.getMaxSize();
}
+ size_t getMaxCacheSegmentSize() const {
+ return cached_logs_.getMaxItemSize();
+ }
+
size_t getMaxCompressedSize() const {
return compressed_logs_.getMaxSize();
}
+ size_t getMaxCompressedSegmentSize() const {
+ return compressed_logs_.getMaxItemSize();
+ }
+
private:
enum class CompressionResult {
Success,
diff --git a/libminifi/include/utils/StagingQueue.h b/libminifi/include/utils/StagingQueue.h
index 854ea3430..6751c5301 100644
--- a/libminifi/include/utils/StagingQueue.h
+++ b/libminifi/include/utils/StagingQueue.h
@@ -143,6 +143,10 @@ class StagingQueue {
return max_size_;
}
+ size_t getMaxItemSize() const {
+ return max_item_size_;
+ }
+
void discardOverflow() {
while (total_size_ > max_size_) {
Item item;
@@ -157,6 +161,10 @@ class StagingQueue {
return total_size_;
}
+ size_t itemCount() const {
+ return queue_.size();
+ }
+
private:
void commit(std::unique_lock<std::mutex>& /*lock*/) {
queue_.enqueue(active_item_.commit());
diff --git a/libminifi/src/FlowController.cpp b/libminifi/src/FlowController.cpp
index 451ac2d18..59b3b9261 100644
--- a/libminifi/src/FlowController.cpp
+++ b/libminifi/src/FlowController.cpp
@@ -473,8 +473,10 @@ std::vector<BackTrace> FlowController::getTraces() {
std::map<std::string, std::unique_ptr<io::InputStream>> FlowController::getDebugInfo() {
std::map<std::string, std::unique_ptr<io::InputStream>> debug_info;
- if (auto logs = core::logging::LoggerConfiguration::getCompressedLog(true)) {
- debug_info["minifi.log.gz"] = std::move(logs);
+ auto logs = core::logging::LoggerConfiguration::getCompressedLogs();
+ for (size_t i = 0; i < logs.size(); ++i) {
+ std::string index_str = i == logs.size() - 1 ? "" : "." + std::to_string(logs.size() - 1 - i);
+ debug_info["minifi.log" + index_str + ".gz"] = std::move(logs[i]);
}
if (auto opt_flow_path = flow_configuration_->getConfigurationPath()) {
debug_info["config.yml"] = std::make_unique<io::FileStream>(opt_flow_path.value(), 0, false);
diff --git a/libminifi/src/core/logging/internal/CompressionManager.cpp b/libminifi/src/core/logging/internal/CompressionManager.cpp
index 3a85100fb..83da24ee2 100644
--- a/libminifi/src/core/logging/internal/CompressionManager.cpp
+++ b/libminifi/src/core/logging/internal/CompressionManager.cpp
@@ -51,7 +51,8 @@ std::shared_ptr<LogCompressorSink> CompressionManager::initialize(
return sink_;
}
// do not create new sink if all relevant parameters match
- if (!sink_ || sink_->getMaxCacheSize() != cached_log_max_size || sink_->getMaxCompressedSize() != compressed_log_max_size) {
+ if (!sink_ || sink_->getMaxCacheSize() != cached_log_max_size || sink_->getMaxCompressedSize() != compressed_log_max_size ||
+ sink_->getMaxCacheSegmentSize() != cache_segment_size || sink_->getMaxCompressedSegmentSize() != compressed_segment_size) {
sink_ = std::make_shared<internal::LogCompressorSink>(
LogQueueSize{cached_log_max_size, cache_segment_size},
LogQueueSize{compressed_log_max_size, compressed_segment_size},
diff --git a/libminifi/test/unit/LoggerTests.cpp b/libminifi/test/unit/LoggerTests.cpp
index 00b7c84e7..adf8ce71d 100644
--- a/libminifi/test/unit/LoggerTests.cpp
+++ b/libminifi/test/unit/LoggerTests.cpp
@@ -20,6 +20,7 @@
#include <memory>
#include <vector>
#include <ctime>
+#include <random>
#include "../TestBase.h"
#include "../Catch.h"
#include "core/logging/LoggerConfiguration.h"
@@ -189,7 +190,7 @@ TEST_CASE("Test ShortenNames", "[ttl8]") {
using namespace minifi::io;
-std::string decompress(const std::shared_ptr<InputStream>& input) {
+std::string decompress(const std::unique_ptr<InputStream>& input) {
auto output = std::make_unique<BufferStream>();
auto decompressor = std::make_shared<ZlibDecompressStream>(gsl::make_not_null(output.get()));
minifi::internal::pipe(*input, *decompressor);
@@ -214,9 +215,9 @@ TEST_CASE("Test Compression", "[ttl9]") {
log_config.initialize(properties);
auto logger = log_config.getLogger(className);
logger->log_error("Hi there");
- std::shared_ptr<InputStream> compressed_log{logging::LoggerConfiguration::getCompressedLog(true)};
- REQUIRE(compressed_log);
- auto logs = decompress(compressed_log);
+ auto compressed_logs = logging::LoggerConfiguration::getCompressedLogs();
+ REQUIRE(compressed_logs.size() == 1);
+ auto logs = decompress(compressed_logs[0]);
REQUIRE(logs.find("Hi there") != std::string::npos);
}
@@ -234,6 +235,9 @@ class LoggerTestAccessor {
static size_t getCompressedSize(logging::LoggerConfiguration& log_config) {
return log_config.compression_manager_.getSink()->compressed_logs_.size();
}
+ static void runCompression(logging::LoggerConfiguration& log_config) {
+ while (logging::internal::LogCompressorSink::CompressionResult::Success == log_config.compression_manager_.getSink()->compress()){}
+ }
};
TEST_CASE("Test Compression cache overflow is discarded intermittently", "[ttl10]") {
@@ -258,17 +262,17 @@ TEST_CASE("Test Compression cache overflow is discarded intermittently", "[ttl10
TEST_CASE("Setting either properties to 0 disables in-memory compressed logs", "[ttl11]") {
auto& log_config = logging::LoggerConfiguration::getConfiguration();
auto properties = std::make_shared<logging::LoggerProperties>();
- bool is_nullptr = false;
+ bool is_empty = false;
SECTION("Cached log size is set to 0") {
- is_nullptr = true;
+ is_empty = true;
properties->set(logging::internal::CompressionManager::compression_cached_log_max_size_, "0");
}
SECTION("Compressed log size is set to 0") {
- is_nullptr = true;
+ is_empty = true;
properties->set(logging::internal::CompressionManager::compression_compressed_log_max_size_, "0");
}
SECTION("Sanity check") {
- is_nullptr = false;
+ is_empty = false;
// pass
}
// by default the root logger is OFF
@@ -276,7 +280,7 @@ TEST_CASE("Setting either properties to 0 disables in-memory compressed logs", "
log_config.initialize(properties);
auto logger = log_config.getLogger("DisableCompressionTestLogger");
logger->log_error("Hi there");
- REQUIRE((logging::LoggerConfiguration::getCompressedLog(true) == nullptr) == is_nullptr);
+ REQUIRE(logging::LoggerConfiguration::getCompressedLogs().empty() == is_empty);
}
TEST_CASE("Setting max log entry length property trims long log entries", "[ttl12]") {
@@ -288,9 +292,9 @@ TEST_CASE("Setting max log entry length property trims long log entries", "[ttl1
auto logger = log_config.getLogger("SetMaxLogEntryLengthTestLogger");
logger->log_error("Hi there");
- std::shared_ptr<InputStream> compressed_log{logging::LoggerConfiguration::getCompressedLog(true)};
- REQUIRE(compressed_log);
- auto logs = decompress(compressed_log);
+ auto compressed_logs = logging::LoggerConfiguration::getCompressedLogs();
+ REQUIRE(compressed_logs.size() == 1);
+ auto logs = decompress(compressed_logs[0]);
REQUIRE(logs.find("Hi ") == std::string::npos);
REQUIRE(logs.find("Hi") != std::string::npos);
}
@@ -304,9 +308,9 @@ TEST_CASE("Setting max log entry length property trims long formatted log entrie
auto logger = log_config.getLogger("SetMaxLogEntryLengthTestLogger");
logger->log_error("Hi there %s", "John");
- std::shared_ptr<InputStream> compressed_log{logging::LoggerConfiguration::getCompressedLog(true)};
- REQUIRE(compressed_log);
- auto logs = decompress(compressed_log);
+ auto compressed_logs = logging::LoggerConfiguration::getCompressedLogs();
+ REQUIRE(compressed_logs.size() == 1);
+ auto logs = decompress(compressed_logs[0]);
REQUIRE(logs.find("Hi ") == std::string::npos);
REQUIRE(logs.find("Hi") != std::string::npos);
}
@@ -322,9 +326,9 @@ TEST_CASE("Setting max log entry length to a size larger than the internal buffe
std::string expected_log(1500, 'a');
logger->log_error(log.c_str());
- std::shared_ptr<InputStream> compressed_log{logging::LoggerConfiguration::getCompressedLog(true)};
- REQUIRE(compressed_log);
- auto logs = decompress(compressed_log);
+ auto compressed_logs = logging::LoggerConfiguration::getCompressedLogs();
+ REQUIRE(compressed_logs.size() == 1);
+ auto logs = decompress(compressed_logs[0]);
REQUIRE(logs.find(log) == std::string::npos);
REQUIRE(logs.find(expected_log) != std::string::npos);
}
@@ -344,8 +348,40 @@ TEST_CASE("Setting max log entry length to unlimited results in unlimited log en
std::string log(5000, 'a');
logger->log_error(log.c_str());
- std::shared_ptr<InputStream> compressed_log{logging::LoggerConfiguration::getCompressedLog(true)};
- REQUIRE(compressed_log);
- auto logs = decompress(compressed_log);
+ auto compressed_logs = logging::LoggerConfiguration::getCompressedLogs();
+ REQUIRE(compressed_logs.size() == 1);
+ auto logs = decompress(compressed_logs[0]);
REQUIRE(logs.find(log) != std::string::npos);
}
+
+TEST_CASE("Test sending multiple segments at once", "[ttl16]") {
+ auto& log_config = logging::LoggerConfiguration::getConfiguration();
+ LoggerTestAccessor::setCompressionCompressedSegmentSize(log_config, 100);
+ LoggerTestAccessor::setCompressionCacheSegmentSize(log_config, 100);
+ auto properties = std::make_shared<logging::LoggerProperties>();
+ // by default the root logger is OFF
+ properties->set("logger.root", "INFO");
+ log_config.initialize(properties);
+ auto logger = log_config.getLogger("CompressionTestMultiSegment");
+
+ std::random_device rd;
+ std::mt19937 eng(rd());
+ constexpr const char * TEXT_CHARS = "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ1234567890";
+ const int index_of_last_char = gsl::narrow<int>(strlen(TEXT_CHARS)) - 1;
+ std::uniform_int_distribution<> distr(0, index_of_last_char);
+ std::vector<char> data(100);
+ std::string log_str;
+ const size_t SEGMENT_COUNT = 5;
+ for (size_t idx = 0; idx < SEGMENT_COUNT; ++idx) {
+ std::generate_n(data.begin(), data.size(), [&] { return TEXT_CHARS[static_cast<uint8_t>(distr(eng))]; });
+ log_str = std::string{data.begin(), data.end()} + "." + std::to_string(idx);
+ logger->log_error(log_str.c_str());
+ }
+
+ LoggerTestAccessor::runCompression(log_config);
+
+ auto compressed_logs = logging::LoggerConfiguration::getCompressedLogs();
+ REQUIRE(compressed_logs.size() == SEGMENT_COUNT);
+ auto logs = decompress(compressed_logs[SEGMENT_COUNT - 1]);
+ REQUIRE(logs.find(log_str) != std::string::npos);
+}
[nifi-minifi-cpp] 02/05: MINIFICPP-2209 Small HTTPClient refactor: preconditions and exception safety
Posted by fg...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
fgerlits pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git
commit 6ac5cb94d5d2a03a3be45617c5baee115e166ab6
Author: Marton Szasz <sz...@apache.org>
AuthorDate: Fri Sep 1 01:00:38 2023 +0200
MINIFICPP-2209 Small HTTPClient refactor: preconditions and exception safety
Signed-off-by: Ferenc Gerlits <fg...@gmail.com>
This closes #1648
---
extensions/http-curl/client/HTTPClient.cpp | 30 ++++++++++++------------------
extensions/http-curl/client/HTTPClient.h | 6 ++++--
2 files changed, 16 insertions(+), 20 deletions(-)
diff --git a/extensions/http-curl/client/HTTPClient.cpp b/extensions/http-curl/client/HTTPClient.cpp
index 6e8d41100..8d73f62a4 100644
--- a/extensions/http-curl/client/HTTPClient.cpp
+++ b/extensions/http-curl/client/HTTPClient.cpp
@@ -16,18 +16,15 @@
*/
#include "HTTPClient.h"
-#include <algorithm>
#include <cinttypes>
#include <memory>
#include <string>
#include <utility>
#include <vector>
-#include "Exception.h"
#include "utils/gsl.h"
#include "utils/StringUtils.h"
#include "core/Resource.h"
-#include "utils/RegexUtils.h"
#include "range/v3/algorithm/all_of.hpp"
#include "range/v3/action/transform.hpp"
#include "utils/HTTPUtils.h"
@@ -328,18 +325,16 @@ void HTTPClient::setRequestHeader(std::string key, std::optional<std::string> va
}
namespace {
-struct CurlSListFreeAll {
- void operator()(struct curl_slist* slist) const {
- curl_slist_free_all(slist);
- }
-};
+using CurlSlistDeleter = decltype([](struct curl_slist* slist) { curl_slist_free_all(slist); });
-std::unique_ptr<struct curl_slist, CurlSListFreeAll> getCurlSList(const std::unordered_map<std::string, std::string>& request_headers) {
- curl_slist* new_list = nullptr;
+std::unique_ptr<struct curl_slist, CurlSlistDeleter> toCurlSlist(const std::unordered_map<std::string, std::string>& request_headers) {
+ gsl::owner<curl_slist*> new_list = nullptr;
+ const auto guard = gsl::finally([&new_list]() { curl_slist_free_all(new_list); });
for (const auto& [header_key, header_value] : request_headers)
- new_list = curl_slist_append(new_list, utils::StringUtils::join_pack(header_key, ": ", header_value).c_str());
+ new_list = (utils::optional_from_ptr(curl_slist_append(new_list, utils::StringUtils::join_pack(header_key, ": ", header_value).c_str()))
+ | utils::orElse([]() { throw std::runtime_error{"curl_slist_append failed"}; })).value();
- return {new_list, {}};
+ return {std::exchange(new_list, nullptr), {}};
}
} // namespace
@@ -366,7 +361,7 @@ bool HTTPClient::submit() {
curl_easy_setopt(http_session_.get(), CURLOPT_NOPROGRESS, 1);
}
- auto headers = getCurlSList(request_headers_);
+ const auto headers = toCurlSlist(request_headers_);
if (headers) {
curl_slist_append(headers.get(), "Expect:");
curl_easy_setopt(http_session_.get(), CURLOPT_HTTPHEADER, headers.get());
@@ -452,7 +447,8 @@ void HTTPClient::set_request_method(std::string method) {
}
int HTTPClient::onProgress(void *clientp, curl_off_t /*dltotal*/, curl_off_t dlnow, curl_off_t /*ultotal*/, curl_off_t ulnow) {
- HTTPClient& client = *reinterpret_cast<HTTPClient*>(clientp);
+ gsl_Expects(clientp);
+ HTTPClient& client = *static_cast<HTTPClient*>(clientp);
auto now = std::chrono::steady_clock::now();
auto elapsed = now - client.progress_.last_transferred_;
if (dlnow != client.progress_.downloaded_data_ || ulnow != client.progress_.uploaded_data_) {
@@ -522,12 +518,10 @@ std::string HTTPClient::replaceInvalidCharactersInHttpHeaderFieldName(std::strin
return "X-MiNiFi-Empty-Attribute-Name";
}
- std::string result;
- result.reserve(field_name.size());
// RFC822 3.1.2: The field-name must be composed of printable ASCII characters
// (i.e., characters that have values between 33. and 126., decimal, except colon).
- ranges::actions::transform(field_name, [](char ch) {
- return (ch >= 33 && ch <= 126 && ch != ':') ? ch : '-';
+ ranges::actions::transform(field_name, [](char ch) { // NOLINT: false positive: Add #include <algorithm> for transform [build/include_what_you_use] [4]
+ return (ch >= 33 && ch <= 126 && ch != ':') ? ch : '-';
});
return field_name;
}
diff --git a/extensions/http-curl/client/HTTPClient.h b/extensions/http-curl/client/HTTPClient.h
index d6a1894f7..144edb305 100644
--- a/extensions/http-curl/client/HTTPClient.h
+++ b/extensions/http-curl/client/HTTPClient.h
@@ -207,8 +207,10 @@ class HTTPClient : public utils::BaseHTTPClient, public core::Connectable {
protected:
static CURLcode configure_ssl_context(CURL* /*curl*/, void *ctx, void *param) {
#ifdef OPENSSL_SUPPORT
- auto* ssl_context_service = static_cast<minifi::controllers::SSLContextService*>(param);
- if (!ssl_context_service->configure_ssl_context(static_cast<SSL_CTX*>(ctx))) {
+ gsl_Expects(ctx);
+ gsl_Expects(param);
+ auto& ssl_context_service = *static_cast<minifi::controllers::SSLContextService*>(param);
+ if (!ssl_context_service.configure_ssl_context(static_cast<SSL_CTX*>(ctx))) {
return CURLE_FAILED_INIT;
}
return CURLE_OK;
[nifi-minifi-cpp] 05/05: MINIFICPP-2223 Add .asf.yaml file to configure repository labels
Posted by fg...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
fgerlits pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git
commit 73ccd5eefea4fce0ff55269ab4150f856f19aa32
Author: Gabor Gyimesi <ga...@gmail.com>
AuthorDate: Tue Sep 19 17:46:57 2023 +0200
MINIFICPP-2223 Add .asf.yaml file to configure repository labels
Signed-off-by: Ferenc Gerlits <fg...@gmail.com>
This closes #1663
---
.asf.yaml | 36 ++++++++++++++++++++++++++++++++++++
1 file changed, 36 insertions(+)
diff --git a/.asf.yaml b/.asf.yaml
new file mode 100644
index 000000000..0472044ca
--- /dev/null
+++ b/.asf.yaml
@@ -0,0 +1,36 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+github:
+ description: "Apache NiFi - MiNiFi C++"
+ homepage: https://nifi.apache.org/minifi/
+ labels:
+ - nifi
+ - minifi
+ - cpp
+ - hacktoberfest
+ features:
+ wiki: false
+ issues: false
+ projects: false
+ enabled_merge_buttons:
+ squash: true
+ merge: true
+ rebase: true
+notifications:
+ commits: commits@nifi.apache.org
+ issues: issues@nifi.apache.org
+ pullrequests: issues@nifi.apache.org
+ jira_options: link label worklog
[nifi-minifi-cpp] 04/05: MINIFICPP-2222 Update CONTRIB.md file with new contributor access request process
Posted by fg...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
fgerlits pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git
commit d9537edf034b0bbf011771513bdee154fce3bd8b
Author: Gabor Gyimesi <ga...@gmail.com>
AuthorDate: Tue Sep 19 17:27:34 2023 +0200
MINIFICPP-2222 Update CONTRIB.md file with new contributor access request process
Signed-off-by: Ferenc Gerlits <fg...@gmail.com>
This closes #1662
---
CONTRIB.md => CONTRIBUTING.md | 16 ++++++++--------
1 file changed, 8 insertions(+), 8 deletions(-)
diff --git a/CONTRIB.md b/CONTRIBUTING.md
similarity index 90%
rename from CONTRIB.md
rename to CONTRIBUTING.md
index 482955001..694901a26 100644
--- a/CONTRIB.md
+++ b/CONTRIBUTING.md
@@ -19,7 +19,7 @@ We welcome all contributions to Apache MiNiFi. All new files must include a copy
To make development easier, we've included the linter for the Google Style guide. Google provides an Eclipse formatter
for their style guide. It is located
[here](https://github.com/google/styleguide/blob/gh-pages/eclipse-cpp-google-style.xml).
-New contributions are expected to follow the
+New contributions are expected to follow the
[Google C++ Style Guide](https://google.github.io/styleguide/cppguide.html), except for the following points:
- Use .cpp extension for implementation files
- Use lowerCamelCase for functions, including accessors/mutators
@@ -67,13 +67,13 @@ as part of the verification process, that is applied to all new contributions.
## Issues and Pull Requests
Issues within MiNiFi C++ are tracked in the official [Apache JIRA](https://issues.apache.org/jira/projects/MINIFICPP/issues).
-New users can register freely and ask for contributor access on the
-[Developers Mailing List](https://nifi.apache.org/mailing_lists.html).
-Unassigned tickets may be assigned to yourself or filed via this JIRA instance. Pull requests can be submitted via our
-[Github Mirror](https://github.com/apache/nifi-minifi-cpp) . When doing so please create a JIRA issue for your pull
-request.
+New users can freely request JIRA access on the
+[Self-Service Portal](https://selfserve.apache.org/jira-account.html) for the nifi project. An Apache PMC member will review and
+approve the request as prudent. Unassigned tickets may be assigned to yourself or filed via this JIRA instance. As a new user looking to contribute,
+any tickets that are tagged as "beginner", "starter", or "newbie" are good places to start. Pull requests can be submitted via our
+[Github](https://github.com/apache/nifi-minifi-cpp). If your contribution does not have a corresponding JIRA issue for it then please create one.
-Apache NiFi MiNiFi C++ is a review then commit community. As a result, we will review your commits and merge them following
+Apache NiFi MiNiFi C++ is a review then commit community. As a result, we will review your commits and merge them following
review. We ask that you provide tests and documentation when possible. Typically PRs are merged after they get 2-3
approvals, usually in a week or two.
@@ -94,7 +94,7 @@ This will provide output for all source files.
Please see [ThirdParties.md](ThirdParties.md) on how MiNiFi builds and uses third party libraries and how you can add new ones.
-### Extensions
+### Extensions
MiNiFi C++ contains a dynamic loading mechanism that loads arbitrary objects. To maintain consistency of development amongst the NiFi ecosystem, it is called a class loader. If you
are contributing a custom Processor or Controller Service, the mechanism to register your class into the default class loader is a pragma definition named: