You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "featzhang (Jira)" <ji...@apache.org> on 2026/03/31 11:14:00 UTC
[jira] [Created] (FLINK-39367) [connector-http] Add configurable error logging with detailed context
featzhang created FLINK-39367:
---------------------------------
Summary: [connector-http] Add configurable error logging with detailed context
Key: FLINK-39367
URL: https://issues.apache.org/jira/browse/FLINK-39367
Project: Flink
Issue Type: Improvement
Components: Connectors / HTTP
Reporter: featzhang
h2. Background
Currently, when the HTTP Connector encounters errors (request failures, unexpected status codes, etc.), the log output provides minimal information,
making it difficult to diagnose issues in production environments. The existing {{http.logging.level}} mechanism only controls the verbosity of
request/response content logging (MIN/REQ_RESP/MAX) and is fixed at DEBUG level, which does not address the following needs:
- Logging errors at WARN or INFO level instead of only DEBUG
- Capturing richer context at error time (HTTP method, URL, response status code, retry attempt number)
- Optionally including request/response bodies in error logs, with a configurable size limit to prevent log flooding
- Automatically masking sensitive headers (e.g., Authorization, Cookie, api-key) to avoid credential exposure
h2. Requirements
Introduce a new {{HttpErrorLogger}} class providing configurable error log level and verbosity
Add the following three new configuration options:
** {{http.error.log.level}} — log level for error events; one of {{ERROR}}/{{WARN}}/{{INFO}}, default {{ERROR}}
** {{http.error.log.include.body}} — whether to include request/response body in error logs, default {{false}}
** {{http.error.log.body.max.size}} — maximum number of characters to log from body content, default {{1024}}
Sensitive headers (authorization, cookie, api-key, x-api-key) must be automatically masked in all error log output
Refactor {{GenericJsonAndUrlQueryCreator}} to replace the regex-based {{http.request.body-template}} with a more explicit and type-safe approach using
{{http.request.body-fields}} and {{http.request.additional-body-json}}
h2. Proposed Solution
- Add {{HttpErrorLogLevel}} enum (ERROR / WARN / INFO)
- Add {{HttpErrorLogger}} class implementing {{Serializable}} for use in distributed Flink environments; reads configuration from {{Properties}} at
construction time
- Integrate {{HttpErrorLogger}} into both {{JavaNetHttpPollingClient}} (lookup source) and {{JavaNetSinkHttpClient}} (sink)
- Refactor {{GenericJsonAndUrlQueryCreatorFactory}}:
** Replace {{http.request.body-template}} (regex placeholder substitution) with:
*** {{http.request.body-fields}}: explicit list of column names to include in the request body for PUT/POST operations
*** {{http.request.additional-body-json}}: a static JSON object string whose fields are merged into the generated request body; parsed once at factory
initialization for runtime efficiency
** Validate at factory creation time that {{additional-body-json}} fields do not conflict with join key fields defined in {{body-fields}}
h2. Notes
{{HttpErrorLogger}} and the existing {{HttpLogger}} serve complementary purposes:
- {{HttpLogger}}: controls the verbosity of HTTP content logging (MIN / REQ_RESP / MAX), always at DEBUG level — intended for development and
troubleshooting
- {{HttpErrorLogger}}: controls the severity level at which errors are logged (ERROR / WARN / INFO) — intended for production operational monitoring
These two mechanisms are independent and do not conflict. Users may configure both as needed.
h2. Related PR
https://github.com/apache/flink-connector-http/pull/35
--
This message was sent by Atlassian Jira
(v8.20.10#820010)