You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "featzhang (Jira)" <ji...@apache.org> on 2026/03/31 11:14:00 UTC

[jira] [Created] (FLINK-39367) [connector-http] Add configurable error logging with detailed context

featzhang created FLINK-39367:
---------------------------------

             Summary: [connector-http] Add configurable error logging with detailed context
                 Key: FLINK-39367
                 URL: https://issues.apache.org/jira/browse/FLINK-39367
             Project: Flink
          Issue Type: Improvement
          Components: Connectors / HTTP
            Reporter: featzhang


  h2. Background

  Currently, when the HTTP Connector encounters errors (request failures, unexpected status codes, etc.), the log output provides minimal information,
  making it difficult to diagnose issues in production environments. The existing {{http.logging.level}} mechanism only controls the verbosity of
  request/response content logging (MIN/REQ_RESP/MAX) and is fixed at DEBUG level, which does not address the following needs:
  - Logging errors at WARN or INFO level instead of only DEBUG
  - Capturing richer context at error time (HTTP method, URL, response status code, retry attempt number)
  - Optionally including request/response bodies in error logs, with a configurable size limit to prevent log flooding
  - Automatically masking sensitive headers (e.g., Authorization, Cookie, api-key) to avoid credential exposure

  h2. Requirements

  Introduce a new {{HttpErrorLogger}} class providing configurable error log level and verbosity

  Add the following three new configuration options:

  ** {{http.error.log.level}} — log level for error events; one of {{ERROR}}/{{WARN}}/{{INFO}}, default {{ERROR}}
  ** {{http.error.log.include.body}} — whether to include request/response body in error logs, default {{false}}
  ** {{http.error.log.body.max.size}} — maximum number of characters to log from body content, default {{1024}}
  Sensitive headers (authorization, cookie, api-key, x-api-key) must be automatically masked in all error log output

  Refactor {{GenericJsonAndUrlQueryCreator}} to replace the regex-based {{http.request.body-template}} with a more explicit and type-safe approach using
  {{http.request.body-fields}} and {{http.request.additional-body-json}}

  h2. Proposed Solution

  - Add {{HttpErrorLogLevel}} enum (ERROR / WARN / INFO)
  - Add {{HttpErrorLogger}} class implementing {{Serializable}} for use in distributed Flink environments; reads configuration from {{Properties}} at
  construction time
  - Integrate {{HttpErrorLogger}} into both {{JavaNetHttpPollingClient}} (lookup source) and {{JavaNetSinkHttpClient}} (sink)
  - Refactor {{GenericJsonAndUrlQueryCreatorFactory}}:
  ** Replace {{http.request.body-template}} (regex placeholder substitution) with:
  *** {{http.request.body-fields}}: explicit list of column names to include in the request body for PUT/POST operations
  *** {{http.request.additional-body-json}}: a static JSON object string whose fields are merged into the generated request body; parsed once at factory
  initialization for runtime efficiency
  ** Validate at factory creation time that {{additional-body-json}} fields do not conflict with join key fields defined in {{body-fields}}

  h2. Notes

  {{HttpErrorLogger}} and the existing {{HttpLogger}} serve complementary purposes:
  - {{HttpLogger}}: controls the verbosity of HTTP content logging (MIN / REQ_RESP / MAX), always at DEBUG level — intended for development and
  troubleshooting
  - {{HttpErrorLogger}}: controls the severity level at which errors are logged (ERROR / WARN / INFO) — intended for production operational monitoring

  These two mechanisms are independent and do not conflict. Users may configure both as needed.

  h2. Related PR

  https://github.com/apache/flink-connector-http/pull/35




--
This message was sent by Atlassian Jira
(v8.20.10#820010)