[jira] [Created] (FLINK-37932) Set a Default User Agent for GCS FileSystem Connector for Better Observability

["Shruti Singhania (Jira)" <ji...@apache.org>]

Shruti Singhania created FLINK-37932:
----------------------------------------

             Summary: Set a Default User Agent for GCS FileSystem Connector for Better Observability
                 Key: FLINK-37932
                 URL: https://issues.apache.org/jira/browse/FLINK-37932
             Project: Flink
          Issue Type: Improvement
          Components: Connectors / Hadoop Compatibility, FileSystems
            Reporter: Shruti Singhania


*1. Problem Statement*

When Apache Flink interacts with Google Cloud Storage (GCS) via the {{flink-gs-fs-hadoop}} connector, the requests made to the GCS API do not contain any Flink-specific identifiers by default. While users can manually configure a user agent suffix via the {{fs.gs.user.agent.suffix}} property in {{{}core-site.xml{}}}, most users do not.

This lack of a default identifier makes it difficult for users and cloud administrators to distinguish Flink-originated GCS traffic from other applications in GCP Cloud Audit Logs. This complicates monitoring, debugging, and cost attribution for Flink jobs operating in a cloud-native environment.

 

*2. Proposed Solution*

This proposal is to programmatically set a default user agent suffix within the Flink GCS filesystem factory. This suffix would be added _only if_ one is not already provided by the user in their configuration.

The proposed default user agent suffix will include the Flink version, for example: {{{}Apache Flink 1.19.0{}}}.

This provides a "sensible default" that enhances the experience for users running Flink on Google Cloud, while fully respecting any custom configuration.

 

*3. Implementation Details*

The change would be implemented in the {{flink-gs-fs-hadoop}} module.
 * *Module:* {{flink-gs-fs-hadoop}}
 * *Class:* {{org.apache.flink.fs.gs.GSFileSystemFactory}}
 * *Method:* {{create(URI fsUri)}}

The implementation logic is as follows:
 # In the {{create}} method, after loading the Hadoop configuration, check if the {{fs.gs.user.agent.suffix}} key is already set.
 # If the key is not set (i.e., its value is {{{}null{}}}), programmatically set it on the {{Configuration}} object. The value should be dynamically generated using {{EnvironmentInformation.getVersion()}} to ensure version accuracy.
 # If the key is already set, do nothing, preserving the user's configuration.
 # Proceed with {{FileSystem}} instantiation using the (now guaranteed to be populated) {{Configuration}} object.

 

*4. Impact on Users*
 * This is a {*}non-breaking change{*}.
 * Users who have already configured a custom {{fs.gs.user.agent.suffix}} will see no difference in behavior.
 * Users who have _not_ configured this property will automatically gain improved observability in their GCP logs without needing to make any changes.

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)