[jira] [Commented] (FLINK-39342) [Iceberg] Support hadoop.conf.* prefix to pass Hadoop configuration properties

["Xiqian Yu (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-39342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18120511#comment-18120511 ] 

Xiqian Yu commented on FLINK-39342:
-----------------------------------

Merged into master via 1a1e81c21c67dc4dd09a01888c70058eca9c6396.

>  [Iceberg] Support hadoop.conf.* prefix to pass Hadoop configuration properties
> -------------------------------------------------------------------------------
>
>                 Key: FLINK-39342
>                 URL: https://issues.apache.org/jira/browse/FLINK-39342
>             Project: Flink
>          Issue Type: New Feature
>          Components: Flink CDC
>    Affects Versions: cdc-3.5.0
>            Reporter: ericzeng
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: cdc-3.7.0
>
>
> The Iceberg pipeline connector currently relies on classpath-based Hadoop configuration files (core-site.xml, hdfs-site.xml) to configure Hadoop settings. There is no way to pass Hadoop configuration properties (e.g., S3     
>   credentials, HDFS endpoint, Kerberos settings) directly through the connector's job configuration.                                                                                                                                                                         
> *Motivation*                                                
> In many deployment environments (containerized, cloud-native, or multi-tenant clusters), users cannot easily place Hadoop XML config files on the classpath. They need a way to set Hadoop properties programmatically via the   
>   pipeline job configuration — similar to how catalog.properties.* is already supported.
> *Description*
> Add support for a new hadoop.conf.* prefix in the Iceberg pipeline sink connector. Any property with this prefix will be stripped of the prefix and applied to the underlying Hadoop Configuration object before it is passed to 
>   CatalogUtil.buildIcebergCatalog(), IcebergWriter, IcebergCommitter, and CompactionOperator.
>  
> *Example usage:*                                            
>   sink:
>     type: iceberg
>     catalog.properties.type: hadoop
>     catalog.properties.warehouse: s3a://my-bucket/warehouse
>     hadoop.conf.fs.s3a.access.key: xxxxxx
>     hadoop.conf.fs.s3a.secret.key: xxx
>     hadoop.conf.fs.s3a.endpoint: s3.us-east-1.amazonaws.com



--
This message was sent by Atlassian Jira
(v8.20.10#820010)