You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "ericzeng (Jira)" <ji...@apache.org> on 2026/03/27 07:34:00 UTC

[jira] [Created] (FLINK-39342) [Iceberg] Support hadoop.conf.* prefix to pass Hadoop configuration properties

ericzeng created FLINK-39342:
--------------------------------

             Summary:  [Iceberg] Support hadoop.conf.* prefix to pass Hadoop configuration properties
                 Key: FLINK-39342
                 URL: https://issues.apache.org/jira/browse/FLINK-39342
             Project: Flink
          Issue Type: New Feature
          Components: Flink CDC
    Affects Versions: cdc-3.5.0
            Reporter: ericzeng


The Iceberg pipeline connector currently relies on classpath-based Hadoop configuration files (core-site.xml, hdfs-site.xml) to configure Hadoop settings. There is no way to pass Hadoop configuration properties (e.g., S3     
  credentials, HDFS endpoint, Kerberos settings) directly through the connector's job configuration.                                                                                                                                                                         
*Motivation*                                                

In many deployment environments (containerized, cloud-native, or multi-tenant clusters), users cannot easily place Hadoop XML config files on the classpath. They need a way to set Hadoop properties programmatically via the   
  pipeline job configuration — similar to how catalog.properties.* is already supported.

*Description*

Add support for a new hadoop.conf.* prefix in the Iceberg pipeline sink connector. Any property with this prefix will be stripped of the prefix and applied to the underlying Hadoop Configuration object before it is passed to 
  CatalogUtil.buildIcebergCatalog(), IcebergWriter, IcebergCommitter, and CompactionOperator.

 

*Example usage:*                                            

  sink:
    type: iceberg
    catalog.properties.type: hadoop
    catalog.properties.warehouse: s3a://my-bucket/warehouse
    hadoop.conf.fs.s3a.access.key: xxxxxx
    hadoop.conf.fs.s3a.secret.key: xxx
    hadoop.conf.fs.s3a.endpoint: s3.us-east-1.amazonaws.com



--
This message was sent by Atlassian Jira
(v8.20.10#820010)