[GitHub] [iceberg] stevenzwu commented on pull request #5967: Flink: Support read options in flink source

[GitBox <gi...@apache.org>]

stevenzwu commented on PR #5967:
URL: https://github.com/apache/iceberg/pull/5967#issuecomment-1331526303

   @hililiwei I understand the motivation of this change. I have two questions. It will be great to get inputs from @rdblue and @chenjunjiedada on these two questions too.
   
   1.  When `streaming=true` is set in the SQL hint options per Iceberg source, there is no ambiguity. But if `streaming=true` is set in Flink Configuration via table environment, it can be ambiguous due to the lack of prefix. if there is a naming collision with other Flink configuration, does it mean the Flink job execution mode (streaming vs  batch) or the source config (streaming vs batch)?  This is already a problem with `FlinkWriteConf`. I wish we have enforced prefix(like `connector.iceberg.`) for configs from Flink environment. For configs from table properties or Iceberg source hint options, there are no worry about collision and ambiguity and hence prefixes are not required.
   
   2. some of the read configs don't make sense to set in table environment configs, as they are tied to a specific source/table. How should handle this situation?
   
   ```
   public static final ConfigOption<Long> SNAPSHOT_ID =
         ConfigOptions.key("snapshot-id").longType().defaultValue(null);
   
     public static final ConfigOption<Long> START_SNAPSHOT_ID =
         ConfigOptions.key("start-snapshot-id").longType().defaultValue(null);
   
     public static final ConfigOption<Long> END_SNAPSHOT_ID =
         ConfigOptions.key("end-snapshot-id").longType().defaultValue(null);
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org