[GitHub] [iceberg] hililiwei commented on pull request #5967: Flink: Support read options in flink source

[GitBox <gi...@apache.org>]

hililiwei commented on PR #5967:
URL: https://github.com/apache/iceberg/pull/5967#issuecomment-1277151870

   > @hililiwei can you explain the value of this change? isn't `ScanContext` serving the same purpose?
   
   hi, @stevenzwu, thanks for your review. 
   Let me try to explain the motive.
   1. The current ScanContext, which lumps configuration items and parsing together, is bloated.
   2. Even when it comes to parsing the configuration, ScanContext doesn't seem to do a good enough job. Currently, there is no unified way to parse these configuration: 1. The source and priority of each configuration is clearly defined in `Flink Read Conf`, which tells us whether the configuration can come from `Table Properties`, `Session`, or `Options`, which is not done in `ScanContext`. 2. At the same time, its parsing is incomplete. The `fromProperties(Map<String, String> properties)` of ScanContext creates a new `Configuration`, which ignores the value set by the `Set k=v` in flink Session. In our use, The `Set` does not take effect.  
   3. We support many available configurations, but they are not described in the documentation. I think this is bad for users. This makes the user think we're missing functionality.
   
   As I wrote this code, I considered removing `ScanContext` to fully align to `Flink Read Conf`, which would bring it exactly in line with the style of Flik Sink.
   
   What's your opinion? Look forward to hearing from you.
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org