[GitHub] [incubator-iceberg] aokolnychyi commented on issue #569: Use broadcast variables in IcebergSource

[GitBox <gi...@apache.org>]

aokolnychyi commented on issue #569: Use broadcast variables in IcebergSource
URL: https://github.com/apache/incubator-iceberg/pull/569#issuecomment-560389968
 
 
   Then we can do smth like below.
   
   - Introduce `BaseHadoopFileIO`:
   ```
   public abstract class BaseHadoopFileIO implements FileIO {
   
     public abstract Configuration hadoopConf();
   
     // all existing logic from HadoopFileIO
   }
   
   ```
   
   - Make `HadoopFileIO` extend `BaseHadoopFileIO` and use Iceberg's `SerializableConfiguration`:
   
   ```
   public class HadoopFileIO implements BaseHadoopFileIO {
   
     private final SerializableConfiguration hadoopConf;
   
     public HadoopFileIO(Configuration hadoopConf) {
       this.hadoopConf = new SerializableConfiguration(hadoopConf);
     }
   
     public Configuration hadoopConf() {
       return hadoopConf.get();
     }
   }
   ```
   
   - Introduce `SparkHadoopFileIO` that extends `BaseHadoopFileIO` and uses Spark's `SerializableConfiguration`.
   
   - Replace  `HadoopFileIO` with `SparkHadoopFileIO` in `IcebergSource`.
   
   Still ugly and won't work for arbitrary `FileIO` implementations but I don't see a clean solution without extra configs.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org