[GitHub] yifeih commented on a change in pull request #107: Integrate encryption into datasource

[GitBox <gi...@apache.org>]

yifeih commented on a change in pull request #107: Integrate encryption into datasource
URL: https://github.com/apache/incubator-iceberg/pull/107#discussion_r258654537
 
 

 ##########
 File path: spark/src/main/java/com/netflix/iceberg/spark/source/Reader.java
 ##########
 @@ -273,14 +282,23 @@ private Schema lazyExpectedSchema() {
     private Iterator<InternalRow> currentIterator = null;
     private Closeable currentCloseable = null;
     private InternalRow current = null;
+    private Iterator<InputFile> inputFiles = null;
 
 Review comment:
   Hmm ok I wasn't what the best way to do this was because I need to get all the `InputFiles` into a single iterator to pipe them through the `EncryptionManager`, but then I'd have to group them back with their respective FileScanTasks. A few options that I thought of are:
   
   1. Create an Iterator that combines the two different iterators, ensuring that they're always used at the same time (see latest commit)
   2. Create a map from the `FileScanTask` to the `InputFile` locations, as well as a map from the `InputFile` location to the decrypted `InputFile`, and use those maps to associate the appropriate decrypted `InputFile` with the `FileScanTask`... lots of bookkeeping though and a bit confusing. 
   3. Don't use the encryption batch API. This makes it pretty straightforward to keep everything as a single iterator. 
   
   Please let me know if I'm missing something though! 

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org