[GitHub] [incubator-iceberg] massdosage commented on pull request #933: [WIP] Mapred input format

[GitBox <gi...@apache.org>]

massdosage commented on pull request #933:
URL: https://github.com/apache/incubator-iceberg/pull/933#issuecomment-624723722


   This is ready for another round of review. We've tried to refactor as much of the shared code between the InputFormats as possible but it's still not perfect. What we'd like input on:
   
   1. General feedback on the code layout, suggestions for improvement, any other shared code that could be cleaned up.
   2. What to do with the Hive dependencies that are creeping into the InputFormat? To make it really useful in Hive we'd actually need to add more (for example you can look at our downstream project that we are adding features in before attempting to merge them here to see what a more complete InputFormat for Hive would look like - https://github.com/ExpediaGroup/hiveberg/blob/master/src/main/java/com/expediagroup/hiveberg/IcebergInputFormat.java). For a more useful Hive integration we would also add a SerDe, StorageHandler etc. We could decide to put all of that code in `iceberg-hive` or create a new `hive-exec` subproject. 
   3. Once we've decided that we need to decide how best to share code between what's in `mr` and whether it's worth having a "base" `mapred` InputFormat in `mr`and then possibly an extension of it in the hive/hive-exec subproject which extends it and adds Hive-specific functionality (this will be more complicated than just moving the whole thing there).
   4. And a gentle reminder that none of this will be usable in Hive until #935 has been completed.
   
   Thoughts?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org