[GitHub] steveloughran opened a new pull request #515: HADOOP-16134 001- initial design of a WriteOperationsContext

[GitBox <gi...@apache.org>]

steveloughran opened a new pull request #515: HADOOP-16134 001- initial design of a WriteOperationsContext
URL: https://github.com/apache/hadoop/pull/515
 
 
   *Does not compile*
   
   This adds
   * a context which is passed round with writes
   * a parent delete policy as part of this (unused)
   
   This PoC shows that adding a new context everywhere is overcomplex as you now need to retrofit it through the stack, even though a  (single, shared) WriteOperationsHelper is already passed in
   
   This doesn't compile: I put it together while half-listening to an online talk, and now I've done I've learned enough to say "not the right approach"
   
   Better strategy:
   * include the WriteOperationsContext in the WriteOperationsHelper;  instantiating a new one each time. This will automatically add it to all bits of the FS code which write data
   * add a default/configurable delete policy to the FS, *but allow operations to explicitly overwrite this*. Example: completing all the committed work in a job commit, because we can rely on the write of the _SUCCESS file to do the work (so only do it for one file, not every file created)
   
   We're also a bit constrained by how the MPU API of HADOOP-13186 tries to be independent of the FS instance -this is one of those cases where it complicates life even more. The FS/FC MUST be the factory for MPU instances.
   
   Change-Id: I0de1d4b97fdf4c4f0ece1a27245ba9bb38a29559
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org