[GitHub] SamHjelmfelt opened a new pull request #4129: Adding YARNContainerFactory. This allows OpenWhisk to run actions on Apache Hadoop clusters.

[GitBox <gi...@apache.org>]

SamHjelmfelt opened a new pull request #4129: Adding YARNContainerFactory. This allows OpenWhisk to run actions on Apache Hadoop clusters.
URL: https://github.com/apache/incubator-openwhisk/pull/4129
 
 
   Thousands of organizations have Apache Hadoop clusters today. By implementing a YARNContainerFactory, OpenWhisk can run actions on Hadoop clusters. This will lower the barrier to adoption and expand the potential use cases for OpenWhisk.
   
   ## Description
   The YARNContainerFactory uses the Apache Hadoop Services REST API to create a single YARN service with a component for each action type. Both simple authentication and Kerberos/SPNEGO are supported. This was tested with Apache Hadoop 3.1.1.
   https://hadoop.apache.org/docs/r3.1.1/hadoop-yarn/hadoop-yarn-site/yarn-service/YarnServiceAPI.html
   The implemention is based on the MesosContainerFactory.
   
   This was first implemented using Akka HTTP, but was rewritten to use the Apache HTTP client in order to suport SPNEGO.
   
   There is a MockYARNRM in the tests directory. This mock RM simulates the YARN Resource Manager REST API and is used for the YARNContainerFactory tests.
   
   ## Related issue and scope
   - [ ] I opened an issue to propose and discuss this change (#????)
   
   ## My changes affect the following components
   - [ ] API
   - [ ] Controller
   - [ ] Message Bus (e.g., Kafka)
   - [ ] Loadbalancer
   - [x] Invoker
   - [ ] Intrinsic actions (e.g., sequences, conductors)
   - [ ] Data stores (e.g., CouchDB)
   - [x] Tests
   - [ ] Deployment
   - [ ] CLI
   - [ ] General tooling
   - [x] Documentation
   
   ## Types of changes
   - [ ] Bug fix (generally a non-breaking change which closes an issue).
   - [x] Enhancement or new feature (adds new functionality).
   - [ ] Breaking change (a bug fix or enhancement which changes existing behavior).
   
   ## Checklist:
   - [x] I signed an [Apache CLA](https://github.com/apache/incubator-openwhisk/blob/master/CONTRIBUTING.md).
   - [x] I reviewed the [style guides](https://github.com/apache/incubator-openwhisk/wiki/Contributing:-Git-guidelines#code-readiness) and followed the recommendations (Travis CI will check :).
   - [x] I added tests to cover my changes.
   - [x] My changes require further changes to the documentation.
           The website should be updated as well
   - [x] I updated the documentation where necessary.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services