MR3 1.1 released

[Sungwoo Park <gl...@gmail.com>]

We are pleased to announce the release of MR3 1.1. Three main improvements
in MR3 1.1 are:

1. Hive on MR3 on Kubernetes now runs almost as fast as Hive on MR3 on
Hadoop. For experimental results, please see a new blog article "Why you
should run Hive on Kubernetes, even in a Hadoop cluster".

https://www.datamonad.com/post/2020-07-19-why-hive-k8s/

2. Fetch delays rarely occur thanks to: 1) speculative execution and 2)
support for multiple shuffle handlers in a single ContainerWorker. For
experimental results, please see the following page in MR3docs:

https://mr3docs.datamonad.com/docs/mr3/features/fetchdelay/

3. Hive 4 on MR3 now runs stable (after applying HIVE-23114 which fixes a
design bug reported by the MR3 team).

Please visit https://mr3docs.datamonad.com/ for the full documentation on
MR3. The quick start guide for Hive on MR3 on Kubernetes has been updated
for MR3 1.1 (https://mr3docs.datamonad.com/docs/quick/).

For Hive on MR3 on Kubernetes, one can quickly test with a pre-built Docker
image available at DockerHub (https://hub.docker.com/u/mr3project) and the
scripts available at GitHub (https://github.com/mr3project/mr3-run-k8s/).

Thank you!

--- Sungwoo

=== Features and improvements in MR3 1.1

## MR3

Mapping DAGs to Task queues:
https://mr3docs.datamonad.com/docs/mr3/features/dag-scheduling/
Multiple shuffle handlers in a single ContainerWorker:
https://mr3docs.datamonad.com/docs/mr3/features/shufflehandler/
Speculative execution:
https://mr3docs.datamonad.com/docs/mr3/features/speculative/
Eliminating fetch delays:
https://mr3docs.datamonad.com/docs/mr3/features/fetchdelay/
Running shuffle handlers in a separate process on Kubernetes:
https://mr3docs.datamonad.com/docs/mr3/guide/use-shufflehandler/

## Hive on MR3 on Kubernetes

Fast recovery in Hive on MR3 on Kubernetes:
https://mr3docs.datamonad.com/docs/k8s/features/recovery/
Using HDFS instead of PersistentVolumes:
https://mr3docs.datamonad.com/docs/k8s/advanced/use-hdfs/
Using Amazon S3 instead of PersistentVolumes:
https://mr3docs.datamonad.com/docs/k8s/advanced/use-s3/
Configuring kernel parameters in Docker containers:
https://mr3docs.datamonad.com/docs/k8s/advanced/configure-kernel/
Performance tuning for Hive on MR3 on Kubernetes:
https://mr3docs.datamonad.com/docs/k8s/advanced/performance-tuning/
Using S3 instead of EFS on Amazon EKS:
https://mr3docs.datamonad.com/docs/k8s/eks/use-s3/

=== Release notes for MR3 1.1

## MR3

  - Support DAG scheduling schemes (specified by `mr3.dag.queue.scheme`).
  - Optimize DAGAppMaster by freeing memory for messages to Tasks when
fault tolerance is disabled (with `mr3.am.task.max.failed.attempts` set to
1).
  - Fix a minor memory leak in DaemonTask (which also prevents MR3 from
running more than 2^30 DAGs when using the shuffle handler).
  - Improve the chance of assigning TaskAttempts to ContainerWorkers that
match location hints.
  - TaskScheduler can use location hints produced by `ONE_TO_ONE` edges.
  - TaskScheduler can use location hints from HDFS when assigning
TaskAttempts to ContainerWorker Pods on Kubernetes (with `
mr3.convert.container.address.host.name`).
  - Introduce `mr3.k8s.pod.cpu.cores.max.multiplier` to specify the
multiplier for the limit of CPU cores.
  - Introduce `mr3.k8s.pod.memory.max.multiplier` to specify the multiplier
for the limit of memory.
  - Introduce `mr3.k8s.pod.worker.security.context.sysctls` to configure
kernel parameters of ContainerWorker Pods using init containers.
  - Support speculative execution of TaskAttempts (with
`mr3.am.task.concurrent.run.threshold.percent`).
  - A ContainerWorker can run multiple instances of shuffle handlers each
with a different port. The configuration key
`mr3.use.daemon.shufflehandler` now specifies the number of shuffle handler
instances in each ContainerWorker.
  - With speculative execution and multiple instances of shuffle handlers
in a single ContainerWorker, fetch delays rarely occur.
  - A ContainerWorker Pod can run shuffle handlers in a separate container
(with `mr3.k8s.shuffle.process.ports`).
  - On Kubernetes, DAGAppMaster uses ReplicationController instead of Pod,
thus making recovery much faster.
  - On Kubernetes, ConfigMaps `mr3conf-configmap-master` and
`mr3conf-configmap-worker` survive MR3, so the user should delete them
manually.
  - Java 8u251/8u252 can be used on Kubernetes 1.17 and later.

## Hive on MR3 on Hadoop

  - CrossProductHandler asks MR3 DAGAppMaster to set
TEZ_CARTESIAN_PRODUCT_MAX_PARALLELISM (Cf. HIVE-16690, Hive 3/4).
  - Hive 4 on MR3 is stable (currently using 4.0.0-SNAPSHOT).
  - No longer support Hive 1.

## Hive on MR3 on Kubernetes

  - Ranger uses a local directory (emptyDir volume) for logging.
  - The open file limit for Solr (in Ranger) is not limited to 1024.
  - HiveServer2 and DAGAppMaster create readiness and liveness probes.