[Discussion][Improvement] The management and specification of the HOME environment variables of multiple executors

[程世文 <ch...@bizseer.com>]

Hi, everyone

I want to discuss the management and specification of the HOME environment variables of multiple executors, and am looking forward to your suggestions.

It is recommended to directly refer to the issue: https://github.com/apache/incubator-dolphinscheduler/issues/5035


Describe the question

The `HOME` environment variables of multiple executors are as follows:

```
export HADOOP_HOME=/opt/soft/hadoop
export HADOOP_CONF_DIR=/opt/soft/hadoop/etc/hadoop
export SPARK_HOME1=/opt/soft/spark1
export SPARK_HOME2=/opt/soft/spark2
export PYTHON_HOME=/opt/soft/python
export JAVA_HOME=/opt/soft/java
export HIVE_HOME=/opt/soft/hive
export FLINK_HOME=/opt/soft/flink
export DATAX_HOME=/opt/soft/datax/bin/datax.py

export PATH=$HADOOP_HOME/bin:$SPARK_HOME1/bin:$SPARK_HOME2/bin:$PYTHON_HOME:$JAVA_HOME/bin:$HIVE_HOME/bin:$PATH:$FLINK_HOME/bin:$DATAX_HOME:$PATH
```

But the value of some `HOME` variables is not reasonable, like `PYTHON_HOME` and `DATAX_HOME`. Both of them are the file path, so `$PYTHON_HOME` and `$DATAX_HOME` in `PATH` will not take effect. This will make people feel confused.

**Python/Datax related issues**: #113 #708 #2548 #2620 #2868 #3853 #4122 #4158 #5018 #5024

At the same time, for multiple versions of an executor, we will encounter scalability problems, like `SPARK_HOME1` and `SPARK_HOME2`, since the Spark 3.0.0 has released on June 18, 2020 (Now the latest version is 3.1.1). And `$SPARK_HOME1/bin:$SPARK_HOME2/bin` in `PATH` can lead to influence each other.

For `PYTHON_HOME`, `python2` is usually built into the operating system. So `PYTHON_HOME` is mainly set for `python3`.

So, we need to think about the management and specification of `HOME` environment variables for multiple executors (maybe with multiple versions).

**What are the current deficiencies and the benefits of improvement**

Specifically, as shown below:

- The `HOME` environment variables of all executors should use the executor root directory instead of the binary path
- Use absolute path of an executor binary based on the `HOME` environment variable when executing a task command
- There are two solutions for multiple versions of an executor:
  - Just use **ONE** `HOME` environment variable for an executor, so `SPARK_HOME1` and `SPARK_HOME2` can be integrated into `SPARK_HOME`. Use `worker.groups` to distinguish different versions of an executor. for example, set `SPARK_HOME=/opt/spark-1.6.3` on node1 and `SPARK_HOME=/opt/spark-2.4.7` on node2, we only set `worker.groups=spark1` on node1 and `worker.groups=spark2` on node2. **The cost of this solution is small**.
  - Use **a table in the database** to manage multiple versions like `t_ds_executor_definition`, and it's managed and edited by user. The table schema contains `task type`, `executor name`, `executor HOME path` and etc. **This solution has very good scalability**.
    - This feature can fix anaconda problem like #4158 by adding `ANACONDA_HOME`

**Spark/Flink related issues**:  #839 #3359 #3744 #3785

**Which version of DolphinScheduler:**
 -[dev]


Best Regards

--
DolphinScheduler(Incubator) Contributor
Shiwen Cheng 程世文
Mobile: (+86)15201523580
Email: chengshiwen@bizseer.com