do you need to add additional configuration for Python version switching and path configuration switching

[赵渊 <15...@163.com>]

Topic: do you need to add additional configuration for Python version switching and path configuration switching




CCweixiao_ intsig_ leojie:

@ peacewong@WDS Here, the user selects the python version and the corresponding Python path

After restarting the engine, I found that I had been using the path of python2 to execute the code,

The version switching and path configuration of python2 and python3 are still not very transparent to me. I don't know what you did




peacewong@WDS :

The python switch of spark is mainly based on the spark in your diagram python. Version, you can specify a path, or you can directly use Python and python 3.

Then, when starting the Python interpreter, go back to set env to reset pyspark_ Python environment variable

So you only need to set spark python. Just version




CCweixiao_ intsig_ leojie:

spark. python. Version I set the acanda path for python3python3 through spark pyspark. Python biography?

The path of python2 is also the virtual environment path of acanda




peacewong@WDS :

All through spark python. Just pass the version to the absolute path




CCweixiao_ intsig_ leojie:

It feels like it can be optimized for the purpose of planning the version. Therefore, the code also adds judgment. If it is Python 3, the variable pyspark will be read_ PYTHON3_ Path to get the specific path.

Initial configuration, because there are Python and python 3 commands on your machine by default, which is really confusing

Add pyspark_ DRIVER_ Python3, or add spark python. Path, get pyspark if you don't pass it_ DRIVER_ Python path




peacewong@WDS :

Based on the current spark python. Version may be better because compatibility upgrade should be considered. This parameter is available on the management console by default.

The main modification point may be to consider setting pyspark when starting the Python interpreter_ Set Python so that the driver and executor will be consistent.




Summary: do not add additional configuration of Python version related paths for the time being