[GitHub] [incubator-kyuubi] bowenliang123 commented on issue #3896: [Improvement] Support to execute python language with yarn cluster mode

[GitBox <gi...@apache.org>]

bowenliang123 commented on issue #3896:
URL: https://github.com/apache/incubator-kyuubi/issues/3896#issuecomment-1336421144

   It's a known unsatisfied capability as initial pyspark support was introduced.
   The challenge for preparing pyspark env for the python session worker, that comes with,
   1. make cluster mode upload the pyspark module or python folder 
   2. or install pyspark to python on every node in cluster mode
   3. correctly identify SPARK_HOME with pyspark module and tell the engine use it
   
   A possible temporary workaround in my mind is to fallback to pyspark installed on cluster nodes, and defer error message.
   1. allow empty $SPARK_HOME env before executing `execute_python.py`
   2. rely on python env on cluster nodes for the pyspark spport.
   3. check `if "pyspark" not in sys.modules:` with empty $SPARK_HOME env and thow the error message


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org