[GitHub] srowen edited a comment on issue #23715: [SPARK-26803][PYTHON] Add sbin subdirectory to pyspark

[GitBox <gi...@apache.org>]

srowen edited a comment on issue #23715: [SPARK-26803][PYTHON] Add sbin subdirectory to pyspark
URL: https://github.com/apache/spark/pull/23715#issuecomment-459518081
 
 
   The pyspark pip package is the client side... I don't even think it would have the Spark binaries to run, like the history server. I don't think pip is a good way to distribute all that. It would be 100s of MB of JAR files that almost no app actually needs.
   
   You're meant to run Spark (which is more than the Pyspark wrapper) by getting a distribution of, well, all of Spark.
   
   EDIT: I looked at my pyspark install and it does actually have a bunch of JARs, hm. I presume this is for running in local mode, but here I'm out of my depth. Maybe that's good news? if you really mean to just run spark locally. But yeah for managing a standalone cluster or the history server, that's really beyond pyspark and you'd need a full distribution.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org