You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "George Pongracz (Jira)" <ji...@apache.org> on 2020/06/17 21:36:00 UTC

[jira] [Created] (SPARK-32017) Make Pyspark Hadoop 3.2+ Variant available in PyPI

George Pongracz created SPARK-32017:
---------------------------------------

             Summary: Make Pyspark Hadoop 3.2+ Variant available in PyPI
                 Key: SPARK-32017
                 URL: https://issues.apache.org/jira/browse/SPARK-32017
             Project: Spark
          Issue Type: Improvement
          Components: PySpark
    Affects Versions: 3.0.0
            Reporter: George Pongracz
             Fix For: 3.0.1


The version of Pyspark 3.0.0 currently available in PyPI currently uses hadoop 2.7.4.

Could a variant (or the default) have its version of Hadoop aligned to 3.2.0 as per the downloadable spark binaries.

This would enable the PyPI version to be compatible with session token authorisations and assist in accessing data residing in object stores with stronger encryption methods.

If not PyPI then as a tar file in the apache download archives at the least please.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org