[GitHub] [spark] shaneknapp edited a comment on issue #24379: [SPARK-25079][python][branch-2.4] update python3 executable to 3.6.x

[GitBox <gi...@apache.org>]

shaneknapp edited a comment on issue #24379: [SPARK-25079][python][branch-2.4] update python3 executable to 3.6.x
URL: https://github.com/apache/spark/pull/24379#issuecomment-483764704
 
 
   > Is it possible to keep pandas and pyarrow versions the same as before (0.19.2 and 0.8.0) for envs of branches 2.3/2.4 or do they need to share the same env as master?
   > 
   they need to share the same env as master (or we change all of the testing framework for all branches to create temporary python envs for each branch....  which isn't actually a horrible idea but a much bigger project).
   
   regarding pandas 0.19.2, it seems that pandas 0.24.2 is the minimum according to conda forge?
   
   (output below trimmed for readability)
   ```
   $ conda install -c conda-forge pyarrow=0.12.1
   <snip>
     added / updated specs:
       - pyarrow=0.12.1
   
   <snip>
   
   The following NEW packages will be INSTALLED:
   
     arrow-cpp          conda-forge/linux-64::arrow-cpp-0.12.1-py36h0e61e49_0
     mkl_fft            conda-forge/linux-64::mkl_fft-1.0.11-py36h14c3975_0
     mkl_random         conda-forge/linux-64::mkl_random-1.0.2-py36h637b7d7_2
     parquet-cpp        conda-forge/noarch::parquet-cpp-1.5.1-4
     pyarrow            conda-forge/linux-64::pyarrow-0.12.1-py36hbbcf98d_0
   
   The following packages will be UPDATED:
   
     numpy                              1.11.3-py36h7e9f1db_12 --> 1.16.2-py36h7e9f1db_0
     numpy-base                         1.11.3-py36hde5b4d6_12 --> 1.16.2-py36hde5b4d6_0
     pandas                                 0.19.2-np111py36_1 --> 0.24.2-py36hf484d3e_0   <-----  NOOOOO!
   ```
   
   i'll try and see if i can get pandas to 0.19.2, but it's looking to be kinda difficult.  i hacked a conda spec file and manually set pandas to 0.19.2 and will run the tests against it and see what happens.
   
   > The failures here have been fixed in master from various PRs, but not backported. It's possible to apply them, but it would take some time and could be a bit risky..
   
   i'll take time/risk vs not having any python test coverage for 2.3 and 2.4...  but we'll need a commitment from dev@ to help make this stuff work.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org