[jira] [Commented] (SPARK-27911) PySpark Packages should automatically choose correct scala version

["Hyukjin Kwon (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-27911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16854151#comment-16854151 ] 

Hyukjin Kwon commented on SPARK-27911:
--------------------------------------

Hi [~marmbrus], 

the issues looks because users used Scala 2.11 complied Spark (2.4.2).

{code}
pyspark --packages io.delta:delta-core_2.12:0.1.0
{code}

Does this mean we should replace {{io.delta:delta-core_2.12:0.1.0}} to {{io.delta:delta-core_2.11:0.1.0}} automatically? If so, I am wondering if this is something that should be done in Scala side so that Scala shell can detect the right version.
https://github.com/delta-io/delta/issues/6 this issue seems describing Scala side.

Would you mind if I ask to elaborate why in this JIRA? it looks a general problem but wondering why this targets PySpark only. 


> PySpark Packages should automatically choose correct scala version
> ------------------------------------------------------------------
>
>                 Key: SPARK-27911
>                 URL: https://issues.apache.org/jira/browse/SPARK-27911
>             Project: Spark
>          Issue Type: New Feature
>          Components: PySpark
>    Affects Versions: 2.4.3
>            Reporter: Michael Armbrust
>            Priority: Major
>
> Today, users of pyspark (and Scala) need to manually specify the version of Scala that their Spark installation is using when adding a Spark package to their application. This extra configuration is confusing to users who may not even know which version of Scala they are using (for example, if they installed Spark using {{pip}}). The confusion here is exacerbated by releases in Spark that have changed the default from {{2.11}} -> {{2.12}} -> {{2.11}}.
> https://spark.apache.org/releases/spark-release-2-4-2.html
> https://spark.apache.org/releases/spark-release-2-4-3.html
> Since Spark can know which version of Scala it was compiled for, we should give users the option to automatically choose the correct version.  This could be as simple as a substitution for {{$scalaVersion}} or something when resolving a package (similar to SBTs support for automatically handling scala dependencies).
> Here are some concrete examples of users getting it wrong and getting confused:
> https://github.com/delta-io/delta/issues/6
> https://github.com/delta-io/delta/issues/63



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org