[GitHub] [spark] HeartSaVioR edited a comment on issue #21339: [SPARK-24287][Core] Spark -packages option should support classifier, no-transitive, and custom conf

[GitBox <gi...@apache.org>]

HeartSaVioR edited a comment on issue #21339: [SPARK-24287][Core] Spark -packages option should support classifier, no-transitive, and custom conf
URL: https://github.com/apache/spark/pull/21339#issuecomment-476107562
 
 
   I've pulled this patch and played with functionalities, and works well.
   
   I've used hive-exec which has default classifier as well as `core` classifier: hive-exec with default classifier includes some dependencies into jar (`org.json:json` is included) which `core` classifier contains it as dependency (not included in jar).
   
   > ./bin/spark-shell
   
   ```
   scala> import org.json.JSON
   <console>:24: error: object json is not a member of package org
          import org.json.JSON
   ```
   
   Not able to load since it's not in default Spark classpath. (expected)
   
   > ./bin/spark-shell --packages "org.apache.hive:hive-exec:3.1.1?classifier=core"
   
   ```
   scala> import org.json.JSON
   import org.json.JSON
   ```
   
   With classifier it properly loads it as transitive dependencies. (works well)
   
   > ./bin/spark-shell --packages "org.apache.hive:hive-exec:3.1.1?classifier=core&transitive=false"
   
   ```
   scala> import org.json.JSON
   <console>:23: error: object json is not a member of package org
          import org.json.JSON
   ```
   
   It ignores whole transitive dependencies, hence `org.json:json` is not pulled. (works well)
   
   > ./bin/spark-shell --packages "org.apache.hive:hive-exec:3.1.1?classifier=core&exclude=json"
   
   ```
   scala> import org.json.JSON
   <console>:23: error: object json is not a member of package org
          import org.json.JSON
   ```
   
   It excludes `org.json:json` from transitive dependencies. (works well)
   
   > ./bin/spark-shell --packages "org.apache.hive:hive-exec:3.1.1"
   
   ```
   scala> import org.json.JSON
   import org.json.JSON
   ```
   
   The jar itself contains `org.json:json` so should be able to load. (This represents classifier works as expected.)
   
   > ./bin/spark-shell --packages "org.apache.hive:hive-exec:3.1.1?exclude=json"
   
   ```
   scala> import org.json.JSON
   import org.json.JSON
   ```
   
   Even we exclude `org.json:json` it's still able to load, since the jar contains it. (This represents classifier works as expected.)
   
   FYI I had to fix scalastyle before building. I'll start reviewing the code and leave comments.
   
   cc. @srowen @gaborgsomogyi You might be interested on this PR.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org