[GitHub] [submarine] gumartinm edited a comment on pull request #319: SUBMARINE-537. missing JacksonJsonProvider in uber jar

[GitBox <gi...@apache.org>]

gumartinm edited a comment on pull request #319:
URL: https://github.com/apache/submarine/pull/319#issuecomment-647705165


   > I have checked the spark 2.4 maven dependencies
   > 
   > https://github.com/apache/spark/blob/47a73b2baac38ff2fa4e0aad63a79891bbce7cf0/dev/deps/spark-deps-hadoop-2.6#L94
   > 
   > Anything I missed?
   
   Its package is shaded because of this configuration: https://github.com/apache/submarine/blob/master/submarine-security/spark-security/pom.xml#L389
   
   If it is not included in the uber jar with its shaded package, submarine throws this exception (because it is trying to load the shaded package):
   
   `
   java.lang.NoClassDefFoundError: submarine_spark_ranger_project/org/codehaus/jackson/jaxrs/JacksonJsonProvider
   	at org.apache.ranger.plugin.util.RangerRESTClient.buildClient(RangerRESTClient.java:209)
   	at org.apache.ranger.plugin.util.RangerRESTClient.getClient(RangerRESTClient.java:176)
   	at org.apache.ranger.plugin.util.RangerRESTClient.getResource(RangerRESTClient.java:156)
   	at org.apache.ranger.admin.client.RangerAdminRESTClient.createWebResource(RangerAdminRESTClient.java:635)
   	at org.apache.ranger.admin.client.RangerAdminRESTClient.getServicePoliciesIfUpdated(RangerAdminRESTClient.java:142)
   	at org.apache.ranger.plugin.util.PolicyRefresher.loadPolicyfromPolicyAdmin(PolicyRefresher.java:251)
   	at org.apache.ranger.plugin.util.PolicyRefresher.loadPolicy(PolicyRefresher.java:191)
   	at org.apache.ranger.plugin.util.PolicyRefresher.startRefresher(PolicyRefresher.java:137)
   	at org.apache.ranger.plugin.service.RangerBasePlugin.init(RangerBasePlugin.java:229)
   	at org.apache.submarine.spark.security.RangerSparkPlugin$.init(RangerSparkPlugin.scala:43)
   	at org.apache.submarine.spark.security.RangerSparkPlugin$.<init>(RangerSparkPlugin.scala:64)
   	at org.apache.submarine.spark.security.RangerSparkPlugin$.<clinit>(RangerSparkPlugin.scala)
   	at org.apache.submarine.spark.security.RangerSparkAuthorizer$.org$apache$submarine$spark$security$RangerSparkAuthorizer$$getSparkResource(RangerSparkAuthorizer.scala:257)
   	at org.apache.submarine.spark.security.RangerSparkAuthorizer$$anonfun$addAccessRequest$1$1.apply(RangerSparkAuthorizer.scala:75)
   	at org.apache.submarine.spark.security.RangerSparkAuthorizer$$anonfun$addAccessRequest$1$1.apply(RangerSparkAuthorizer.scala:74)
   	at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
   	at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
   	at org.apache.submarine.spark.security.RangerSparkAuthorizer$.addAccessRequest$1(RangerSparkAuthorizer.scala:74)
   	at org.apache.submarine.spark.security.RangerSparkAuthorizer$.checkPrivileges(RangerSparkAuthorizer.scala:98)
   	at org.apache.spark.sql.catalyst.optimizer.SubmarineSparkRangerAuthorizationExtension.apply(SubmarineSparkRangerAuthorizationExtension.scala:64)
   	at org.apache.spark.sql.catalyst.optimizer.SubmarineSparkRangerAuthorizationExtension.apply(SubmarineSparkRangerAuthorizationExtension.scala:39)
   	at org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$execute$1$$anonfun$apply$1.apply(RuleExecutor.scala:87)
   	at org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$execute$1$$anonfun$apply$1.apply(RuleExecutor.scala:84)
   	at scala.collection.LinearSeqOptimized$class.foldLeft(LinearSeqOptimized.scala:124)
   	at scala.collection.immutable.List.foldLeft(List.scala:84)
   	at org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$execute$1.apply(RuleExecutor.scala:84)
   	at org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$execute$1.apply(RuleExecutor.scala:76)
   	at scala.collection.immutable.List.foreach(List.scala:392)
   	at org.apache.spark.sql.catalyst.rules.RuleExecutor.execute(RuleExecutor.scala:76)
   	at org.apache.spark.sql.execution.QueryExecution.optimizedPlan$lzycompute(QueryExecution.scala:67)
   	at org.apache.spark.sql.execution.QueryExecution.optimizedPlan(QueryExecution.scala:67)
   	at org.apache.spark.sql.execution.QueryExecution.sparkPlan$lzycompute(QueryExecution.scala:73)
   	at org.apache.spark.sql.execution.QueryExecution.sparkPlan(QueryExecution.scala:69)
   	at org.apache.spark.sql.execution.QueryExecution.executedPlan$lzycompute(QueryExecution.scala:78)
   	at org.apache.spark.sql.execution.QueryExecution.executedPlan(QueryExecution.scala:78)
   	at org.apache.spark.sql.Dataset.withAction(Dataset.scala:3365)
   	at org.apache.spark.sql.Dataset.<init>(Dataset.scala:194)
   	at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:79)
   	at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:642)
   	at org.apache.spark.sql.SQLContext.sql(SQLContext.scala:694)
   Caused by: java.lang.ClassNotFoundException: submarine_spark_ranger_project.org.codehaus.jackson.jaxrs.JacksonJsonProvider
   	at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
   	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
   	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
   	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
   `
   
   I guess, we have two options (not sure if there could be some other solution):
   
   **a)** We could remove this line and package would not be shaded: https://github.com/apache/submarine/blob/master/submarine-security/spark-security/pom.xml#L389
   
   **b)** Apply this PR and make `org.codehaus.jackson:jackson-jaxrs` be included in the uber jar.
   
   
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org