You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "wangyum (via GitHub)" <gi...@apache.org> on 2023/08/11 03:05:16 UTC
[GitHub] [spark] wangyum opened a new pull request, #42446: [SPARK-44719][SQL] Fix NoClassDefFoundError when using Hive UDF
wangyum opened a new pull request, #42446:
URL: https://github.com/apache/spark/pull/42446
### What changes were proposed in this pull request?
This PR changes jackson-mapper-asl's scope from test to ${hive.deps.scope}.
### Why are the changes needed?
Fix `NoClassDefFoundError` when using Hive UDF:
```
spark-sql (default)> add jar /Users/yumwang/Downloads/HiveUDFs-1.0-SNAPSHOT.jar;
Time taken: 0.413 seconds
spark-sql (default)> CREATE TEMPORARY FUNCTION long_to_ip as 'net.petrabarus.hiveudfs.LongToIP';
Time taken: 0.038 seconds
spark-sql (default)> SELECT long_to_ip(2130706433L) FROM range(10);
23/08/08 20:17:58 ERROR SparkSQLDriver: Failed in [SELECT long_to_ip(2130706433L) FROM range(10)]
java.lang.NoClassDefFoundError: org/codehaus/jackson/map/type/TypeFactory
at org.apache.hadoop.hive.ql.udf.UDFJson.<clinit>(UDFJson.java:64)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:348)
...
```
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
manual test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] wangyum commented on a diff in pull request #42446: [SPARK-44719][SQL] Fix NoClassDefFoundError when using Hive UDF
Posted by "wangyum (via GitHub)" <gi...@apache.org>.
wangyum commented on code in PR #42446:
URL: https://github.com/apache/spark/pull/42446#discussion_r1290858738
##########
pom.xml:
##########
@@ -1838,7 +1838,7 @@
<groupId>org.codehaus.jackson</groupId>
<artifactId>jackson-mapper-asl</artifactId>
<version>${codehaus.jackson.version}</version>
- <scope>test</scope>
+ <scope>${hive.deps.scope}</scope>
Review Comment:
The master branch uses Hadoop 3.3.6. So Hadoop doesn't need this.
The branch-3.5 uses Hadoop 3.3.4. so Hadoop still need this.
https://issues.apache.org/jira/browse/HADOOP-13332
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] wangyum commented on a diff in pull request #42446: [SPARK-44719][SQL] Fix NoClassDefFoundError when using Hive UDF
Posted by "wangyum (via GitHub)" <gi...@apache.org>.
wangyum commented on code in PR #42446:
URL: https://github.com/apache/spark/pull/42446#discussion_r1290858738
##########
pom.xml:
##########
@@ -1838,7 +1838,7 @@
<groupId>org.codehaus.jackson</groupId>
<artifactId>jackson-mapper-asl</artifactId>
<version>${codehaus.jackson.version}</version>
- <scope>test</scope>
+ <scope>${hive.deps.scope}</scope>
Review Comment:
Hadoop 3.3.5+ do not need this: https://issues.apache.org/jira/browse/HADOOP-13332
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] yaooqinn closed pull request #42446: [SPARK-44719][SQL] Fix NoClassDefFoundError when using Hive UDF
Posted by "yaooqinn (via GitHub)" <gi...@apache.org>.
yaooqinn closed pull request #42446: [SPARK-44719][SQL] Fix NoClassDefFoundError when using Hive UDF
URL: https://github.com/apache/spark/pull/42446
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] yaooqinn commented on pull request #42446: [SPARK-44719][SQL] Fix NoClassDefFoundError when using Hive UDF
Posted by "yaooqinn (via GitHub)" <gi...@apache.org>.
yaooqinn commented on PR #42446:
URL: https://github.com/apache/spark/pull/42446#issuecomment-1675659953
thanks, merged to master
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-44719][SQL] Fix NoClassDefFoundError when using Hive UDF [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #42446:
URL: https://github.com/apache/spark/pull/42446#issuecomment-1957481706
In addition to `jackson-mapper-asl`, `jackson-core-asl` is also recovered. So, this is a revert of SPARK-43225 in terms of the dependency.
> This PR changes jackson-mapper-asl's scope from test to ${hive.deps.scope}.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org