You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "wangyum (via GitHub)" <gi...@apache.org> on 2023/08/11 03:37:55 UTC
[GitHub] [spark] wangyum opened a new pull request, #42447: [SPARK-44719][SQL][3.5] Fix NoClassDefFoundError when using Hive UDF
wangyum opened a new pull request, #42447:
URL: https://github.com/apache/spark/pull/42447
### What changes were proposed in this pull request?
This PR partially reverts SPARK-43225. Added jackson-core-asl and jackson-mapper-asl back to pre-built distributions.
### Why are the changes needed?
Fix `NoClassDefFoundError` when using Hive UDF:
```
spark-sql (default)> add jar /Users/yumwang/Downloads/HiveUDFs-1.0-SNAPSHOT.jar;
Time taken: 0.413 seconds
spark-sql (default)> CREATE TEMPORARY FUNCTION long_to_ip as 'net.petrabarus.hiveudfs.LongToIP';
Time taken: 0.038 seconds
spark-sql (default)> SELECT long_to_ip(2130706433L) FROM range(10);
23/08/08 20:17:58 ERROR SparkSQLDriver: Failed in [SELECT long_to_ip(2130706433L) FROM range(10)]
java.lang.NoClassDefFoundError: org/codehaus/jackson/map/type/TypeFactory
at org.apache.hadoop.hive.ql.udf.UDFJson.<clinit>(UDFJson.java:64)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:348)
...
```
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
manual test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] yaooqinn closed pull request #42447: [SPARK-44719][SQL][3.5] Fix NoClassDefFoundError when using Hive UDF
Posted by "yaooqinn (via GitHub)" <gi...@apache.org>.
yaooqinn closed pull request #42447: [SPARK-44719][SQL][3.5] Fix NoClassDefFoundError when using Hive UDF
URL: https://github.com/apache/spark/pull/42447
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] wangyum commented on a diff in pull request #42447: [SPARK-44719][SQL][3.5] Fix NoClassDefFoundError when using Hive UDF
Posted by "wangyum (via GitHub)" <gi...@apache.org>.
wangyum commented on code in PR #42447:
URL: https://github.com/apache/spark/pull/42447#discussion_r1290863628
##########
pom.xml:
##########
@@ -1821,12 +1825,17 @@
</exclusion>
</exclusions>
</dependency>
- <!-- Hive 2.3 need this to init Hive's FunctionRegistry -->
+ <dependency>
+ <groupId>org.codehaus.jackson</groupId>
+ <artifactId>jackson-core-asl</artifactId>
+ <version>${codehaus.jackson.version}</version>
+ <scope>${hadoop.deps.scope}</scope>
Review Comment:
Hadoop 3.3.4 still need this: https://issues.apache.org/jira/browse/HADOOP-13332
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a diff in pull request #42447: [SPARK-44719][SQL][3.5] Fix NoClassDefFoundError when using Hive UDF
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #42447:
URL: https://github.com/apache/spark/pull/42447#discussion_r1291578404
##########
core/pom.xml:
##########
@@ -481,6 +481,14 @@
<groupId>commons-logging</groupId>
<artifactId>commons-logging</artifactId>
</exclusion>
+ <exclusion>
+ <groupId>org.codehaus.jackson</groupId>
+ <artifactId>jackson-mapper-asl</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>org.codehaus.jackson</groupId>
+ <artifactId>jackson-core-asl</artifactId>
+ </exclusion>
Review Comment:
Thank you for the reply, @wangyum .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] wangyum commented on a diff in pull request #42447: [SPARK-44719][SQL][3.5] Fix NoClassDefFoundError when using Hive UDF
Posted by "wangyum (via GitHub)" <gi...@apache.org>.
wangyum commented on code in PR #42447:
URL: https://github.com/apache/spark/pull/42447#discussion_r1290863628
##########
pom.xml:
##########
@@ -1821,12 +1825,17 @@
</exclusion>
</exclusions>
</dependency>
- <!-- Hive 2.3 need this to init Hive's FunctionRegistry -->
+ <dependency>
+ <groupId>org.codehaus.jackson</groupId>
+ <artifactId>jackson-core-asl</artifactId>
+ <version>${codehaus.jackson.version}</version>
+ <scope>${hadoop.deps.scope}</scope>
Review Comment:
The master branch uses Hadoop 3.3.6. So Hadoop doesn't need this.
The branch-3.5 uses Hadoop 3.3.4. so Hadoop still need this.
https://issues.apache.org/jira/browse/HADOOP-13332
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] wangyum commented on pull request #42447: [SPARK-44719][SQL][3.5] Fix NoClassDefFoundError when using Hive UDF
Posted by "wangyum (via GitHub)" <gi...@apache.org>.
wangyum commented on PR #42447:
URL: https://github.com/apache/spark/pull/42447#issuecomment-1676199420
> Do you think we can have a test coverage, @wangyum ?
The test cannot cover it because jackson-mapper-asl is in the classpath: https://github.com/apache/spark/blob/branch-3.5/pom.xml#L1825-L1829
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a diff in pull request #42447: [SPARK-44719][SQL][3.5] Fix NoClassDefFoundError when using Hive UDF
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #42447:
URL: https://github.com/apache/spark/pull/42447#discussion_r1290892578
##########
core/pom.xml:
##########
@@ -481,6 +481,14 @@
<groupId>commons-logging</groupId>
<artifactId>commons-logging</artifactId>
</exclusion>
+ <exclusion>
+ <groupId>org.codehaus.jackson</groupId>
+ <artifactId>jackson-mapper-asl</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>org.codehaus.jackson</groupId>
+ <artifactId>jackson-core-asl</artifactId>
+ </exclusion>
Review Comment:
May I ask why this is added to `core` module instead of `sql/hive` module?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] wangyum commented on a diff in pull request #42447: [SPARK-44719][SQL][3.5] Fix NoClassDefFoundError when using Hive UDF
Posted by "wangyum (via GitHub)" <gi...@apache.org>.
wangyum commented on code in PR #42447:
URL: https://github.com/apache/spark/pull/42447#discussion_r1290895304
##########
core/pom.xml:
##########
@@ -481,6 +481,14 @@
<groupId>commons-logging</groupId>
<artifactId>commons-logging</artifactId>
</exclusion>
+ <exclusion>
+ <groupId>org.codehaus.jackson</groupId>
+ <artifactId>jackson-mapper-asl</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>org.codehaus.jackson</groupId>
+ <artifactId>jackson-core-asl</artifactId>
+ </exclusion>
Review Comment:
In order to keep consistent with branch-3.4.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] yaooqinn commented on pull request #42447: [SPARK-44719][SQL][3.5] Fix NoClassDefFoundError when using Hive UDF
Posted by "yaooqinn (via GitHub)" <gi...@apache.org>.
yaooqinn commented on PR #42447:
URL: https://github.com/apache/spark/pull/42447#issuecomment-1678386580
thanks, merged to master
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #42447: [SPARK-44719][SQL][3.5] Fix NoClassDefFoundError when using Hive UDF
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #42447:
URL: https://github.com/apache/spark/pull/42447#issuecomment-1679746012
Got it. Thank you, @wangyum and @yaooqinn .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] yaooqinn commented on pull request #42447: [SPARK-44719][SQL][3.5] Fix NoClassDefFoundError when using Hive UDF
Posted by "yaooqinn (via GitHub)" <gi...@apache.org>.
yaooqinn commented on PR #42447:
URL: https://github.com/apache/spark/pull/42447#issuecomment-1678387753
the GitHub micro 'close' seems not working, close it manually
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org