You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@bigtop.apache.org by "Evans Ye (Jira)" <ji...@apache.org> on 2020/04/19 08:08:00 UTC
[jira] [Comment Edited] (BIGTOP-3333) Fix Hive build after
upgrading Spark to 2.4.5
[ https://issues.apache.org/jira/browse/BIGTOP-3333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17086843#comment-17086843 ]
Evans Ye edited comment on BIGTOP-3333 at 4/19/20, 8:07 AM:
------------------------------------------------------------
Thanks for digging into it [~sekikn]. I understand your concern, but I think we can look for other distribution/cloud vendors as a reference as well.
- EMR 5.X has Hive 2.3.6 with Spark 2.4.4.
This indicates that the combination can somewhat reach production grade.
Surely that they must have applied some internal fixes/patches to get them work. But I believe the work is not that much as the risk will also goes high.
May I know what's the technical issue currently blocking us form integrating Spark 2.4, or you just want to be careful?
was (Author: evans_ye):
Thanks for digging into it [~sekikn]. I understand your concern, but I think we can look for other distribution vendor/cloud vendor as a reference as well.
- EMR 5.X has Hive 2.3.6 with Spark 2.4.4.
This indicates that the combination can somewhat reach production grade.
Surely that they must have applied some internal fixes/patches to get them work. But I believe the work is not that much as the risk will also goes high.
May I know what's the technical issue currently blocking us form integrating Spark 2.4, or you just want to be careful?
> Fix Hive build after upgrading Spark to 2.4.5
> ---------------------------------------------
>
> Key: BIGTOP-3333
> URL: https://issues.apache.org/jira/browse/BIGTOP-3333
> Project: Bigtop
> Issue Type: Bug
> Components: build, hive
> Reporter: Kengo Seki
> Assignee: Kengo Seki
> Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> After BIGTOP-3165, building Hive came to fail as follows, due to the API change of Spark:
> {code}
> $ ./gradlew hive-pkg
> (snip)
> [INFO] -------------------------------------------------------------
> [ERROR] COMPILATION ERROR :
> [INFO] -------------------------------------------------------------
> [ERROR] /home/sekikn/repos/bigtop/output/hive/hive-2.3.6/ql/src/java/org/apache/hadoop/hive/ql/exec/spark/LocalHiveSparkClient.java:[87,26] cannot find symbol
> symbol: method addListener(org.apache.hadoop.hive.ql.exec.spark.status.impl.JobMetricsListener)
> location: class org.apache.spark.scheduler.LiveListenerBus
> [INFO] 1 error
> [INFO] -------------------------------------------------------------
> [INFO] ------------------------------------------------------------------------
> [INFO] Reactor Summary for Hive 2.3.6:
> [INFO]
> [INFO] Hive ............................................... SUCCESS [ 2.115 s]
> [INFO] Hive Shims Common .................................. SUCCESS [ 3.889 s]
> [INFO] Hive Shims 0.23 .................................... SUCCESS [ 2.686 s]
> [INFO] Hive Shims Scheduler ............................... SUCCESS [ 0.976 s]
> [INFO] Hive Shims ......................................... SUCCESS [ 0.793 s]
> [INFO] Hive Common ........................................ SUCCESS [ 5.762 s]
> [INFO] Hive Service RPC ................................... SUCCESS [ 2.480 s]
> [INFO] Hive Serde ......................................... SUCCESS [ 4.136 s]
> [INFO] Hive Metastore ..................................... SUCCESS [ 16.040 s]
> [INFO] Hive Vector-Code-Gen Utilities ..................... SUCCESS [ 0.306 s]
> [INFO] Hive Llap Common ................................... SUCCESS [ 3.953 s]
> [INFO] Hive Llap Client ................................... SUCCESS [ 2.334 s]
> [INFO] Hive Llap Tez ...................................... SUCCESS [ 2.591 s]
> [INFO] Spark Remote Client ................................ SUCCESS [ 3.028 s]
> [INFO] Hive Query Language ................................ FAILURE [ 17.783 s]
> [INFO] Hive Llap Server ................................... SKIPPED
> [INFO] Hive Service ....................................... SKIPPED
> [INFO] Hive Accumulo Handler .............................. SKIPPED
> [INFO] Hive JDBC .......................................... SKIPPED
> [INFO] Hive Beeline ....................................... SKIPPED
> [INFO] Hive CLI ........................................... SKIPPED
> [INFO] Hive Contrib ....................................... SKIPPED
> [INFO] Hive Druid Handler ................................. SKIPPED
> [INFO] Hive HBase Handler ................................. SKIPPED
> [INFO] Hive JDBC Handler .................................. SKIPPED
> [INFO] Hive HCatalog ...................................... SKIPPED
> [INFO] Hive HCatalog Core ................................. SKIPPED
> [INFO] Hive HCatalog Pig Adapter .......................... SKIPPED
> [INFO] Hive HCatalog Server Extensions .................... SKIPPED
> [INFO] Hive HCatalog Webhcat Java Client .................. SKIPPED
> [INFO] Hive HCatalog Webhcat .............................. SKIPPED
> [INFO] Hive HCatalog Streaming ............................ SKIPPED
> [INFO] Hive HPL/SQL ....................................... SKIPPED
> [INFO] Hive Llap External Client .......................... SKIPPED
> [INFO] Hive Shims Aggregator .............................. SKIPPED
> [INFO] Hive TestUtils ..................................... SKIPPED
> [INFO] Hive Packaging ..................................... SKIPPED
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD FAILURE
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 01:09 min
> [INFO] Finished at: 2020-04-03T10:55:51+09:00
> [INFO] ------------------------------------------------------------------------
> [ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.1:compile (default-compile) on project hive-exec: Compilation failure
> [ERROR] /home/sekikn/repos/bigtop/output/hive/hive-2.3.6/ql/src/java/org/apache/hadoop/hive/ql/exec/spark/LocalHiveSparkClient.java:[87,26] cannot find symbol
> [ERROR] symbol: method addListener(org.apache.hadoop.hive.ql.exec.spark.status.impl.JobMetricsListener)
> [ERROR] location: class org.apache.spark.scheduler.LiveListenerBus
> [ERROR]
> [ERROR] -> [Help 1]
> [ERROR]
> [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> [ERROR]
> [ERROR] For more information about the errors and possible solutions, please read the following articles:
> [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
> [ERROR]
> [ERROR] After correcting the problems, you can resume the build with the command
> [ERROR] mvn <args> -rf :hive-exec
> debian/rules:32: recipe for target 'override_dh_auto_build' failed
> make[1]: *** [override_dh_auto_build] Error 1
> make[1]: Leaving directory '/home/sekikn/repos/bigtop/output/hive/hive-2.3.6'
> debian/rules:29: recipe for target 'build' failed
> make: *** [build] Error 2
> dpkg-buildpackage: error: debian/rules build subprocess returned exit status 2
> debuild: fatal error at line 1152:
> dpkg-buildpackage -rfakeroot -us -uc -ui -b
> failed
> > Task :hive-deb FAILED
> FAILURE: Build failed with an exception.
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)