You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by "Mykhailo Kysliuk (JIRA)" <ji...@apache.org> on 2018/06/08 19:25:00 UTC
[jira] [Created] (HIVE-19836) Import to hive as parquet data format
failed Could not initialize class org.apache.derby.jdbc.AutoloadedDriver40
Mykhailo Kysliuk created HIVE-19836:
---------------------------------------
Summary: Import to hive as parquet data format failed Could not initialize class org.apache.derby.jdbc.AutoloadedDriver40
Key: HIVE-19836
URL: https://issues.apache.org/jira/browse/HIVE-19836
Project: Hive
Issue Type: Bug
Affects Versions: 2.3.3
Reporter: Mykhailo Kysliuk
Components:
hive-2.3.3
hadoop-2.7.0
sqoop-1.4.7
I am trying to use sqoop import command (first time command could succeed, but if you run second time it fails):
{code:java}
sqoop import-all-tables --connect jdbc:mysql://127.0.0.1:3306/db1 --username root --password 123456 --as-parquetfile --hive-import --hive-database test --hive-overwrite -m 1
{code}
But I am receiving error:
{code:java}
18/06/08 22:03:40 ERROR tool.ImportAllTablesTool: Error during import: Import job failed!
{code}
Logs:
{code:java}
vim /opt/hadoop/logs/userlogs/application_1528483696847_0003/container_1528483696847_0003_01_000004/syslog
{code}
{code:java}
2018-06-08 22:03:33,357 FATAL [main] org.apache.hadoop.mapred.YarnChild: Error running child : java.lang.NoClassDefFoundError: Could not initialize class org.apache.derby.jdbc.AutoloadedDriver40
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:348)
at java.sql.DriverManager.isDriverAllowed(DriverManager.java:556)
at java.sql.DriverManager.getConnection(DriverManager.java:661)
at java.sql.DriverManager.getConnection(DriverManager.java:247)
at org.apache.sqoop.mapreduce.db.DBConfiguration.getConnection(DBConfiguration.java:302)
at org.apache.sqoop.mapreduce.db.DBInputFormat.getConnection(DBInputFormat.java:216)
at org.apache.sqoop.mapreduce.db.DBInputFormat.setDbConf(DBInputFormat.java:168)
at org.apache.sqoop.mapreduce.db.DBInputFormat.setConf(DBInputFormat.java:161)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:76)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:136)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:749)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
{code}
My HADOOP_CLASSPATH:
{code}
[hadoop@hadoop bin]$ echo $HADOOP_CLASSPATH
.:/opt/jdk1.8.0_171/jre/lib:/opt/jdk1.8.0_171/lib:/opt/jdk1.8.0_171/lib/tools.jar:/opt/hadoop/lib/*:/opt/hive/lib/*
{code}
I noticed that there are ambiguous versions of derby jars at hive libs:
{code}
[hadoop@hadoop lib]$ ll lib | grep derby
-rw-r--r--. 1 hadoop root 2838580 січ 16 2016 derby-10.10.2.0.jar
-rw-r--r--. 1 hadoop root 583719 гру 19 2016 derbyclient-10.11.1.1.jar
-rw-r--r--. 1 hadoop root 266316 січ 17 2016 derbynet-10.11.1.1.jar
{code}
The error disappears if I remove the derbyclient-10.11.1.1.jar and derbynet-10.11.1.1.jar jars.
There are pretty much jars that have different versions for the same artifact:
{code}
-rw-r--r--. 1 hadoop root 1034049 січ 16 2016 ant-1.6.5.jar
-rw-r--r--. 1 hadoop root 1997485 січ 16 2016 ant-1.9.1.jar
-rw-r--r--. 1 hadoop root 539912 січ 16 2016 jetty-6.1.26.jar
-rw-r--r--. 1 hadoop root 1681148 січ 16 2016 jetty-all-7.6.0.v20120127.jar
-rw-r--r--. 1 hadoop root 228092 гру 19 2016 jetty-client-9.2.5.v20141112.jar
-rw-r--r--. 1 hadoop root 15989 гру 19 2016 jetty-continuation-9.2.5.v20141112.jar
-rw-r--r--. 1 hadoop root 105258 гру 19 2016 jetty-http-9.2.5.v20141112.jar
-rw-r--r--. 1 hadoop root 105419 гру 19 2016 jetty-io-9.2.5.v20141112.jar
-rw-r--r--. 1 hadoop root 50644 гру 19 2016 jetty-proxy-9.2.5.v20141112.jar
-rw-r--r--. 1 hadoop root 95924 гру 19 2016 jetty-security-9.2.5.v20141112.jar
-rw-r--r--. 1 hadoop root 417278 гру 19 2016 jetty-server-9.2.5.v20141112.jar
-rw-r--r--. 1 hadoop root 115473 гру 19 2016 jetty-servlet-9.2.5.v20141112.jar
-rw-r--r--. 1 hadoop root 121692 гру 19 2016 jetty-servlets-9.2.5.v20141112.jar
-rw-r--r--. 1 hadoop root 18891 січ 16 2016 jetty-sslengine-6.1.26.jar
-rw-r--r--. 1 hadoop root 177131 січ 16 2016 jetty-util-6.1.26.jar
-rw-r--r--. 1 hadoop root 347382 гру 19 2016 jetty-util-9.2.5.v20141112.jar
-rw-r--r--. 1 hadoop root 1024680 січ 16 2016 jsp-2.1-6.1.14.jar
-rw-r--r--. 1 hadoop root 50493 січ 16 2016 jsp-api-2.0.jar
-rw-r--r--. 1 hadoop root 134910 січ 16 2016 jsp-api-2.1-6.1.14.jar
-rw-r--r--. 1 hadoop root 100636 січ 16 2016 jsp-api-2.1.jar
-rw-r--r--. 1 hadoop root 82123 січ 16 2016 metrics-core-2.2.0.jar
-rw-r--r--. 1 hadoop root 111908 січ 16 2016 metrics-core-3.1.0.jar
-rw-r--r--. 1 hadoop root 1199572 січ 16 2016 netty-3.6.2.Final.jar
-rw-r--r--. 1 hadoop root 2275047 жов 3 2017 netty-all-4.0.52.Final.jar
-rw-r--r--. 1 hadoop root 97693 січ 16 2016 servlet-api-2.4.jar
-rw-r--r--. 1 hadoop root 132368 січ 16 2016 servlet-api-2.5-6.1.14.jar
{code}
Could this issues with maven ambiguous dependencies at hive be a problem? I think not only derby can be a problem.
Do we need to remove it? But there will be too much places to exclude jetty lib at poms, for example.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)