You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@sqoop.apache.org by Lee S <sl...@gmail.com> on 2015/08/14 10:10:59 UTC
execute sqoop job with exception
Hi all:
I try to import from rdms to hive with kite connector in sqoop-shell.
And I submit the job successfully , but when I try to track the job staus
on yarn web-ui, I found there are errors in container log as below.
And the job continues running ,never stops.
The log shows *java.lang.NoClassDefFoundError:
org/apache/hadoop/hive/ql/io/HiveOutputFormat.*
*Anybody can help?*
*Showing 4096 bytes. Click here
<http://pdm-03:8042/node/containerlogs/container_1439241066552_0044_01_000002/root/stderr/?start=0>
for
full log*
*utFormatLoadExecutor - SqoopOutputFormatLoadExecutor consumer thread
is starting
2015-08-14 15:39:10,765 [OutputFormatLoader-consumer] INFO
org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - Running
loader class org.apache.sqoop.connector.kite.KiteLoader
2015-08-14 15:39:10,771 [main] INFO
org.apache.sqoop.job.mr.SqoopMapper - Starting progress service
2015-08-14 15:39:10,772 [main] INFO
org.apache.sqoop.job.mr.SqoopMapper - Running extractor class
org.apache.sqoop.connector.jdbc.GenericJdbcExtractor
2015-08-14 15:39:10,981 [OutputFormatLoader-consumer] INFO
org.apache.sqoop.connector.kite.KiteLoader - Constructed temporary
dataset URI: dataset:hive:wangjun/temp_5bfec97713e04374b2f2efde2dc5e4de?auth:host=pdm-03&auth:port=9083
2015-08-14 15:39:11,093 [main] INFO
org.apache.sqoop.connector.jdbc.GenericJdbcExtractor - Using query:
SELECT id FROM bcpdm.history WHERE 1368 <= id AND id <= 1546
2015-08-14 15:39:11,537 [OutputFormatLoader-consumer] ERROR
org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - Error while
loading data out of MR job.
java.lang.NoClassDefFoundError: org/apache/hadoop/hive/ql/io/HiveOutputFormat
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:800)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:190)
at org.kitesdk.data.spi.hive.HiveUtils.getHiveParquetOutputFormat(HiveUtils.java:446)
at org.kitesdk.data.spi.hive.HiveUtils.<clinit>(HiveUtils.java:91)
at org.kitesdk.data.spi.hive.HiveManagedMetadataProvider.create(HiveManagedMetadataProvider.java:83)
at org.kitesdk.data.spi.hive.HiveManagedDatasetRepository.create(HiveManagedDatasetRepository.java:77)
at org.kitesdk.data.Datasets.create(Datasets.java:239)
at org.kitesdk.data.Datasets.create(Datasets.java:307)
at org.kitesdk.data.Datasets.create(Datasets.java:335)
at org.apache.sqoop.connector.kite.KiteDatasetExecutor.createDataset(KiteDatasetExecutor.java:70)
at org.apache.sqoop.connector.kite.KiteLoader.getExecutor(KiteLoader.java:52)
at org.apache.sqoop.connector.kite.KiteLoader.load(KiteLoader.java:62)
at org.apache.sqoop.connector.kite.KiteLoader.load(KiteLoader.java:36)
at org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor$ConsumerThread.run(SqoopOutputFormatLoadExecutor.java:250)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.ClassNotFoundException:
org.apache.hadoop.hive.ql.io.HiveOutputFormat
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
... 31 more
2015-08-14 15:39:11,540 [main] INFO
org.apache.sqoop.job.mr.SqoopMapper - Stopping progress service
2015-08-14 15:39:11,540 [main] INFO
org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor -
SqoopOutputFormatLoadExecutor::SqoopRecordWriter is about to be
closed*
p.s. sqoop version: 1.99.6
hadoop version 2.6.0
Re: execute sqoop job with exception
Posted by Lee S <sl...@gmail.com>.
p.s. I changed the format to FileFormat.AVRO , the job runs well.
On Mon, Aug 17, 2015 at 4:03 PM, Lee S <sl...@gmail.com> wrote:
> @Abe,I've solved the problem by adding
>
>
> jars.add(ClassUtils.jarForClass("org.apache.hadoop.hive.ql.io.HiveOutputFormat"));
> in the KiteToInitializer.java.
>
> But another prolem comes out. Exception as below:
>
> 2015-08-17 15:10:28,757 [OutputFormatLoader-consumer] ERROR org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - Error while loading data out of MR job.
> org.kitesdk.data.UnknownFormatException: No known serde for format:csv
> at org.kitesdk.data.spi.hive.HiveUtils.tableForDescriptor(HiveUtils.java:232)
> at org.kitesdk.data.spi.hive.HiveManagedMetadataProvider.create(HiveManagedMetadataProvider.java:83)
> at org.kitesdk.data.spi.hive.HiveManagedDatasetRepository.create(HiveManagedDatasetRepository.java:77)
> at org.kitesdk.data.Datasets.create(Datasets.java:239)
> at org.kitesdk.data.Datasets.create(Datasets.java:307)
> at org.kitesdk.data.Datasets.create(Datasets.java:335)
> at org.apache.sqoop.connector.kite.KiteDatasetExecutor.createDataset(KiteDatasetExecutor.java:70)
> at org.apache.sqoop.connector.kite.KiteLoader.getExecutor(KiteLoader.java:52)
> at org.apache.sqoop.connector.kite.KiteLoader.load(KiteLoader.java:62)
> at org.apache.sqoop.connector.kite.KiteLoader.load(KiteLoader.java:36)
> at org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor$ConsumerThread.run(SqoopOutputFormatLoadExecutor.java:250)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
>
>
> I read the related codes of kitesdk, in Kite, only two file format Avro
> /parquet is supported,
> And FileFormat.CSV is not ok.
>
> What can I do if I want to import into hive with text format by Kite
> Connector ?
> Any idea?
>
> On Mon, Aug 17, 2015 at 10:40 AM, Lee S <sl...@gmail.com> wrote:
>
>> @Abe, Hive dependeces have been added to the comon.loader at
>> conf/catalina.properties.
>> And this line
>> jars.add(ClassUtils.jarForClass("org.apache.hadoop.hive.conf.HiveConf"));
>> can find the hive jar which also contains the not founded
>> class(hive.ql.outputformat).
>>
>>
>> On Mon, Aug 17, 2015 at 12:30 AM, Abraham Elmahrek <ab...@cloudera.com>
>> wrote:
>>
>>> Sqoop2 delegates Parquet support to Kite. Each file format might be a
>>> different code path in Kite.
>>>
>>> On Fri, Aug 14, 2015 at 6:52 PM, Lee S <sl...@gmail.com> wrote:
>>>
>>>> Hi Abe:
>>>> I'll try to add hive deps in the codes and rebuild sqoop2.
>>>> But I have set the file format to FileFormat.CSV, why it comes about
>>>> the parquet.
>>>>
>>>> On Sat, Aug 15, 2015 at 1:00 AM, Abraham Elmahrek <ab...@cloudera.com>
>>>> wrote:
>>>>
>>>>> Hey man,
>>>>>
>>>>> It looks like certain Hive jars are missing from the job for some
>>>>> reason. Seems like we need to add more jars to
>>>>> https://github.com/apache/sqoop/blob/sqoop2/connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteFromInitializer.java#L71
>>>>> and
>>>>> https://github.com/apache/sqoop/blob/sqoop2/connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteToInitializer.java#L78
>>>>> .
>>>>>
>>>>> I've created https://issues.apache.org/jira/browse/SQOOP-2489 to
>>>>> track this bug. I've also created
>>>>> https://issues.apache.org/jira/browse/SQOOP-2490 to provide a
>>>>> facility to work around these kinds of issues in the future.
>>>>>
>>>>> Sqoop2 is a work in progress and still needs some battle testing. With
>>>>> that in mind, can you use the Avro integration instead? Otherwise, you
>>>>> might need to rebuild Sqoop2 with the fix to get this working at the
>>>>> moment. Again, https://issues.apache.org/jira/browse/SQOOP-2490 will
>>>>> change all of that.
>>>>>
>>>>> -Abe
>>>>>
>>>>> On Fri, Aug 14, 2015 at 1:10 AM, Lee S <sl...@gmail.com> wrote:
>>>>>
>>>>>> Hi all:
>>>>>> I try to import from rdms to hive with kite connector in sqoop-shell.
>>>>>> And I submit the job successfully , but when I try to track the job
>>>>>> staus
>>>>>> on yarn web-ui, I found there are errors in container log as below.
>>>>>> And the job continues running ,never stops.
>>>>>> The log shows *java.lang.NoClassDefFoundError:
>>>>>> org/apache/hadoop/hive/ql/io/HiveOutputFormat.*
>>>>>>
>>>>>> *Anybody can help?*
>>>>>>
>>>>>> *Showing 4096 bytes. Click here
>>>>>> <http://pdm-03:8042/node/containerlogs/container_1439241066552_0044_01_000002/root/stderr/?start=0> for
>>>>>> full log*
>>>>>>
>>>>>> *utFormatLoadExecutor - SqoopOutputFormatLoadExecutor consumer thread is starting
>>>>>> 2015-08-14 15:39:10,765 [OutputFormatLoader-consumer] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - Running loader class org.apache.sqoop.connector.kite.KiteLoader
>>>>>> 2015-08-14 15:39:10,771 [main] INFO org.apache.sqoop.job.mr.SqoopMapper - Starting progress service
>>>>>> 2015-08-14 15:39:10,772 [main] INFO org.apache.sqoop.job.mr.SqoopMapper - Running extractor class org.apache.sqoop.connector.jdbc.GenericJdbcExtractor
>>>>>> 2015-08-14 15:39:10,981 [OutputFormatLoader-consumer] INFO org.apache.sqoop.connector.kite.KiteLoader - Constructed temporary dataset URI: dataset:hive:wangjun/temp_5bfec97713e04374b2f2efde2dc5e4de?auth:host=pdm-03&auth:port=9083
>>>>>> 2015-08-14 15:39:11,093 [main] INFO org.apache.sqoop.connector.jdbc.GenericJdbcExtractor - Using query: SELECT id FROM bcpdm.history WHERE 1368 <= id AND id <= 1546
>>>>>> 2015-08-14 15:39:11,537 [OutputFormatLoader-consumer] ERROR org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - Error while loading data out of MR job.
>>>>>> java.lang.NoClassDefFoundError: org/apache/hadoop/hive/ql/io/HiveOutputFormat
>>>>>> at java.lang.ClassLoader.defineClass1(Native Method)
>>>>>> at java.lang.ClassLoader.defineClass(ClassLoader.java:800)
>>>>>> at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
>>>>>> at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
>>>>>> at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
>>>>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
>>>>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
>>>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
>>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
>>>>>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
>>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
>>>>>> at java.lang.Class.forName0(Native Method)
>>>>>> at java.lang.Class.forName(Class.java:190)
>>>>>> at org.kitesdk.data.spi.hive.HiveUtils.getHiveParquetOutputFormat(HiveUtils.java:446)
>>>>>> at org.kitesdk.data.spi.hive.HiveUtils.<clinit>(HiveUtils.java:91)
>>>>>> at org.kitesdk.data.spi.hive.HiveManagedMetadataProvider.create(HiveManagedMetadataProvider.java:83)
>>>>>> at org.kitesdk.data.spi.hive.HiveManagedDatasetRepository.create(HiveManagedDatasetRepository.java:77)
>>>>>> at org.kitesdk.data.Datasets.create(Datasets.java:239)
>>>>>> at org.kitesdk.data.Datasets.create(Datasets.java:307)
>>>>>> at org.kitesdk.data.Datasets.create(Datasets.java:335)
>>>>>> at org.apache.sqoop.connector.kite.KiteDatasetExecutor.createDataset(KiteDatasetExecutor.java:70)
>>>>>> at org.apache.sqoop.connector.kite.KiteLoader.getExecutor(KiteLoader.java:52)
>>>>>> at org.apache.sqoop.connector.kite.KiteLoader.load(KiteLoader.java:62)
>>>>>> at org.apache.sqoop.connector.kite.KiteLoader.load(KiteLoader.java:36)
>>>>>> at org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor$ConsumerThread.run(SqoopOutputFormatLoadExecutor.java:250)
>>>>>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>>>>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>> Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hive.ql.io.HiveOutputFormat
>>>>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
>>>>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
>>>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
>>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
>>>>>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
>>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
>>>>>> ... 31 more
>>>>>> 2015-08-14 15:39:11,540 [main] INFO org.apache.sqoop.job.mr.SqoopMapper - Stopping progress service
>>>>>> 2015-08-14 15:39:11,540 [main] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - SqoopOutputFormatLoadExecutor::SqoopRecordWriter is about to be closed*
>>>>>>
>>>>>>
>>>>>> p.s. sqoop version: 1.99.6
>>>>>> hadoop version 2.6.0
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>
Re: execute sqoop job with exception
Posted by Lee S <sl...@gmail.com>.
@Abe,I've solved the problem by adding
jars.add(ClassUtils.jarForClass("org.apache.hadoop.hive.ql.io.HiveOutputFormat"));
in the KiteToInitializer.java.
But another prolem comes out. Exception as below:
2015-08-17 15:10:28,757 [OutputFormatLoader-consumer] ERROR
org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - Error while
loading data out of MR job.
org.kitesdk.data.UnknownFormatException: No known serde for format:csv
at org.kitesdk.data.spi.hive.HiveUtils.tableForDescriptor(HiveUtils.java:232)
at org.kitesdk.data.spi.hive.HiveManagedMetadataProvider.create(HiveManagedMetadataProvider.java:83)
at org.kitesdk.data.spi.hive.HiveManagedDatasetRepository.create(HiveManagedDatasetRepository.java:77)
at org.kitesdk.data.Datasets.create(Datasets.java:239)
at org.kitesdk.data.Datasets.create(Datasets.java:307)
at org.kitesdk.data.Datasets.create(Datasets.java:335)
at org.apache.sqoop.connector.kite.KiteDatasetExecutor.createDataset(KiteDatasetExecutor.java:70)
at org.apache.sqoop.connector.kite.KiteLoader.getExecutor(KiteLoader.java:52)
at org.apache.sqoop.connector.kite.KiteLoader.load(KiteLoader.java:62)
at org.apache.sqoop.connector.kite.KiteLoader.load(KiteLoader.java:36)
at org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor$ConsumerThread.run(SqoopOutputFormatLoadExecutor.java:250)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
I read the related codes of kitesdk, in Kite, only two file format Avro
/parquet is supported,
And FileFormat.CSV is not ok.
What can I do if I want to import into hive with text format by Kite
Connector ?
Any idea?
On Mon, Aug 17, 2015 at 10:40 AM, Lee S <sl...@gmail.com> wrote:
> @Abe, Hive dependeces have been added to the comon.loader at
> conf/catalina.properties.
> And this line
> jars.add(ClassUtils.jarForClass("org.apache.hadoop.hive.conf.HiveConf"));
> can find the hive jar which also contains the not founded
> class(hive.ql.outputformat).
>
>
> On Mon, Aug 17, 2015 at 12:30 AM, Abraham Elmahrek <ab...@cloudera.com>
> wrote:
>
>> Sqoop2 delegates Parquet support to Kite. Each file format might be a
>> different code path in Kite.
>>
>> On Fri, Aug 14, 2015 at 6:52 PM, Lee S <sl...@gmail.com> wrote:
>>
>>> Hi Abe:
>>> I'll try to add hive deps in the codes and rebuild sqoop2.
>>> But I have set the file format to FileFormat.CSV, why it comes about
>>> the parquet.
>>>
>>> On Sat, Aug 15, 2015 at 1:00 AM, Abraham Elmahrek <ab...@cloudera.com>
>>> wrote:
>>>
>>>> Hey man,
>>>>
>>>> It looks like certain Hive jars are missing from the job for some
>>>> reason. Seems like we need to add more jars to
>>>> https://github.com/apache/sqoop/blob/sqoop2/connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteFromInitializer.java#L71
>>>> and
>>>> https://github.com/apache/sqoop/blob/sqoop2/connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteToInitializer.java#L78
>>>> .
>>>>
>>>> I've created https://issues.apache.org/jira/browse/SQOOP-2489 to track
>>>> this bug. I've also created
>>>> https://issues.apache.org/jira/browse/SQOOP-2490 to provide a facility
>>>> to work around these kinds of issues in the future.
>>>>
>>>> Sqoop2 is a work in progress and still needs some battle testing. With
>>>> that in mind, can you use the Avro integration instead? Otherwise, you
>>>> might need to rebuild Sqoop2 with the fix to get this working at the
>>>> moment. Again, https://issues.apache.org/jira/browse/SQOOP-2490 will
>>>> change all of that.
>>>>
>>>> -Abe
>>>>
>>>> On Fri, Aug 14, 2015 at 1:10 AM, Lee S <sl...@gmail.com> wrote:
>>>>
>>>>> Hi all:
>>>>> I try to import from rdms to hive with kite connector in sqoop-shell.
>>>>> And I submit the job successfully , but when I try to track the job
>>>>> staus
>>>>> on yarn web-ui, I found there are errors in container log as below.
>>>>> And the job continues running ,never stops.
>>>>> The log shows *java.lang.NoClassDefFoundError:
>>>>> org/apache/hadoop/hive/ql/io/HiveOutputFormat.*
>>>>>
>>>>> *Anybody can help?*
>>>>>
>>>>> *Showing 4096 bytes. Click here
>>>>> <http://pdm-03:8042/node/containerlogs/container_1439241066552_0044_01_000002/root/stderr/?start=0> for
>>>>> full log*
>>>>>
>>>>> *utFormatLoadExecutor - SqoopOutputFormatLoadExecutor consumer thread is starting
>>>>> 2015-08-14 15:39:10,765 [OutputFormatLoader-consumer] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - Running loader class org.apache.sqoop.connector.kite.KiteLoader
>>>>> 2015-08-14 15:39:10,771 [main] INFO org.apache.sqoop.job.mr.SqoopMapper - Starting progress service
>>>>> 2015-08-14 15:39:10,772 [main] INFO org.apache.sqoop.job.mr.SqoopMapper - Running extractor class org.apache.sqoop.connector.jdbc.GenericJdbcExtractor
>>>>> 2015-08-14 15:39:10,981 [OutputFormatLoader-consumer] INFO org.apache.sqoop.connector.kite.KiteLoader - Constructed temporary dataset URI: dataset:hive:wangjun/temp_5bfec97713e04374b2f2efde2dc5e4de?auth:host=pdm-03&auth:port=9083
>>>>> 2015-08-14 15:39:11,093 [main] INFO org.apache.sqoop.connector.jdbc.GenericJdbcExtractor - Using query: SELECT id FROM bcpdm.history WHERE 1368 <= id AND id <= 1546
>>>>> 2015-08-14 15:39:11,537 [OutputFormatLoader-consumer] ERROR org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - Error while loading data out of MR job.
>>>>> java.lang.NoClassDefFoundError: org/apache/hadoop/hive/ql/io/HiveOutputFormat
>>>>> at java.lang.ClassLoader.defineClass1(Native Method)
>>>>> at java.lang.ClassLoader.defineClass(ClassLoader.java:800)
>>>>> at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
>>>>> at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
>>>>> at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
>>>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
>>>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
>>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
>>>>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
>>>>> at java.lang.Class.forName0(Native Method)
>>>>> at java.lang.Class.forName(Class.java:190)
>>>>> at org.kitesdk.data.spi.hive.HiveUtils.getHiveParquetOutputFormat(HiveUtils.java:446)
>>>>> at org.kitesdk.data.spi.hive.HiveUtils.<clinit>(HiveUtils.java:91)
>>>>> at org.kitesdk.data.spi.hive.HiveManagedMetadataProvider.create(HiveManagedMetadataProvider.java:83)
>>>>> at org.kitesdk.data.spi.hive.HiveManagedDatasetRepository.create(HiveManagedDatasetRepository.java:77)
>>>>> at org.kitesdk.data.Datasets.create(Datasets.java:239)
>>>>> at org.kitesdk.data.Datasets.create(Datasets.java:307)
>>>>> at org.kitesdk.data.Datasets.create(Datasets.java:335)
>>>>> at org.apache.sqoop.connector.kite.KiteDatasetExecutor.createDataset(KiteDatasetExecutor.java:70)
>>>>> at org.apache.sqoop.connector.kite.KiteLoader.getExecutor(KiteLoader.java:52)
>>>>> at org.apache.sqoop.connector.kite.KiteLoader.load(KiteLoader.java:62)
>>>>> at org.apache.sqoop.connector.kite.KiteLoader.load(KiteLoader.java:36)
>>>>> at org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor$ConsumerThread.run(SqoopOutputFormatLoadExecutor.java:250)
>>>>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>>>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>> Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hive.ql.io.HiveOutputFormat
>>>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
>>>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
>>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
>>>>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
>>>>> ... 31 more
>>>>> 2015-08-14 15:39:11,540 [main] INFO org.apache.sqoop.job.mr.SqoopMapper - Stopping progress service
>>>>> 2015-08-14 15:39:11,540 [main] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - SqoopOutputFormatLoadExecutor::SqoopRecordWriter is about to be closed*
>>>>>
>>>>>
>>>>> p.s. sqoop version: 1.99.6
>>>>> hadoop version 2.6.0
>>>>>
>>>>
>>>>
>>>
>>
>
Re: execute sqoop job with exception
Posted by Lee S <sl...@gmail.com>.
@Abe, Hive dependeces have been added to the comon.loader at
conf/catalina.properties.
And this line
jars.add(ClassUtils.jarForClass("org.apache.hadoop.hive.conf.HiveConf"));
can find the hive jar which also contains the not founded
class(hive.ql.outputformat).
On Mon, Aug 17, 2015 at 12:30 AM, Abraham Elmahrek <ab...@cloudera.com> wrote:
> Sqoop2 delegates Parquet support to Kite. Each file format might be a
> different code path in Kite.
>
> On Fri, Aug 14, 2015 at 6:52 PM, Lee S <sl...@gmail.com> wrote:
>
>> Hi Abe:
>> I'll try to add hive deps in the codes and rebuild sqoop2.
>> But I have set the file format to FileFormat.CSV, why it comes about
>> the parquet.
>>
>> On Sat, Aug 15, 2015 at 1:00 AM, Abraham Elmahrek <ab...@cloudera.com>
>> wrote:
>>
>>> Hey man,
>>>
>>> It looks like certain Hive jars are missing from the job for some
>>> reason. Seems like we need to add more jars to
>>> https://github.com/apache/sqoop/blob/sqoop2/connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteFromInitializer.java#L71
>>> and
>>> https://github.com/apache/sqoop/blob/sqoop2/connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteToInitializer.java#L78
>>> .
>>>
>>> I've created https://issues.apache.org/jira/browse/SQOOP-2489 to track
>>> this bug. I've also created
>>> https://issues.apache.org/jira/browse/SQOOP-2490 to provide a facility
>>> to work around these kinds of issues in the future.
>>>
>>> Sqoop2 is a work in progress and still needs some battle testing. With
>>> that in mind, can you use the Avro integration instead? Otherwise, you
>>> might need to rebuild Sqoop2 with the fix to get this working at the
>>> moment. Again, https://issues.apache.org/jira/browse/SQOOP-2490 will
>>> change all of that.
>>>
>>> -Abe
>>>
>>> On Fri, Aug 14, 2015 at 1:10 AM, Lee S <sl...@gmail.com> wrote:
>>>
>>>> Hi all:
>>>> I try to import from rdms to hive with kite connector in sqoop-shell.
>>>> And I submit the job successfully , but when I try to track the job
>>>> staus
>>>> on yarn web-ui, I found there are errors in container log as below.
>>>> And the job continues running ,never stops.
>>>> The log shows *java.lang.NoClassDefFoundError:
>>>> org/apache/hadoop/hive/ql/io/HiveOutputFormat.*
>>>>
>>>> *Anybody can help?*
>>>>
>>>> *Showing 4096 bytes. Click here
>>>> <http://pdm-03:8042/node/containerlogs/container_1439241066552_0044_01_000002/root/stderr/?start=0> for
>>>> full log*
>>>>
>>>> *utFormatLoadExecutor - SqoopOutputFormatLoadExecutor consumer thread is starting
>>>> 2015-08-14 15:39:10,765 [OutputFormatLoader-consumer] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - Running loader class org.apache.sqoop.connector.kite.KiteLoader
>>>> 2015-08-14 15:39:10,771 [main] INFO org.apache.sqoop.job.mr.SqoopMapper - Starting progress service
>>>> 2015-08-14 15:39:10,772 [main] INFO org.apache.sqoop.job.mr.SqoopMapper - Running extractor class org.apache.sqoop.connector.jdbc.GenericJdbcExtractor
>>>> 2015-08-14 15:39:10,981 [OutputFormatLoader-consumer] INFO org.apache.sqoop.connector.kite.KiteLoader - Constructed temporary dataset URI: dataset:hive:wangjun/temp_5bfec97713e04374b2f2efde2dc5e4de?auth:host=pdm-03&auth:port=9083
>>>> 2015-08-14 15:39:11,093 [main] INFO org.apache.sqoop.connector.jdbc.GenericJdbcExtractor - Using query: SELECT id FROM bcpdm.history WHERE 1368 <= id AND id <= 1546
>>>> 2015-08-14 15:39:11,537 [OutputFormatLoader-consumer] ERROR org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - Error while loading data out of MR job.
>>>> java.lang.NoClassDefFoundError: org/apache/hadoop/hive/ql/io/HiveOutputFormat
>>>> at java.lang.ClassLoader.defineClass1(Native Method)
>>>> at java.lang.ClassLoader.defineClass(ClassLoader.java:800)
>>>> at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
>>>> at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
>>>> at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
>>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
>>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
>>>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
>>>> at java.lang.Class.forName0(Native Method)
>>>> at java.lang.Class.forName(Class.java:190)
>>>> at org.kitesdk.data.spi.hive.HiveUtils.getHiveParquetOutputFormat(HiveUtils.java:446)
>>>> at org.kitesdk.data.spi.hive.HiveUtils.<clinit>(HiveUtils.java:91)
>>>> at org.kitesdk.data.spi.hive.HiveManagedMetadataProvider.create(HiveManagedMetadataProvider.java:83)
>>>> at org.kitesdk.data.spi.hive.HiveManagedDatasetRepository.create(HiveManagedDatasetRepository.java:77)
>>>> at org.kitesdk.data.Datasets.create(Datasets.java:239)
>>>> at org.kitesdk.data.Datasets.create(Datasets.java:307)
>>>> at org.kitesdk.data.Datasets.create(Datasets.java:335)
>>>> at org.apache.sqoop.connector.kite.KiteDatasetExecutor.createDataset(KiteDatasetExecutor.java:70)
>>>> at org.apache.sqoop.connector.kite.KiteLoader.getExecutor(KiteLoader.java:52)
>>>> at org.apache.sqoop.connector.kite.KiteLoader.load(KiteLoader.java:62)
>>>> at org.apache.sqoop.connector.kite.KiteLoader.load(KiteLoader.java:36)
>>>> at org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor$ConsumerThread.run(SqoopOutputFormatLoadExecutor.java:250)
>>>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>> Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hive.ql.io.HiveOutputFormat
>>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
>>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
>>>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
>>>> ... 31 more
>>>> 2015-08-14 15:39:11,540 [main] INFO org.apache.sqoop.job.mr.SqoopMapper - Stopping progress service
>>>> 2015-08-14 15:39:11,540 [main] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - SqoopOutputFormatLoadExecutor::SqoopRecordWriter is about to be closed*
>>>>
>>>>
>>>> p.s. sqoop version: 1.99.6
>>>> hadoop version 2.6.0
>>>>
>>>
>>>
>>
>
Re: execute sqoop job with exception
Posted by Abraham Elmahrek <ab...@cloudera.com>.
Sqoop2 delegates Parquet support to Kite. Each file format might be a
different code path in Kite.
On Fri, Aug 14, 2015 at 6:52 PM, Lee S <sl...@gmail.com> wrote:
> Hi Abe:
> I'll try to add hive deps in the codes and rebuild sqoop2.
> But I have set the file format to FileFormat.CSV, why it comes about the
> parquet.
>
> On Sat, Aug 15, 2015 at 1:00 AM, Abraham Elmahrek <ab...@cloudera.com>
> wrote:
>
>> Hey man,
>>
>> It looks like certain Hive jars are missing from the job for some reason.
>> Seems like we need to add more jars to
>> https://github.com/apache/sqoop/blob/sqoop2/connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteFromInitializer.java#L71
>> and
>> https://github.com/apache/sqoop/blob/sqoop2/connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteToInitializer.java#L78
>> .
>>
>> I've created https://issues.apache.org/jira/browse/SQOOP-2489 to track
>> this bug. I've also created
>> https://issues.apache.org/jira/browse/SQOOP-2490 to provide a facility
>> to work around these kinds of issues in the future.
>>
>> Sqoop2 is a work in progress and still needs some battle testing. With
>> that in mind, can you use the Avro integration instead? Otherwise, you
>> might need to rebuild Sqoop2 with the fix to get this working at the
>> moment. Again, https://issues.apache.org/jira/browse/SQOOP-2490 will
>> change all of that.
>>
>> -Abe
>>
>> On Fri, Aug 14, 2015 at 1:10 AM, Lee S <sl...@gmail.com> wrote:
>>
>>> Hi all:
>>> I try to import from rdms to hive with kite connector in sqoop-shell.
>>> And I submit the job successfully , but when I try to track the job
>>> staus
>>> on yarn web-ui, I found there are errors in container log as below.
>>> And the job continues running ,never stops.
>>> The log shows *java.lang.NoClassDefFoundError:
>>> org/apache/hadoop/hive/ql/io/HiveOutputFormat.*
>>>
>>> *Anybody can help?*
>>>
>>> *Showing 4096 bytes. Click here
>>> <http://pdm-03:8042/node/containerlogs/container_1439241066552_0044_01_000002/root/stderr/?start=0> for
>>> full log*
>>>
>>> *utFormatLoadExecutor - SqoopOutputFormatLoadExecutor consumer thread is starting
>>> 2015-08-14 15:39:10,765 [OutputFormatLoader-consumer] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - Running loader class org.apache.sqoop.connector.kite.KiteLoader
>>> 2015-08-14 15:39:10,771 [main] INFO org.apache.sqoop.job.mr.SqoopMapper - Starting progress service
>>> 2015-08-14 15:39:10,772 [main] INFO org.apache.sqoop.job.mr.SqoopMapper - Running extractor class org.apache.sqoop.connector.jdbc.GenericJdbcExtractor
>>> 2015-08-14 15:39:10,981 [OutputFormatLoader-consumer] INFO org.apache.sqoop.connector.kite.KiteLoader - Constructed temporary dataset URI: dataset:hive:wangjun/temp_5bfec97713e04374b2f2efde2dc5e4de?auth:host=pdm-03&auth:port=9083
>>> 2015-08-14 15:39:11,093 [main] INFO org.apache.sqoop.connector.jdbc.GenericJdbcExtractor - Using query: SELECT id FROM bcpdm.history WHERE 1368 <= id AND id <= 1546
>>> 2015-08-14 15:39:11,537 [OutputFormatLoader-consumer] ERROR org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - Error while loading data out of MR job.
>>> java.lang.NoClassDefFoundError: org/apache/hadoop/hive/ql/io/HiveOutputFormat
>>> at java.lang.ClassLoader.defineClass1(Native Method)
>>> at java.lang.ClassLoader.defineClass(ClassLoader.java:800)
>>> at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
>>> at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
>>> at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
>>> at java.security.AccessController.doPrivileged(Native Method)
>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
>>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
>>> at java.lang.Class.forName0(Native Method)
>>> at java.lang.Class.forName(Class.java:190)
>>> at org.kitesdk.data.spi.hive.HiveUtils.getHiveParquetOutputFormat(HiveUtils.java:446)
>>> at org.kitesdk.data.spi.hive.HiveUtils.<clinit>(HiveUtils.java:91)
>>> at org.kitesdk.data.spi.hive.HiveManagedMetadataProvider.create(HiveManagedMetadataProvider.java:83)
>>> at org.kitesdk.data.spi.hive.HiveManagedDatasetRepository.create(HiveManagedDatasetRepository.java:77)
>>> at org.kitesdk.data.Datasets.create(Datasets.java:239)
>>> at org.kitesdk.data.Datasets.create(Datasets.java:307)
>>> at org.kitesdk.data.Datasets.create(Datasets.java:335)
>>> at org.apache.sqoop.connector.kite.KiteDatasetExecutor.createDataset(KiteDatasetExecutor.java:70)
>>> at org.apache.sqoop.connector.kite.KiteLoader.getExecutor(KiteLoader.java:52)
>>> at org.apache.sqoop.connector.kite.KiteLoader.load(KiteLoader.java:62)
>>> at org.apache.sqoop.connector.kite.KiteLoader.load(KiteLoader.java:36)
>>> at org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor$ConsumerThread.run(SqoopOutputFormatLoadExecutor.java:250)
>>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>> at java.lang.Thread.run(Thread.java:745)
>>> Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hive.ql.io.HiveOutputFormat
>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
>>> at java.security.AccessController.doPrivileged(Native Method)
>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
>>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
>>> ... 31 more
>>> 2015-08-14 15:39:11,540 [main] INFO org.apache.sqoop.job.mr.SqoopMapper - Stopping progress service
>>> 2015-08-14 15:39:11,540 [main] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - SqoopOutputFormatLoadExecutor::SqoopRecordWriter is about to be closed*
>>>
>>>
>>> p.s. sqoop version: 1.99.6
>>> hadoop version 2.6.0
>>>
>>
>>
>
Re: execute sqoop job with exception
Posted by Lee S <sl...@gmail.com>.
Hi Abe:
I'll try to add hive deps in the codes and rebuild sqoop2.
But I have set the file format to FileFormat.CSV, why it comes about the
parquet.
On Sat, Aug 15, 2015 at 1:00 AM, Abraham Elmahrek <ab...@cloudera.com> wrote:
> Hey man,
>
> It looks like certain Hive jars are missing from the job for some reason.
> Seems like we need to add more jars to
> https://github.com/apache/sqoop/blob/sqoop2/connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteFromInitializer.java#L71
> and
> https://github.com/apache/sqoop/blob/sqoop2/connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteToInitializer.java#L78
> .
>
> I've created https://issues.apache.org/jira/browse/SQOOP-2489 to track
> this bug. I've also created
> https://issues.apache.org/jira/browse/SQOOP-2490 to provide a facility to
> work around these kinds of issues in the future.
>
> Sqoop2 is a work in progress and still needs some battle testing. With
> that in mind, can you use the Avro integration instead? Otherwise, you
> might need to rebuild Sqoop2 with the fix to get this working at the
> moment. Again, https://issues.apache.org/jira/browse/SQOOP-2490 will
> change all of that.
>
> -Abe
>
> On Fri, Aug 14, 2015 at 1:10 AM, Lee S <sl...@gmail.com> wrote:
>
>> Hi all:
>> I try to import from rdms to hive with kite connector in sqoop-shell.
>> And I submit the job successfully , but when I try to track the job staus
>> on yarn web-ui, I found there are errors in container log as below.
>> And the job continues running ,never stops.
>> The log shows *java.lang.NoClassDefFoundError:
>> org/apache/hadoop/hive/ql/io/HiveOutputFormat.*
>>
>> *Anybody can help?*
>>
>> *Showing 4096 bytes. Click here
>> <http://pdm-03:8042/node/containerlogs/container_1439241066552_0044_01_000002/root/stderr/?start=0> for
>> full log*
>>
>> *utFormatLoadExecutor - SqoopOutputFormatLoadExecutor consumer thread is starting
>> 2015-08-14 15:39:10,765 [OutputFormatLoader-consumer] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - Running loader class org.apache.sqoop.connector.kite.KiteLoader
>> 2015-08-14 15:39:10,771 [main] INFO org.apache.sqoop.job.mr.SqoopMapper - Starting progress service
>> 2015-08-14 15:39:10,772 [main] INFO org.apache.sqoop.job.mr.SqoopMapper - Running extractor class org.apache.sqoop.connector.jdbc.GenericJdbcExtractor
>> 2015-08-14 15:39:10,981 [OutputFormatLoader-consumer] INFO org.apache.sqoop.connector.kite.KiteLoader - Constructed temporary dataset URI: dataset:hive:wangjun/temp_5bfec97713e04374b2f2efde2dc5e4de?auth:host=pdm-03&auth:port=9083
>> 2015-08-14 15:39:11,093 [main] INFO org.apache.sqoop.connector.jdbc.GenericJdbcExtractor - Using query: SELECT id FROM bcpdm.history WHERE 1368 <= id AND id <= 1546
>> 2015-08-14 15:39:11,537 [OutputFormatLoader-consumer] ERROR org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - Error while loading data out of MR job.
>> java.lang.NoClassDefFoundError: org/apache/hadoop/hive/ql/io/HiveOutputFormat
>> at java.lang.ClassLoader.defineClass1(Native Method)
>> at java.lang.ClassLoader.defineClass(ClassLoader.java:800)
>> at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
>> at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
>> at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
>> at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
>> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
>> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
>> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
>> at java.lang.Class.forName0(Native Method)
>> at java.lang.Class.forName(Class.java:190)
>> at org.kitesdk.data.spi.hive.HiveUtils.getHiveParquetOutputFormat(HiveUtils.java:446)
>> at org.kitesdk.data.spi.hive.HiveUtils.<clinit>(HiveUtils.java:91)
>> at org.kitesdk.data.spi.hive.HiveManagedMetadataProvider.create(HiveManagedMetadataProvider.java:83)
>> at org.kitesdk.data.spi.hive.HiveManagedDatasetRepository.create(HiveManagedDatasetRepository.java:77)
>> at org.kitesdk.data.Datasets.create(Datasets.java:239)
>> at org.kitesdk.data.Datasets.create(Datasets.java:307)
>> at org.kitesdk.data.Datasets.create(Datasets.java:335)
>> at org.apache.sqoop.connector.kite.KiteDatasetExecutor.createDataset(KiteDatasetExecutor.java:70)
>> at org.apache.sqoop.connector.kite.KiteLoader.getExecutor(KiteLoader.java:52)
>> at org.apache.sqoop.connector.kite.KiteLoader.load(KiteLoader.java:62)
>> at org.apache.sqoop.connector.kite.KiteLoader.load(KiteLoader.java:36)
>> at org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor$ConsumerThread.run(SqoopOutputFormatLoadExecutor.java:250)
>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>> at java.lang.Thread.run(Thread.java:745)
>> Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hive.ql.io.HiveOutputFormat
>> at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
>> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
>> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
>> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
>> ... 31 more
>> 2015-08-14 15:39:11,540 [main] INFO org.apache.sqoop.job.mr.SqoopMapper - Stopping progress service
>> 2015-08-14 15:39:11,540 [main] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - SqoopOutputFormatLoadExecutor::SqoopRecordWriter is about to be closed*
>>
>>
>> p.s. sqoop version: 1.99.6
>> hadoop version 2.6.0
>>
>
>
Re: execute sqoop job with exception
Posted by Abraham Elmahrek <ab...@cloudera.com>.
Hey man,
It looks like certain Hive jars are missing from the job for some reason.
Seems like we need to add more jars to
https://github.com/apache/sqoop/blob/sqoop2/connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteFromInitializer.java#L71
and
https://github.com/apache/sqoop/blob/sqoop2/connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteToInitializer.java#L78
.
I've created https://issues.apache.org/jira/browse/SQOOP-2489 to track this
bug. I've also created https://issues.apache.org/jira/browse/SQOOP-2490 to
provide a facility to work around these kinds of issues in the future.
Sqoop2 is a work in progress and still needs some battle testing. With that
in mind, can you use the Avro integration instead? Otherwise, you might
need to rebuild Sqoop2 with the fix to get this working at the moment.
Again, https://issues.apache.org/jira/browse/SQOOP-2490 will change all of
that.
-Abe
On Fri, Aug 14, 2015 at 1:10 AM, Lee S <sl...@gmail.com> wrote:
> Hi all:
> I try to import from rdms to hive with kite connector in sqoop-shell.
> And I submit the job successfully , but when I try to track the job staus
> on yarn web-ui, I found there are errors in container log as below.
> And the job continues running ,never stops.
> The log shows *java.lang.NoClassDefFoundError:
> org/apache/hadoop/hive/ql/io/HiveOutputFormat.*
>
> *Anybody can help?*
>
> *Showing 4096 bytes. Click here
> <http://pdm-03:8042/node/containerlogs/container_1439241066552_0044_01_000002/root/stderr/?start=0> for
> full log*
>
> *utFormatLoadExecutor - SqoopOutputFormatLoadExecutor consumer thread is starting
> 2015-08-14 15:39:10,765 [OutputFormatLoader-consumer] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - Running loader class org.apache.sqoop.connector.kite.KiteLoader
> 2015-08-14 15:39:10,771 [main] INFO org.apache.sqoop.job.mr.SqoopMapper - Starting progress service
> 2015-08-14 15:39:10,772 [main] INFO org.apache.sqoop.job.mr.SqoopMapper - Running extractor class org.apache.sqoop.connector.jdbc.GenericJdbcExtractor
> 2015-08-14 15:39:10,981 [OutputFormatLoader-consumer] INFO org.apache.sqoop.connector.kite.KiteLoader - Constructed temporary dataset URI: dataset:hive:wangjun/temp_5bfec97713e04374b2f2efde2dc5e4de?auth:host=pdm-03&auth:port=9083
> 2015-08-14 15:39:11,093 [main] INFO org.apache.sqoop.connector.jdbc.GenericJdbcExtractor - Using query: SELECT id FROM bcpdm.history WHERE 1368 <= id AND id <= 1546
> 2015-08-14 15:39:11,537 [OutputFormatLoader-consumer] ERROR org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - Error while loading data out of MR job.
> java.lang.NoClassDefFoundError: org/apache/hadoop/hive/ql/io/HiveOutputFormat
> at java.lang.ClassLoader.defineClass1(Native Method)
> at java.lang.ClassLoader.defineClass(ClassLoader.java:800)
> at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
> at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
> at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:190)
> at org.kitesdk.data.spi.hive.HiveUtils.getHiveParquetOutputFormat(HiveUtils.java:446)
> at org.kitesdk.data.spi.hive.HiveUtils.<clinit>(HiveUtils.java:91)
> at org.kitesdk.data.spi.hive.HiveManagedMetadataProvider.create(HiveManagedMetadataProvider.java:83)
> at org.kitesdk.data.spi.hive.HiveManagedDatasetRepository.create(HiveManagedDatasetRepository.java:77)
> at org.kitesdk.data.Datasets.create(Datasets.java:239)
> at org.kitesdk.data.Datasets.create(Datasets.java:307)
> at org.kitesdk.data.Datasets.create(Datasets.java:335)
> at org.apache.sqoop.connector.kite.KiteDatasetExecutor.createDataset(KiteDatasetExecutor.java:70)
> at org.apache.sqoop.connector.kite.KiteLoader.getExecutor(KiteLoader.java:52)
> at org.apache.sqoop.connector.kite.KiteLoader.load(KiteLoader.java:62)
> at org.apache.sqoop.connector.kite.KiteLoader.load(KiteLoader.java:36)
> at org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor$ConsumerThread.run(SqoopOutputFormatLoadExecutor.java:250)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hive.ql.io.HiveOutputFormat
> at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
> ... 31 more
> 2015-08-14 15:39:11,540 [main] INFO org.apache.sqoop.job.mr.SqoopMapper - Stopping progress service
> 2015-08-14 15:39:11,540 [main] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - SqoopOutputFormatLoadExecutor::SqoopRecordWriter is about to be closed*
>
>
> p.s. sqoop version: 1.99.6
> hadoop version 2.6.0
>