You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@sqoop.apache.org by Silva João <jo...@altran.com> on 2016/04/05 17:47:03 UTC
kite-data-mapreduce missing?
Hi,
I'm running an oozie coordinator to periodically run an oozie workflow
which runs an *incremental* sqoop import from Postgres to Parquet files.
The coordinator was running OK, when no new data was added to the
origin. But after I inserted a new line in the Postgres DB, oozie/sqoop
failed with the following error:
Failing Oozie Launcher, Main class
[org.apache.oozie.action.hadoop.SqoopMain], main() threw exception,
org/kitesdk/data/mapreduce/DatasetKeyOutputFormat
java.lang.NoClassDefFoundError:
org/kitesdk/data/mapreduce/DatasetKeyOutputFormat
at
org.apache.sqoop.mapreduce.DataDrivenImportJob.getOutputFormatClass(DataDrivenImportJob.java:190)
I've read that the jar for kite-data-mapreduce is missing? Is this
correct? How do I install such a package?
Thanks.
--
João M. S. Silva
Re: kite-data-mapreduce missing?
Posted by Silva João <jo...@altran.com>.
Hi Debabrata,
I did that, although in directory /user/hdfs/sqoop/kite-data-mapreduce.jar.
In the logs, I see this jar being loaded but the result is the same.
João M. S. Silva
On 06/04/2016 01:45, Debabrata Pani wrote:
> Hi Joao,
> Should we not add this jar too oozie application lib subdirectory ?
> Generally the workflow.xml is in a folder on HDFS. We can have a lib
> subdirectory under the folder and add jars to it. Can we try doing that ?
>
> On Apr 5, 2016 9:55 PM, "Silva João" <joao.silva@altran.com
> <ma...@altran.com>> wrote:
>
> I just realized that a file
> /usr/hdp/2.4.0.0-169/sqoop/lib/kite-data-mapreduce-1.0.0.jar exists. I
> copied it to HDFS and added it to the workflow.xml, e.g.:
>
> <file>/user/hdfs/sqoop/kite-data-mapreduce.jar</file>
>
> I see it being used in the logs, but it does not change the result.
>
> João M. S. Silva
>
> On 05/04/2016 16:47, Silva João wrote:
> > Hi,
> >
> > I'm running an oozie coordinator to periodically run an oozie
> workflow
> > which runs an *incremental* sqoop import from Postgres to Parquet
> files.
> >
> > The coordinator was running OK, when no new data was added to the
> > origin. But after I inserted a new line in the Postgres DB,
> oozie/sqoop
> > failed with the following error:
> >
> > Failing Oozie Launcher, Main class
> > [org.apache.oozie.action.hadoop.SqoopMain], main() threw exception,
> > org/kitesdk/data/mapreduce/DatasetKeyOutputFormat
> > java.lang.NoClassDefFoundError:
> > org/kitesdk/data/mapreduce/DatasetKeyOutputFormat
> > at
> >
> org.apache.sqoop.mapreduce.DataDrivenImportJob.getOutputFormatClass(DataDrivenImportJob.java:190)
> >
> > I've read that the jar for kite-data-mapreduce is missing? Is this
> > correct? How do I install such a package?
> >
> > Thanks.
> >
>
Re: kite-data-mapreduce missing?
Posted by Debabrata Pani <an...@gmail.com>.
Hi Joao,
Should we not add this jar too oozie application lib subdirectory ?
Generally the workflow.xml is in a folder on HDFS. We can have a lib
subdirectory under the folder and add jars to it. Can we try doing that ?
On Apr 5, 2016 9:55 PM, "Silva João" <jo...@altran.com> wrote:
> I just realized that a file
> /usr/hdp/2.4.0.0-169/sqoop/lib/kite-data-mapreduce-1.0.0.jar exists. I
> copied it to HDFS and added it to the workflow.xml, e.g.:
>
> <file>/user/hdfs/sqoop/kite-data-mapreduce.jar</file>
>
> I see it being used in the logs, but it does not change the result.
>
> João M. S. Silva
>
> On 05/04/2016 16:47, Silva João wrote:
> > Hi,
> >
> > I'm running an oozie coordinator to periodically run an oozie workflow
> > which runs an *incremental* sqoop import from Postgres to Parquet files.
> >
> > The coordinator was running OK, when no new data was added to the
> > origin. But after I inserted a new line in the Postgres DB, oozie/sqoop
> > failed with the following error:
> >
> > Failing Oozie Launcher, Main class
> > [org.apache.oozie.action.hadoop.SqoopMain], main() threw exception,
> > org/kitesdk/data/mapreduce/DatasetKeyOutputFormat
> > java.lang.NoClassDefFoundError:
> > org/kitesdk/data/mapreduce/DatasetKeyOutputFormat
> > at
> >
> org.apache.sqoop.mapreduce.DataDrivenImportJob.getOutputFormatClass(DataDrivenImportJob.java:190)
> >
> > I've read that the jar for kite-data-mapreduce is missing? Is this
> > correct? How do I install such a package?
> >
> > Thanks.
> >
>
Re: kite-data-mapreduce missing?
Posted by Silva João <jo...@altran.com>.
I just realized that a file
/usr/hdp/2.4.0.0-169/sqoop/lib/kite-data-mapreduce-1.0.0.jar exists. I
copied it to HDFS and added it to the workflow.xml, e.g.:
<file>/user/hdfs/sqoop/kite-data-mapreduce.jar</file>
I see it being used in the logs, but it does not change the result.
João M. S. Silva
On 05/04/2016 16:47, Silva João wrote:
> Hi,
>
> I'm running an oozie coordinator to periodically run an oozie workflow
> which runs an *incremental* sqoop import from Postgres to Parquet files.
>
> The coordinator was running OK, when no new data was added to the
> origin. But after I inserted a new line in the Postgres DB, oozie/sqoop
> failed with the following error:
>
> Failing Oozie Launcher, Main class
> [org.apache.oozie.action.hadoop.SqoopMain], main() threw exception,
> org/kitesdk/data/mapreduce/DatasetKeyOutputFormat
> java.lang.NoClassDefFoundError:
> org/kitesdk/data/mapreduce/DatasetKeyOutputFormat
> at
> org.apache.sqoop.mapreduce.DataDrivenImportJob.getOutputFormatClass(DataDrivenImportJob.java:190)
>
> I've read that the jar for kite-data-mapreduce is missing? Is this
> correct? How do I install such a package?
>
> Thanks.
>