You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@sqoop.apache.org by Jarek Jarcec Cecho <ja...@apache.org> on 2013/11/02 02:29:49 UTC
Re: Programmatic (java) Sqoop import from MySQL to HBase Could not
find hadoop core jar
Hi Amit,
what Hadoop version are you using? Would you mind sharing with us listing of the directory that you specified in the --hadoop-home argument?
Please note that Sqoop 1 do not have officially supported client API and thus directly using Sqoop.runTool() is not supported and you might run into a lot of issues such as this one.
Jarcec
On Thu, Oct 31, 2013 at 11:09:05AM +0200, Amit Sela wrote:
> That's the stack trace for the exception I got:
> Encountered IOException running import job: java.io.IOException: Could not
> find hadoop core jar!
> at
> org.apache.sqoop.orm.CompilationManager.compile(CompilationManager.java:146)
> at org.apache.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:97)
> at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:396)
> at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:502)
> at org.apache.sqoop.Sqoop.run(Sqoop.java:145)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
> at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:181)
> at org.apache.sqoop.Sqoop.runTool(Sqoop.java:220)
> at
> com.infolinks.redmap.services.impl.utils.Sqooper.importMysqlToHBase(Sqooper.java:35)
> at
> com.infolinks.redmap.services.impl.submitters.SqoopSubmitter$1.call(SqoopSubmitter.java:79)
> at
> com.infolinks.redmap.services.impl.submitters.SqoopSubmitter$1.call(SqoopSubmitter.java:76)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:722)
>
> My infrastructure allows me to execute MR jobs and to access HBase so I
> don't understand why executing Sqoop results with "Could not find hadoop
> core jar!" unless Sqoop is actually looking for the jar to deploy it for
> some reason...
>
>
>
> On Wed, Oct 30, 2013 at 11:14 PM, Jarek Jarcec Cecho <ja...@apache.org>wrote:
>
> > Hi Amit,
> > would you mind sharing the entire stack trace that you are getting?
> >
> > Please note that Sqoop 1 do not have officially supported client API and
> > thus directly using Sqoop.runTool() is not supported and you might run into
> > a lot of issues such as this one.
> >
> > Jarcec
> >
> > On Sun, Oct 27, 2013 at 05:52:30PM +0200, Amit Sela wrote:
> > > Which jar ?
> > >
> > > Hadoop jars are in "HADOOP_HOME/lib" and also sqoop jar.
> > > I'm trying to execute the import from an OSGi running client, with a
> > maven
> > > dependency and embedding on sqoop.
> > >
> > >
> > >
> > >
> > > On Sun, Oct 27, 2013 at 5:15 PM, Krishnan K <kk...@gmail.com>
> > wrote:
> > >
> > > > Can you try adding the jar to HADOOP_CLASSPATH / CLASSPATH ?
> > > >
> > > >
> > > > On Sun, Oct 27, 2013 at 7:34 PM, Amit Sela <am...@infolinks.com>
> > wrote:
> > > >
> > > >> Hi all,
> > > >>
> > > >> I'm new to Sqoop and I'm trying to execute (from Java code) a sqoop
> > > >> import from MySQL to HBase.
> > > >>
> > > >> I'm using Sqoop.runTool() with a String[] containing all the necessary
> > > >> parameters, I even added "--hadoop-home" with the value of my
> > $HADOOP_HOME.
> > > >>
> > > >> Anyone had this problem before ? any ideas ?
> > > >>
> > > >> Thanks,
> > > >>
> > > >> Amit.
> > > >>
> > > >
> > > >
> >
Re: Programmatic (java) Sqoop import from MySQL to HBase Could not
find hadoop core jar
Posted by Amit Sela <am...@infolinks.com>.
It points to $HADOOP_HOME/bin/.. Meaning $HADOOP_HOME.
I know it looks stupid to write it that way but I wanted it to be a
complete match to the definition as running from command line.
I'll give a try the normal way.
On Nov 3, 2013 11:05 PM, "Jarek Jarcec Cecho" <ja...@apache.org> wrote:
> Hi Amit,
> the parameter --hadoop-home should be pointing to $HADOOP_HOME, not to a
> $HADOOP_HOME/bin.
>
> Jarcec
>
> On Sun, Nov 03, 2013 at 03:06:38PM +0200, Amit Sela wrote:
> > I'm running Hadoop 1.0.4 (and HBAse 0.94.2) and the directory I specify
> for
> > --hadoop-home is: "$HADOOP_HOME/bin/..".
> > That's the exact definition in our system.
> >
> >
> > On Sat, Nov 2, 2013 at 3:29 AM, Jarek Jarcec Cecho <jarcec@apache.org
> >wrote:
> >
> > > Hi Amit,
> > > what Hadoop version are you using? Would you mind sharing with us
> listing
> > > of the directory that you specified in the --hadoop-home argument?
> > >
> > > Please note that Sqoop 1 do not have officially supported client API
> and
> > > thus directly using Sqoop.runTool() is not supported and you might run
> into
> > > a lot of issues such as this one.
> > >
> > > Jarcec
> > >
> > > On Thu, Oct 31, 2013 at 11:09:05AM +0200, Amit Sela wrote:
> > > > That's the stack trace for the exception I got:
> > > > Encountered IOException running import job: java.io.IOException:
> Could
> > > not
> > > > find hadoop core jar!
> > > > at
> > > >
> > >
> org.apache.sqoop.orm.CompilationManager.compile(CompilationManager.java:146)
> > > > at org.apache.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:97)
> > > > at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:396)
> > > > at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:502)
> > > > at org.apache.sqoop.Sqoop.run(Sqoop.java:145)
> > > > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
> > > > at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:181)
> > > > at org.apache.sqoop.Sqoop.runTool(Sqoop.java:220)
> > > > at
> > > >
> > >
> com.infolinks.redmap.services.impl.utils.Sqooper.importMysqlToHBase(Sqooper.java:35)
> > > > at
> > > >
> > >
> com.infolinks.redmap.services.impl.submitters.SqoopSubmitter$1.call(SqoopSubmitter.java:79)
> > > > at
> > > >
> > >
> com.infolinks.redmap.services.impl.submitters.SqoopSubmitter$1.call(SqoopSubmitter.java:76)
> > > > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> > > > at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> > > > at
> > > >
> > >
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
> > > > at
> > > >
> > >
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
> > > > at
> > > >
> > >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> > > > at
> > > >
> > >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> > > > at java.lang.Thread.run(Thread.java:722)
> > > >
> > > > My infrastructure allows me to execute MR jobs and to access HBase
> so I
> > > > don't understand why executing Sqoop results with "Could not find
> hadoop
> > > > core jar!" unless Sqoop is actually looking for the jar to deploy it
> for
> > > > some reason...
> > > >
> > > >
> > > >
> > > > On Wed, Oct 30, 2013 at 11:14 PM, Jarek Jarcec Cecho <
> jarcec@apache.org
> > > >wrote:
> > > >
> > > > > Hi Amit,
> > > > > would you mind sharing the entire stack trace that you are getting?
> > > > >
> > > > > Please note that Sqoop 1 do not have officially supported client
> API
> > > and
> > > > > thus directly using Sqoop.runTool() is not supported and you might
> run
> > > into
> > > > > a lot of issues such as this one.
> > > > >
> > > > > Jarcec
> > > > >
> > > > > On Sun, Oct 27, 2013 at 05:52:30PM +0200, Amit Sela wrote:
> > > > > > Which jar ?
> > > > > >
> > > > > > Hadoop jars are in "HADOOP_HOME/lib" and also sqoop jar.
> > > > > > I'm trying to execute the import from an OSGi running client,
> with a
> > > > > maven
> > > > > > dependency and embedding on sqoop.
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Sun, Oct 27, 2013 at 5:15 PM, Krishnan K <
> kkrishnan88@gmail.com>
> > > > > wrote:
> > > > > >
> > > > > > > Can you try adding the jar to HADOOP_CLASSPATH / CLASSPATH ?
> > > > > > >
> > > > > > >
> > > > > > > On Sun, Oct 27, 2013 at 7:34 PM, Amit Sela <
> amits@infolinks.com>
> > > > > wrote:
> > > > > > >
> > > > > > >> Hi all,
> > > > > > >>
> > > > > > >> I'm new to Sqoop and I'm trying to execute (from Java code) a
> > > sqoop
> > > > > > >> import from MySQL to HBase.
> > > > > > >>
> > > > > > >> I'm using Sqoop.runTool() with a String[] containing all the
> > > necessary
> > > > > > >> parameters, I even added "--hadoop-home" with the value of my
> > > > > $HADOOP_HOME.
> > > > > > >>
> > > > > > >> Anyone had this problem before ? any ideas ?
> > > > > > >>
> > > > > > >> Thanks,
> > > > > > >>
> > > > > > >> Amit.
> > > > > > >>
> > > > > > >
> > > > > > >
> > > > >
> > >
>
Re: Programmatic (java) Sqoop import from MySQL to HBase Could not
find hadoop core jar
Posted by Jarek Jarcec Cecho <ja...@apache.org>.
Hi Amit,
the parameter --hadoop-home should be pointing to $HADOOP_HOME, not to a $HADOOP_HOME/bin.
Jarcec
On Sun, Nov 03, 2013 at 03:06:38PM +0200, Amit Sela wrote:
> I'm running Hadoop 1.0.4 (and HBAse 0.94.2) and the directory I specify for
> --hadoop-home is: "$HADOOP_HOME/bin/..".
> That's the exact definition in our system.
>
>
> On Sat, Nov 2, 2013 at 3:29 AM, Jarek Jarcec Cecho <ja...@apache.org>wrote:
>
> > Hi Amit,
> > what Hadoop version are you using? Would you mind sharing with us listing
> > of the directory that you specified in the --hadoop-home argument?
> >
> > Please note that Sqoop 1 do not have officially supported client API and
> > thus directly using Sqoop.runTool() is not supported and you might run into
> > a lot of issues such as this one.
> >
> > Jarcec
> >
> > On Thu, Oct 31, 2013 at 11:09:05AM +0200, Amit Sela wrote:
> > > That's the stack trace for the exception I got:
> > > Encountered IOException running import job: java.io.IOException: Could
> > not
> > > find hadoop core jar!
> > > at
> > >
> > org.apache.sqoop.orm.CompilationManager.compile(CompilationManager.java:146)
> > > at org.apache.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:97)
> > > at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:396)
> > > at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:502)
> > > at org.apache.sqoop.Sqoop.run(Sqoop.java:145)
> > > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
> > > at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:181)
> > > at org.apache.sqoop.Sqoop.runTool(Sqoop.java:220)
> > > at
> > >
> > com.infolinks.redmap.services.impl.utils.Sqooper.importMysqlToHBase(Sqooper.java:35)
> > > at
> > >
> > com.infolinks.redmap.services.impl.submitters.SqoopSubmitter$1.call(SqoopSubmitter.java:79)
> > > at
> > >
> > com.infolinks.redmap.services.impl.submitters.SqoopSubmitter$1.call(SqoopSubmitter.java:76)
> > > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> > > at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> > > at
> > >
> > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
> > > at
> > >
> > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
> > > at
> > >
> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> > > at
> > >
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> > > at java.lang.Thread.run(Thread.java:722)
> > >
> > > My infrastructure allows me to execute MR jobs and to access HBase so I
> > > don't understand why executing Sqoop results with "Could not find hadoop
> > > core jar!" unless Sqoop is actually looking for the jar to deploy it for
> > > some reason...
> > >
> > >
> > >
> > > On Wed, Oct 30, 2013 at 11:14 PM, Jarek Jarcec Cecho <jarcec@apache.org
> > >wrote:
> > >
> > > > Hi Amit,
> > > > would you mind sharing the entire stack trace that you are getting?
> > > >
> > > > Please note that Sqoop 1 do not have officially supported client API
> > and
> > > > thus directly using Sqoop.runTool() is not supported and you might run
> > into
> > > > a lot of issues such as this one.
> > > >
> > > > Jarcec
> > > >
> > > > On Sun, Oct 27, 2013 at 05:52:30PM +0200, Amit Sela wrote:
> > > > > Which jar ?
> > > > >
> > > > > Hadoop jars are in "HADOOP_HOME/lib" and also sqoop jar.
> > > > > I'm trying to execute the import from an OSGi running client, with a
> > > > maven
> > > > > dependency and embedding on sqoop.
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > On Sun, Oct 27, 2013 at 5:15 PM, Krishnan K <kk...@gmail.com>
> > > > wrote:
> > > > >
> > > > > > Can you try adding the jar to HADOOP_CLASSPATH / CLASSPATH ?
> > > > > >
> > > > > >
> > > > > > On Sun, Oct 27, 2013 at 7:34 PM, Amit Sela <am...@infolinks.com>
> > > > wrote:
> > > > > >
> > > > > >> Hi all,
> > > > > >>
> > > > > >> I'm new to Sqoop and I'm trying to execute (from Java code) a
> > sqoop
> > > > > >> import from MySQL to HBase.
> > > > > >>
> > > > > >> I'm using Sqoop.runTool() with a String[] containing all the
> > necessary
> > > > > >> parameters, I even added "--hadoop-home" with the value of my
> > > > $HADOOP_HOME.
> > > > > >>
> > > > > >> Anyone had this problem before ? any ideas ?
> > > > > >>
> > > > > >> Thanks,
> > > > > >>
> > > > > >> Amit.
> > > > > >>
> > > > > >
> > > > > >
> > > >
> >
Re: Programmatic (java) Sqoop import from MySQL to HBase Could not
find hadoop core jar
Posted by Amit Sela <am...@infolinks.com>.
I'm running Hadoop 1.0.4 (and HBAse 0.94.2) and the directory I specify for
--hadoop-home is: "$HADOOP_HOME/bin/..".
That's the exact definition in our system.
On Sat, Nov 2, 2013 at 3:29 AM, Jarek Jarcec Cecho <ja...@apache.org>wrote:
> Hi Amit,
> what Hadoop version are you using? Would you mind sharing with us listing
> of the directory that you specified in the --hadoop-home argument?
>
> Please note that Sqoop 1 do not have officially supported client API and
> thus directly using Sqoop.runTool() is not supported and you might run into
> a lot of issues such as this one.
>
> Jarcec
>
> On Thu, Oct 31, 2013 at 11:09:05AM +0200, Amit Sela wrote:
> > That's the stack trace for the exception I got:
> > Encountered IOException running import job: java.io.IOException: Could
> not
> > find hadoop core jar!
> > at
> >
> org.apache.sqoop.orm.CompilationManager.compile(CompilationManager.java:146)
> > at org.apache.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:97)
> > at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:396)
> > at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:502)
> > at org.apache.sqoop.Sqoop.run(Sqoop.java:145)
> > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
> > at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:181)
> > at org.apache.sqoop.Sqoop.runTool(Sqoop.java:220)
> > at
> >
> com.infolinks.redmap.services.impl.utils.Sqooper.importMysqlToHBase(Sqooper.java:35)
> > at
> >
> com.infolinks.redmap.services.impl.submitters.SqoopSubmitter$1.call(SqoopSubmitter.java:79)
> > at
> >
> com.infolinks.redmap.services.impl.submitters.SqoopSubmitter$1.call(SqoopSubmitter.java:76)
> > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> > at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> > at
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
> > at
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> > at java.lang.Thread.run(Thread.java:722)
> >
> > My infrastructure allows me to execute MR jobs and to access HBase so I
> > don't understand why executing Sqoop results with "Could not find hadoop
> > core jar!" unless Sqoop is actually looking for the jar to deploy it for
> > some reason...
> >
> >
> >
> > On Wed, Oct 30, 2013 at 11:14 PM, Jarek Jarcec Cecho <jarcec@apache.org
> >wrote:
> >
> > > Hi Amit,
> > > would you mind sharing the entire stack trace that you are getting?
> > >
> > > Please note that Sqoop 1 do not have officially supported client API
> and
> > > thus directly using Sqoop.runTool() is not supported and you might run
> into
> > > a lot of issues such as this one.
> > >
> > > Jarcec
> > >
> > > On Sun, Oct 27, 2013 at 05:52:30PM +0200, Amit Sela wrote:
> > > > Which jar ?
> > > >
> > > > Hadoop jars are in "HADOOP_HOME/lib" and also sqoop jar.
> > > > I'm trying to execute the import from an OSGi running client, with a
> > > maven
> > > > dependency and embedding on sqoop.
> > > >
> > > >
> > > >
> > > >
> > > > On Sun, Oct 27, 2013 at 5:15 PM, Krishnan K <kk...@gmail.com>
> > > wrote:
> > > >
> > > > > Can you try adding the jar to HADOOP_CLASSPATH / CLASSPATH ?
> > > > >
> > > > >
> > > > > On Sun, Oct 27, 2013 at 7:34 PM, Amit Sela <am...@infolinks.com>
> > > wrote:
> > > > >
> > > > >> Hi all,
> > > > >>
> > > > >> I'm new to Sqoop and I'm trying to execute (from Java code) a
> sqoop
> > > > >> import from MySQL to HBase.
> > > > >>
> > > > >> I'm using Sqoop.runTool() with a String[] containing all the
> necessary
> > > > >> parameters, I even added "--hadoop-home" with the value of my
> > > $HADOOP_HOME.
> > > > >>
> > > > >> Anyone had this problem before ? any ideas ?
> > > > >>
> > > > >> Thanks,
> > > > >>
> > > > >> Amit.
> > > > >>
> > > > >
> > > > >
> > >
>