You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@sqoop.apache.org by sh...@accenture.com on 2015/07/30 00:28:33 UTC
Anyone ran a sqoop job using JAVA code
Hi all,
Is there anyone who has written JAVA code in which "SQOOP job -exec JOBID" is being carried out?
Help is appreciated.
Regards,
Shouvanik Haldar
________________________________
This message is for the designated recipient only and may contain privileged, proprietary, or otherwise confidential information. If you have received it in error, please notify the sender immediately and delete the original. Any other use of the e-mail by you is prohibited. Where allowed by local law, electronic communications with Accenture and its affiliates, including e-mail and instant messaging (including content), may be scanned by our systems for the purposes of information security and assessment of internal compliance with Accenture policy.
______________________________________________________________________________________
www.accenture.com
Re: Anyone ran a sqoop job using JAVA code
Posted by Abraham Elmahrek <ab...@cloudera.com>.
Hey Ron,
I think that's a cool idea. Using Sqoop as a library makes a lot of sense
to me. Sqoop2 was supposed to solve a lot of these concerns, but feel free
to post something on dev@ about your idea.
-Abe
On Wed, Jul 29, 2015 at 5:40 PM, Ron Gonzalez <zl...@yahoo.com> wrote:
> I've refactored Sqoop such that I am able to invoke it asynchronously
> such that I can use Sqoop to just submit the job then return the
> application id, and I've wrapped it behind a REST API that returns the
> application id. I didn't have to use Sqoop2 or anything like that.
>
> I'm also wondering if this is something that I could contribute back to
> the Sqoop code base as an option that allows Sqoop to be used as a library
> instead of just as a command line tool.
>
> Thanks,
> Ron
>
>
> On 07/29/2015 05:14 PM, Abraham Elmahrek wrote:
>
> This isn't something usually done (shell script calling Sqoop). The shell
> action can call the Sqoop command with an absolute path, but you'll run
> into classpath issues. I haven't personally trouble shot this, but it
> should work if you resolve all the classpath issues.
>
> Check out http://sqoop.apache.org/docs/1.99.6/index.html for information
> on Sqoop2.
>
> -Abe
>
> On Wed, Jul 29, 2015 at 5:06 PM, <sh...@accenture.com> wrote:
>
>> On a second thought, I was planning to execute it with shell script.
>>
>> So, my shell script has a sqoop command.
>>
>> SQOOP is installed on all the nodes.
>>
>>
>>
>> But, when I run with OOZIE, I get a frustratingly small error.
>>
>>
>>
>> Log Type: stderr
>>
>> Log Length: 362
>>
>> ./abcd.sh: line 3: sqoop: command not found
>>
>> Failing Oozie Launcher, Main class
>> [org.apache.oozie.action.hadoop.ShellMain], exit code [1]
>>
>> log4j:WARN No appenders could be found for logger
>> (amazon.emr.metrics.MetricsSaver).
>>
>> log4j:WARN Please initialize the log4j system properly.
>>
>> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for
>> more info.
>>
>>
>>
>> Log Type: stdout
>>
>>
>>
>> Any idea, folks ?
>>
>>
>>
>>
>>
>>
>>
>> *From:* Abraham Elmahrek [mailto:abe@cloudera.com]
>> *Sent:* Wednesday, July 29, 2015 4:17 PM
>> *To:* user@sqoop.apache.org
>> *Subject:* Re: Anyone ran a sqoop job using JAVA code
>>
>>
>>
>> Have you tried Sqoop2 for Java integration?
>>
>>
>>
>> -Abe
>>
>>
>>
>> On Wed, Jul 29, 2015 at 3:28 PM, <sh...@accenture.com> wrote:
>>
>> Hi all,
>>
>> Is there anyone who has written JAVA code in which “SQOOP job –exec
>> JOBID” is being carried out?
>>
>>
>>
>> Help is appreciated.
>>
>>
>>
>> Regards,
>>
>>
>>
>> *Shouvanik Haldar*
>>
>>
>> ------------------------------
>>
>>
>> This message is for the designated recipient only and may contain
>> privileged, proprietary, or otherwise confidential information. If you have
>> received it in error, please notify the sender immediately and delete the
>> original. Any other use of the e-mail by you is prohibited. Where allowed
>> by local law, electronic communications with Accenture and its affiliates,
>> including e-mail and instant messaging (including content), may be scanned
>> by our systems for the purposes of information security and assessment of
>> internal compliance with Accenture policy.
>>
>> ______________________________________________________________________________________
>>
>> www.accenture.com
>>
>>
>>
>
>
>
Re: Anyone ran a sqoop job using JAVA code
Posted by Ron Gonzalez <zl...@yahoo.com>.
I've refactored Sqoop such that I am able to invoke it asynchronously
such that I can use Sqoop to just submit the job then return the
application id, and I've wrapped it behind a REST API that returns the
application id. I didn't have to use Sqoop2 or anything like that.
I'm also wondering if this is something that I could contribute back to
the Sqoop code base as an option that allows Sqoop to be used as a
library instead of just as a command line tool.
Thanks,
Ron
On 07/29/2015 05:14 PM, Abraham Elmahrek wrote:
> This isn't something usually done (shell script calling Sqoop). The
> shell action can call the Sqoop command with an absolute path, but
> you'll run into classpath issues. I haven't personally trouble shot
> this, but it should work if you resolve all the classpath issues.
>
> Check out http://sqoop.apache.org/docs/1.99.6/index.html for
> information on Sqoop2.
>
> -Abe
>
> On Wed, Jul 29, 2015 at 5:06 PM, <shouvanik.haldar@accenture.com
> <ma...@accenture.com>> wrote:
>
> On a second thought, I was planning to execute it with shell script.
>
> So, my shell script has a sqoop command.
>
> SQOOP is installed on all the nodes.
>
> But, when I run with OOZIE, I get a frustratingly small error.
>
> Log Type: stderr
>
> Log Length: 362
>
> ./abcd.sh: line 3: sqoop: command not found
>
> Failing Oozie Launcher, Main class
> [org.apache.oozie.action.hadoop.ShellMain], exit code [1]
>
> log4j:WARN No appenders could be found for logger
> (amazon.emr.metrics.MetricsSaver).
>
> log4j:WARN Please initialize the log4j system properly.
>
> log4j:WARN See
> http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
>
> Log Type: stdout
>
> Any idea, folks ?
>
> *From:*Abraham Elmahrek [mailto:abe@cloudera.com
> <ma...@cloudera.com>]
> *Sent:* Wednesday, July 29, 2015 4:17 PM
> *To:* user@sqoop.apache.org <ma...@sqoop.apache.org>
> *Subject:* Re: Anyone ran a sqoop job using JAVA code
>
> Have you tried Sqoop2 for Java integration?
>
> -Abe
>
> On Wed, Jul 29, 2015 at 3:28 PM, <shouvanik.haldar@accenture.com
> <ma...@accenture.com>> wrote:
>
> Hi all,
>
> Is there anyone who has written JAVA code in which “SQOOP job
> –exec JOBID” is being carried out?
>
> Help is appreciated.
>
> Regards,
>
> *Shouvanik Haldar*
>
> ------------------------------------------------------------------------
>
>
> This message is for the designated recipient only and may
> contain privileged, proprietary, or otherwise confidential
> information. If you have received it in error, please notify
> the sender immediately and delete the original. Any other use
> of the e-mail by you is prohibited. Where allowed by local
> law, electronic communications with Accenture and its
> affiliates, including e-mail and instant messaging (including
> content), may be scanned by our systems for the purposes of
> information security and assessment of internal compliance
> with Accenture policy.
> ______________________________________________________________________________________
>
> www.accenture.com <http://www.accenture.com>
>
>
RE: Anyone ran a sqoop job using JAVA code
Posted by sh...@accenture.com.
Finally the stuff got worked!
I had to include full path to SQOOP command.
Thanks for your replies.
I really appreciate.
Regards,
Shouvanik
From: Abraham Elmahrek [mailto:abe@cloudera.com]
Sent: Wednesday, July 29, 2015 5:14 PM
To: user@sqoop.apache.org
Subject: Re: Anyone ran a sqoop job using JAVA code
This isn't something usually done (shell script calling Sqoop). The shell action can call the Sqoop command with an absolute path, but you'll run into classpath issues. I haven't personally trouble shot this, but it should work if you resolve all the classpath issues.
Check out http://sqoop.apache.org/docs/1.99.6/index.html for information on Sqoop2.
-Abe
On Wed, Jul 29, 2015 at 5:06 PM, <sh...@accenture.com>> wrote:
On a second thought, I was planning to execute it with shell script.
So, my shell script has a sqoop command.
SQOOP is installed on all the nodes.
But, when I run with OOZIE, I get a frustratingly small error.
Log Type: stderr
Log Length: 362
./abcd.sh: line 3: sqoop: command not found
Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.ShellMain], exit code [1]
log4j:WARN No appenders could be found for logger (amazon.emr.metrics.MetricsSaver).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Log Type: stdout
Any idea, folks ?
From: Abraham Elmahrek [mailto:abe@cloudera.com<ma...@cloudera.com>]
Sent: Wednesday, July 29, 2015 4:17 PM
To: user@sqoop.apache.org<ma...@sqoop.apache.org>
Subject: Re: Anyone ran a sqoop job using JAVA code
Have you tried Sqoop2 for Java integration?
-Abe
On Wed, Jul 29, 2015 at 3:28 PM, <sh...@accenture.com>> wrote:
Hi all,
Is there anyone who has written JAVA code in which “SQOOP job –exec JOBID” is being carried out?
Help is appreciated.
Regards,
Shouvanik Haldar
________________________________
This message is for the designated recipient only and may contain privileged, proprietary, or otherwise confidential information. If you have received it in error, please notify the sender immediately and delete the original. Any other use of the e-mail by you is prohibited. Where allowed by local law, electronic communications with Accenture and its affiliates, including e-mail and instant messaging (including content), may be scanned by our systems for the purposes of information security and assessment of internal compliance with Accenture policy.
______________________________________________________________________________________
www.accenture.com<http://www.accenture.com>
Re: Anyone ran a sqoop job using JAVA code
Posted by Abraham Elmahrek <ab...@cloudera.com>.
This isn't something usually done (shell script calling Sqoop). The shell
action can call the Sqoop command with an absolute path, but you'll run
into classpath issues. I haven't personally trouble shot this, but it
should work if you resolve all the classpath issues.
Check out http://sqoop.apache.org/docs/1.99.6/index.html for information on
Sqoop2.
-Abe
On Wed, Jul 29, 2015 at 5:06 PM, <sh...@accenture.com> wrote:
> On a second thought, I was planning to execute it with shell script.
>
> So, my shell script has a sqoop command.
>
> SQOOP is installed on all the nodes.
>
>
>
> But, when I run with OOZIE, I get a frustratingly small error.
>
>
>
> Log Type: stderr
>
> Log Length: 362
>
> ./abcd.sh: line 3: sqoop: command not found
>
> Failing Oozie Launcher, Main class
> [org.apache.oozie.action.hadoop.ShellMain], exit code [1]
>
> log4j:WARN No appenders could be found for logger
> (amazon.emr.metrics.MetricsSaver).
>
> log4j:WARN Please initialize the log4j system properly.
>
> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for
> more info.
>
>
>
> Log Type: stdout
>
>
>
> Any idea, folks ?
>
>
>
>
>
>
>
> *From:* Abraham Elmahrek [mailto:abe@cloudera.com]
> *Sent:* Wednesday, July 29, 2015 4:17 PM
> *To:* user@sqoop.apache.org
> *Subject:* Re: Anyone ran a sqoop job using JAVA code
>
>
>
> Have you tried Sqoop2 for Java integration?
>
>
>
> -Abe
>
>
>
> On Wed, Jul 29, 2015 at 3:28 PM, <sh...@accenture.com> wrote:
>
> Hi all,
>
> Is there anyone who has written JAVA code in which “SQOOP job –exec JOBID”
> is being carried out?
>
>
>
> Help is appreciated.
>
>
>
> Regards,
>
>
>
> *Shouvanik Haldar*
>
>
> ------------------------------
>
>
> This message is for the designated recipient only and may contain
> privileged, proprietary, or otherwise confidential information. If you have
> received it in error, please notify the sender immediately and delete the
> original. Any other use of the e-mail by you is prohibited. Where allowed
> by local law, electronic communications with Accenture and its affiliates,
> including e-mail and instant messaging (including content), may be scanned
> by our systems for the purposes of information security and assessment of
> internal compliance with Accenture policy.
>
> ______________________________________________________________________________________
>
> www.accenture.com
>
>
>
RE: Anyone ran a sqoop job using JAVA code
Posted by sh...@accenture.com.
On a second thought, I was planning to execute it with shell script.
So, my shell script has a sqoop command.
SQOOP is installed on all the nodes.
But, when I run with OOZIE, I get a frustratingly small error.
Log Type: stderr
Log Length: 362
./abcd.sh: line 3: sqoop: command not found
Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.ShellMain], exit code [1]
log4j:WARN No appenders could be found for logger (amazon.emr.metrics.MetricsSaver).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Log Type: stdout
Any idea, folks ?
From: Abraham Elmahrek [mailto:abe@cloudera.com]
Sent: Wednesday, July 29, 2015 4:17 PM
To: user@sqoop.apache.org
Subject: Re: Anyone ran a sqoop job using JAVA code
Have you tried Sqoop2 for Java integration?
-Abe
On Wed, Jul 29, 2015 at 3:28 PM, <sh...@accenture.com>> wrote:
Hi all,
Is there anyone who has written JAVA code in which “SQOOP job –exec JOBID” is being carried out?
Help is appreciated.
Regards,
Shouvanik Haldar
________________________________
This message is for the designated recipient only and may contain privileged, proprietary, or otherwise confidential information. If you have received it in error, please notify the sender immediately and delete the original. Any other use of the e-mail by you is prohibited. Where allowed by local law, electronic communications with Accenture and its affiliates, including e-mail and instant messaging (including content), may be scanned by our systems for the purposes of information security and assessment of internal compliance with Accenture policy.
______________________________________________________________________________________
www.accenture.com<http://www.accenture.com>
RE: Anyone ran a sqoop job using JAVA code
Posted by sh...@accenture.com.
Thanks for reply, Abe.
How do I do that?
Do you have any document, supporting this?
From: Abraham Elmahrek [mailto:abe@cloudera.com]
Sent: Wednesday, July 29, 2015 4:17 PM
To: user@sqoop.apache.org
Subject: Re: Anyone ran a sqoop job using JAVA code
Have you tried Sqoop2 for Java integration?
-Abe
On Wed, Jul 29, 2015 at 3:28 PM, <sh...@accenture.com>> wrote:
Hi all,
Is there anyone who has written JAVA code in which “SQOOP job –exec JOBID” is being carried out?
Help is appreciated.
Regards,
Shouvanik Haldar
________________________________
This message is for the designated recipient only and may contain privileged, proprietary, or otherwise confidential information. If you have received it in error, please notify the sender immediately and delete the original. Any other use of the e-mail by you is prohibited. Where allowed by local law, electronic communications with Accenture and its affiliates, including e-mail and instant messaging (including content), may be scanned by our systems for the purposes of information security and assessment of internal compliance with Accenture policy.
______________________________________________________________________________________
www.accenture.com<http://www.accenture.com>
Re: Anyone ran a sqoop job using JAVA code
Posted by Abraham Elmahrek <ab...@cloudera.com>.
Have you tried Sqoop2 for Java integration?
-Abe
On Wed, Jul 29, 2015 at 3:28 PM, <sh...@accenture.com> wrote:
> Hi all,
>
> Is there anyone who has written JAVA code in which “SQOOP job –exec JOBID”
> is being carried out?
>
>
>
> Help is appreciated.
>
>
>
> Regards,
>
>
>
> *Shouvanik Haldar*
>
> ------------------------------
>
> This message is for the designated recipient only and may contain
> privileged, proprietary, or otherwise confidential information. If you have
> received it in error, please notify the sender immediately and delete the
> original. Any other use of the e-mail by you is prohibited. Where allowed
> by local law, electronic communications with Accenture and its affiliates,
> including e-mail and instant messaging (including content), may be scanned
> by our systems for the purposes of information security and assessment of
> internal compliance with Accenture policy.
>
> ______________________________________________________________________________________
>
> www.accenture.com
>