You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@sqoop.apache.org by Radhe Radhe <ra...@live.com> on 2014/09/12 04:47:37 UTC
Java API to Import Tables from RDBMs to Hive using Sqoop
Hello everyone,
As per my experience till now I have used the direct *sqoop commands* to import Tables from RDBMs to HiveThe commands itself has a lot more switches and options.
Is there an easy way to use JAVA APIs for importing Tables using Sqoop.
Need suggestions if it's possible and viable to use a JAVA Program to import tables from RDBMs (dynamically based on the source Tables Metadata) to Hive using Sqoop .
Thanks,-RR
Re: Java API to Import Tables from RDBMs to Hive using Sqoop
Posted by pratik khadloya <ti...@gmail.com>.
You can access the incremental mode property in SqoopOptions class by
calling setIncrementalMode(IncrementalMode mode).
public enum IncrementalMode {
None,
AppendRows,
DateLastModified,
}
Also you can set the last value using setIncrementalLastValue(String
lastVal)
~Pratik
On Wed, Sep 24, 2014 at 5:50 AM, Radhe Radhe <ra...@live.com>
wrote:
> Hi Pratik,
>
> I had a look at the below references.
>
> *#1* uses SqoopOptions and finally invoking Sqoop.runTool() method in
> Java Program.
> This is since Sqoop 1.x does not have an official Java API.
>
> I this is a viable solution for performing an increment update using
> sqoop ?
>
> *#2* uses Sqoop 2 Client Java API and is not currently ready for
> production deployment.
> Latest cut of Sqoop2 is 1.99.3 (download
> <http://www.apache.org/dyn/closer.cgi/sqoop/1.99.3>, documentation
> <http://sqoop.apache.org/docs/1.99.3/index.html>). Note that 1.99.3 is
> not compatible with 1.4.5 and not feature complete, it is not intended
> for production deployment.
> I believe that leaves me to use Sqoop 1.x ONLY.
>
>
> Basically I want to do an increment update using sqoop through Java
> Program.
>
> I was thinking of invoking the entire bin/sqoop import commands with
> increment mode options from Java code using a ProcessBuilder is a correct
> option.
>
> Or is there is a better way of doing that programatically ?
>
>
> Thanks,
> -RR
>
> ------------------------------
> From: radhe.krishna.radhe@live.com
> To: user@sqoop.apache.org
> Subject: RE: Java API to Import Tables from RDBMs to Hive using Sqoop
> Date: Tue, 16 Sep 2014 10:56:48 +0530
>
> Thanks Pratik,
>
> I will have a look into.
>
> -RR
>
> ------------------------------
> From: tispratik@gmail.com
> Date: Fri, 12 Sep 2014 10:14:08 -0700
> Subject: Re: Java API to Import Tables from RDBMs to Hive using Sqoop
> To: user@sqoop.apache.org
>
> This may help you
> http://stackoverflow.com/questions/9229611/how-to-use-sqoop-in-java-program
> http://devslogics.blogspot.com/2013/09/sqoop-java-client.html
>
> ~Pratik
>
> On Thu, Sep 11, 2014 at 7:47 PM, Radhe Radhe <radhe.krishna.radhe@live.com
> > wrote:
>
> Hello everyone,
>
> As per my experience till now I have used the direct *sqoop commands* to
> import Tables from RDBMs to Hive
> The commands itself has a lot more switches and options.
>
> Is there an easy way to use JAVA APIs for importing Tables using Sqoop.
>
> Need suggestions if it's possible and viable to use a JAVA Program to
> import tables from RDBMs (*dynamically based on the source Tables
> Metadata*) to Hive using Sqoop
> Thanks,
> -RR
>
>
>
RE: Java API to Import Tables from RDBMs to Hive using Sqoop
Posted by Radhe Radhe <ra...@live.com>.
Hi Pratik,
I had a look at the below references.
#1 uses SqoopOptions and finally invoking Sqoop.runTool() method in Java Program.This is since Sqoop 1.x does not have an official Java API.
I this is a viable solution for performing an increment update using sqoop ?
#2 uses Sqoop 2 Client Java API and is not currently ready for production deployment.Latest cut of Sqoop2 is 1.99.3 (download, documentation). Note that 1.99.3 is not compatible with 1.4.5 and not feature complete, it is not intended for production deployment.I believe that leaves me to use Sqoop 1.x ONLY.
Basically I want to do an increment update using sqoop through Java Program.
I was thinking of invoking the entire bin/sqoop import commands with increment mode options from Java code using a ProcessBuilder is a correct option.
Or is there is a better way of doing that programatically ?
Thanks,-RR
From: radhe.krishna.radhe@live.com
To: user@sqoop.apache.org
Subject: RE: Java API to Import Tables from RDBMs to Hive using Sqoop
Date: Tue, 16 Sep 2014 10:56:48 +0530
Thanks Pratik,
I will have a look into.
-RR
From: tispratik@gmail.com
Date: Fri, 12 Sep 2014 10:14:08 -0700
Subject: Re: Java API to Import Tables from RDBMs to Hive using Sqoop
To: user@sqoop.apache.org
This may help youhttp://stackoverflow.com/questions/9229611/how-to-use-sqoop-in-java-program
http://devslogics.blogspot.com/2013/09/sqoop-java-client.html
~Pratik
On Thu, Sep 11, 2014 at 7:47 PM, Radhe Radhe <ra...@live.com> wrote:
Hello everyone,
As per my experience till now I have used the direct *sqoop commands* to import Tables from RDBMs to HiveThe commands itself has a lot more switches and options.
Is there an easy way to use JAVA APIs for importing Tables using Sqoop.
Need suggestions if it's possible and viable to use a JAVA Program to import tables from RDBMs (dynamically based on the source Tables Metadata) to Hive using Sqoop Thanks,-RR
RE: Java API to Import Tables from RDBMs to Hive using Sqoop
Posted by Radhe Radhe <ra...@live.com>.
Thanks Pratik,
I will have a look into.
-RR
From: tispratik@gmail.com
Date: Fri, 12 Sep 2014 10:14:08 -0700
Subject: Re: Java API to Import Tables from RDBMs to Hive using Sqoop
To: user@sqoop.apache.org
This may help youhttp://stackoverflow.com/questions/9229611/how-to-use-sqoop-in-java-program
http://devslogics.blogspot.com/2013/09/sqoop-java-client.html
~Pratik
On Thu, Sep 11, 2014 at 7:47 PM, Radhe Radhe <ra...@live.com> wrote:
Hello everyone,
As per my experience till now I have used the direct *sqoop commands* to import Tables from RDBMs to HiveThe commands itself has a lot more switches and options.
Is there an easy way to use JAVA APIs for importing Tables using Sqoop.
Need suggestions if it's possible and viable to use a JAVA Program to import tables from RDBMs (dynamically based on the source Tables Metadata) to Hive using Sqoop .
Thanks,-RR
Re: Java API to Import Tables from RDBMs to Hive using Sqoop
Posted by pratik khadloya <ti...@gmail.com>.
This may help you
http://stackoverflow.com/questions/9229611/how-to-use-sqoop-in-java-program
http://devslogics.blogspot.com/2013/09/sqoop-java-client.html
~Pratik
On Thu, Sep 11, 2014 at 7:47 PM, Radhe Radhe <ra...@live.com>
wrote:
> Hello everyone,
>
> As per my experience till now I have used the direct *sqoop commands* to
> import Tables from RDBMs to Hive
> The commands itself has a lot more switches and options.
>
> Is there an easy way to use JAVA APIs for importing Tables using Sqoop.
>
> Need suggestions if it's possible and viable to use a JAVA Program to
> import tables from RDBMs (*dynamically based on the source Tables
> Metadata*) to Hive using Sqoop .
>
> Thanks,
> -RR
>